Model architecture optimized for deep learning on genomic data

Hyperparameter search space

The hyperparameter space used for optimization is listed in Table 1 and described in further detail here.

The first part of the model built by GenomeNet-Architect consists of a series of convolutional blocks (Figure 1), each block consisting of convolutional layers. Number of blocks (N_C.B.) and the number of layers in each block (s_C.B.) Determined by HP. n_C.B. and n_c In the following way: N_C.B. is set directly to n_C.B. unless there is n_c (related to the total number of convolutional layers) is less than that. Therefore, their relationship is

s_C.B. It is calculated by rounding off the ratio. n_c Hyperparameter for the actual number of convolution blocks N_C.B.:

As a result, n_c Determine the approximate total number of convolutional layers, satisfying the constraint that each convolutional block has the same (integer) number of layers. The total number of convolutional layers is determined as:

f₀ and f_end Determine the number of filters for the first or last convolutional layer, respectively. The number of filters in the hidden layer is interpolated exponentially. When residual blocks are used, the number of filters in each convolutional block must be the same, in which case the number of filters varies from block to block. Otherwise, each convolutional layer can have a different number of filters. If there is only one convolutional layer, ⌈f₀⌉ Used as the number of filters in this layer. Therefore, the number of filters is I^th The convolutional layers are:

The convolutional layer kernel size is also exponentially interpolated between: k₀ and k_end. If the model has only one convolutional layer, the kernel size is set as follows: ⌈k₀⌉.Kernel size of convolutional layer I teeth:

Convolutional layers can use dilated convolutions. The expansion factor increases exponentially from 1 to 2. d_end within each convolutional block. Use “”Rem” as the remainder operation, the expansion factor for each layer is:

Apply max pooling after the convolutional layers depending on the total max pooling factor p._end. A stride max-pooling layer and a kernel size of 2 or a power of 2 are inserted between the convolution layers to reduce the sequence length exponentially along the model. p_end represents an approximation of the total shortening of the sequence length before the output of the convolution part is fed to the final GAP or RNN layer, depending on the model type.

For CNN-GAP, the outputs from multiple convolutional blocks can be pooled, concatenated, and fed into a fully connected network.out of N_C.B. output, last minutes(1, ⌈(1 − r_s)× N_C.B.⌉) one of them is fed to the global average pooling layer. r_s Skip rate hyperparameter.

Hyperparameter optimization process

GenomeNet-Architect uses mlrMBO software³⁸ Using the Gaussian process model from the DiceKriging R package³⁹ Consists of the Matérn-3/2 kernel⁴⁰ for optimization. I am using UCB³¹ Filling standards, sampling λ Bulk proposal method from exponential distribution³². In our experiments, we proposed three different configurations simultaneously in each iteration.

For both tasks, we trained the proposed model configuration for a certain amount of time and then evaluated it on a validation set. For each architecture (CNN-GAP and CNN-RNN) and each sequence length (150 nt and 10,000 nt) for the virus classification task, the best performing model configuration found within the optimization settings (2 hours, 6 hours) was saved. it was done. and will be considered for further evaluation. For the pathogenicity detection task, we only evaluated the 2-hour optimization.For each task and sequence length value, the first t = t₁ (2 hours) of optimization evaluated a total of 788 configurations, parallelized on 24 GPUs, and ran for 2.8 days (elapsed time).For virus classification tasks, warm start t = t₂ (6 hours) optimization evaluated an additional 408 configurations and ran for 7.0 days for each sequence length value.

During HPO, the number of samples between model validation evaluations was dynamically set depending on the time taken for a single model training step. Approximately 20 validation evaluations were chosen to be performed for each model in the first phase (t = 2 hours), approximately 100 validation evaluations were performed in the second phase (t = 6 hours). In the first phase, the highest validation accuracy found during model training was used as the target value to be optimized. In the second phase, his second highest validation accuracy found in the past 20 validation evaluations was used as the target value. This was done to avoid rewarding the model with a very noisy training process with performance outliers.

The batch size for each model architecture is chosen to be as large as possible while still fitting in GPU memory. To do this, GenomeNet-Architect performs a binary search to find the largest model that fits on the GPU, subtracting a 10% safety margin to avoid potential training failures.

Architecture evaluation and benchmarking

For virus classification tasks, training and validation samples are generated by randomly sampling FASTA genome files and dividing them into disjoint consecutive subsequences from random starting points. Batch sizes that are a multiple of 3 (number of target classes) are used, with each batch containing the same number of samples from each class. Since we are dealing with datasets with different amounts of data for each class, the minor classes are effectively oversampled compared to the largest class. Performance on the validation set was evaluated periodically after training with a predetermined number of samples (6,000,000 for the 150 nt model and 600,000 for the 10,000 nt model). For evaluation, we used a subsample of the validation set equal to 50% of the training samples seen during each validation. Typical batch sizes during model training were 1200 for a 150 nt model and either 120, 60, or 30 for a 10,000 nt model. Unlike during training and validation, the test set samples were not randomly generated by selecting random FASTA files. Instead, test samples were generated by iterating over all individual files and using consecutive subsequences starting from the first position. For the pathogenicity detection task, validation performance was evaluated periodically on the complete set, specifically once after training on 5,000,000 samples. A batch size of 1000 was used for all models except GAP-RNN, as it was not possible with the memory of the GPU. A batch size of 500 was used in this model.

For both tasks, we chose a learning rate schedule that automatically cut the learning rate in half if the balanced accuracy did not increase over three consecutive evaluations on the validation set. Training was stopped if balance accuracy did not improve after 10 consecutive evaluations. This typically corresponds to stopping training after 40/50 evaluations for a 150 nt model, 25/35 evaluations for a 10,000 nt model for a virus classification task, and 5/15 evaluations for a pathogenicity detection task.

To evaluate the performance of the architecture and HP configuration, the model proposed by GenomeNet-Architect was trained until convergence on the training set. Convergence was checked on the validation set. The resulting model was evaluated on a test set that was not seen during optimization.

data set

For the virus classification task, the arguments -d “genbank, refseq” -g “bacteria”/”virus” -c “all” and -l “complete genome”. All genomes were removed from the multichromosomal bacterial set to remove possible contamination consisting of plasmids and bacteriophages. Filtering plasmids for inconsistent and poor annotation in databases introduces substantial noise into sequences and annotations as they may be incorrectly included or excluded from the genome. can be avoided. We used taxonomic metadata to partition the virus set into eukaryotic or prokaryotic viruses. Overall, this resulted in three subgroups: bacteria, prokaryotic bacteriophages, and eukaryotic viruses (referred to as nonphage viruses, Table 2). To evaluate the generalization performance of the model, we subset the genome into training, validation, and test subsets. We used “Publication Date” metadata to partition the data by publication time. The training data mainly consists of genomes published before 2020, and the validation and test data consist of recently published genomes. Therefore, when applied to newly sequenced DNA, the classification performance of the model on still unknown data is estimated. For small datasets, use average nucleotide identity information (ANI) generated by tools such as Mashtree.⁴¹ Performing a split can also be used to avoid overlap between training and testing data.

Training data is used for model fitting, validation data is used to estimate generalization performance during HPO and check convergence during final model training, and test data is used to compare the performance of the final model and draw conclusions. It was used for. Test data was not recognized by the optimization process. The training, validation, and test sets account for approximately 70%, 20%, and 10% of the total data, respectively.

The number of FASTA files in the set of virus classification tasks and the number of unique samples in the set are shown in Table 2. Listed is the number of different distinct sequences that could theoretically be extracted from the dataset. Split into consecutive subsequences. However, each time the training process reads the file again, for example in another epoch, the starting point of the sampled sequence is randomized, thus significantly increasing the number of possible distinct (but overlapping) samples. Increase. Because the test set size is unbalanced, we report class-balanced measurements, i.e., measurements computed for each class separately and averaged over all classes.

For the pathogenicity classification task, we downloaded the dataset from https://zenodo.org/records/367856313. Specifically, the training files used are nonpathogenic_train.fasta.gz, pathogenic_train.fasta.gz, the validation files used are pathogenic_val.fasta.gz, nonpathogenic_val.fasta.gz, and the test files used are nonpathogenic_test_1.fasta. .gz, nonpathogenic_test_2. fasta.gz, pathogenic_test_1.fasta.gz, pathogenic_test_2.fasta.gz.

Report overview

For more information on the study design, please see the Nature Portfolio Reporting Summary linked in this article.