Balancing ethics and statistics: machine learning facilitates highly accurate classification of mice according to their trait anxiety with reduced sample sizes

Obtaining labeled training data for supervised ML from multiple animal cohorts

In our foundational work on establishing a behavioral pipeline to study trait anxiety in female and male inbred mice, we employed auditory aversive conditioning (AAC) protocols with prolonged memory retrieval (MR) sessions [9]. Extending conditioned stimulus (CS) presentation to 6 min instead of commonly used brief CS exposures of 30 s unlocked a range of freezing responses that allowed us to cluster animals into sustained and phasic responders based on their individual freezing behavior. To capture its dynamics, the freezing response during retrieval session was analyzed in 30-s time bins. We performed one MR session 24 h and the second MR session either 48 h or on day 28 after conditioning (Fig. 1a). Originally, each experimental batch was clustered separately using Gaussian mixture models (GMM). To obtain more reliable estimates of the population-level freezing response distributions, we therefore decided to pool data from multiple experimental cohorts together and repeat the clustering procedure (Fig. 1a). Consequently, we used data from 5 female and 8 male experimental batches (Supplementary Table 1).

In agreement with our previous results [9], we observed consistently higher freezing values in females compared to males in MR1 (Fig. 1b) and MR2 (Supplementary Fig. 1a). Furthermore, average sustained freezing (the last 3.5 min of CS presentation), identified as the main behavioral readout reflecting the animal’s anxiety trait [9], exhibited a right-skewed distribution towards lower values in male mice. In contrast, females showed a wider spread across the range of sustained freezing values (Fig. 1c, Supplementary Fig. 1b). These consistent sex-related differences in freezing behavior across multiple animal cohorts justified fitting separate models to assign female and male mice to distinct freezing phenotypes.

To this end, we employed our previously described strategy [9]. First, we utilized log-linear regression to model the freezing response of each animal during CS presentation (Fig. 1d, g; Supplementary Fig. 1c, f). Next, we employed GMM clustering using the intercept and slope (decay rate of freezing) of the fitted freezing curve as well as the average freezing response during the last 3.5 min of CS presentation, that we previously defined as sustained phase of the freezing response. Thus, we obtained two distinct subgroups of mice for each sex and MR (Fig. 1e, h; Supplementary Fig. 1d, g). Sustained responders were associated with significantly higher freezing during MR1 and MR2 for both sexes compared to phasic responders (Fig. 1f, i; Supplementary Fig. 1e, h). Differences were especially pronounced in the sustained phase of the freezing response during the last 3.5 min of CS presentation, where we already previously observed the highest inter-individual variability [9].

Assessing clustering stability

While we pooled multiple animal cohorts in an effort to reliably model the population-level freezing response distributions, our clustering into sustained and phasic responders could still be influenced by sampling error. Therefore, we employed a bootstrap sampling procedure to ascertain the validity and robustness of our clustering results. To this end, we drew 200 random samples with replacement for female and male mice at MR1 and MR2 (see Methods) and compared clusters to the original phenotype assignment. Notably, we observed highly similar average freezing curves for sustained and phasic female and male mice in the bootstrap samples at MR1 (Fig. 2a, b, d, e) and MR2 (Supplementary Fig. 2a, b, d, e). Furthermore, average freezing remained higher for sustained compared to phasic responders even in the bootstrap samples. Next, we calculated the Jaccard index (see Methods) to investigate how often each animal retained its original cluster assignment in the bootstrap samples. We observed highly robust clustering results for female sustained and phasic responders at both MR1 (Fig. 2c) and MR2 (Supplementary Fig. 2c) as indicated by median Jaccard index values higher than 0.99. In contrast, we detected a higher variability and slightly lower median Jaccard index values for males compared to females at MR1 (Fig. 2f) and MR2 (Supplementary Fig. 2f). However, at the median, male animals also retained their original cluster assignment in more than 89% of the cases. These results clearly indicated that our animal cohorts reliably reflect population-level freezing distributions and clustering into sustained and phasic responders remains robust towards sampling errors for both female and male mice.

Training supervised ML models to assign mice to sustained and phasic freezing phenotypes

Having demonstrated that pooling multiple experimental cohorts together results in stable clusters, we proceeded to use these labeled data to train supervised ML models. Notably, assigning mice to sustained and phasic phenotypes using ML classifiers trained on this larger dataset should produce more stable and reliable results in contrast to de novo clustering experimental batches separately in new experiments that could still be influenced by the sample size.

For model training, we utilized several well-established ML algorithms including support vector machines (SVM), logistic regression, linear discriminant analysis (LDA) and random forests. Since we applied our clustering procedure separately to each sex and MR, we trained individual ML models for each distinct experimental set-up. To evaluate the performance of the ML models, we split the labeled datasets into training and test sets with a ratio of 70:30%, respectively. However, we observed high variances between different training iterations, which is likely due to the limited sample sizes of the training sets. To address this issue, we performed 1000 iterations of randomly partitioning the dataset into training and test splits, which is a method known as Monte Carlo cross-validation [13]. Thus, we aimed at obtaining more robust estimates of each ML model’s performance by averaging the respective performance metric over all cross-validation iterations.

Remarkably, all ML models yielded high average accuracies exceeding 90% for each experimental scenario. Interestingly, ML models trained on data from female animals resulted in higher average accuracies than models trained on male animals for both MR sessions (Fig. 3a–d). Furthermore, models trained with MR2 datasets showed higher average accuracies than those trained with MR1 datasets for both sexes. Notably, for all scenarios except for male animals at MR1, several iterations reached perfect classification accuracies of 100%. For both female MR1 and MR2 training datasets, logistic regression yielded the highest average accuracy (Supplementary Table 2). For male animals, the random forest model and the SVM model with a linear kernel showed the highest average accuracy for MR1 and MR2, respectively. Based on this evaluation, we selected the best performing ML models (Table 1) and subsequently trained them using the complete respective dataset for further use in our study.

Performance of ML models on independent animal batches

To assess whether the performance of our ML classifiers generalizes to independent data from different mouse lines, we used a batch (FC85) of 38 female Arc-nuGFP animals (Methods) that was not included in the pooled clustering and ML model training steps. FC85 females were subjected to our standard AAC pipeline and MRs were performed on day 2 and day 39. Subsequently, we used individual freezing parameters to assign mice to a sustained or phasic phenotype employing both our original clustering procedure and our supervised ML models. Notably, we observed an almost perfect agreement between both approaches for MR1 (Fig. 4a–c) and MR2 (Supplementary Fig. 3a–c) as also indicated by high Cohen’s kappa values greater than 0.885 (Fig. 4c, Supplementary Fig. 3c). Indeed, only two animals for both MR1 and MR2, respectively, were assigned to a different group using our ML classifier compared to standard GMM clustering.

While clustering and classification results were highly concordant, one significant advantage of ML models is that once they have been trained using a sufficiently large data set, their classification performance on new data is independent of sample size. In contrast, clustering results can quickly become unstable with small samples when de novo clustering is performed each time in independent animal cohorts. To further investigate this phenomenon empirically, we employed a subsampling procedure. In 1000 iterations, we drew subsamples of the female and male mice used for ML model training of varying sizes starting from 100 animals and going down to 5 animals. For each subsample, we then clustered animals into sustained and phasic responders and compared the average accuracy to the clustering results obtained with the total animal population. We observed a substantial drop in accuracy as subsample sizes decreased for both female and male animals in MR1 and MR2 (Fig. 4d, left). Next, we applied the same strategy with our pre-trained ML models. The performance remained unaffected by sample size as indicated by the uniformly high accuracy values (Fig. 4d, right). To address the bias that we tested the ML models on the data that was used for training, we performed the subsampling procedure with the independent batch (FC85) with subsample sizes ranging from 38 to 3 animals. Again, we observed a substantial decline in clustering accuracy for smaller samples. In contrast, the ML classifier maintained uniformly high accuracy around 0.95 across all sample sizes (Fig. 4e). These results clearly demonstrated the advantage of using pre-trained ML models to assign animals to sustained and phasic responders and highlighted the potential for substantial reduction of sample sizes in future experiments, supporting the 3R principles in animal research.

ML classification into sustained and phasic responders remains robust towards AAC protocol modifications and batch variability

In our original AAC protocol, we had two training sessions that were performed within 5 h of each other. To assess whether a single training session could still induce inter-individual variability and similar group-level freezing responses, we modified the conditioning protocol and tested it on a batch of 40 female mice (FC87, Fig. 5a).

Average freezing behavior of FC87 females was consistently lower at MR1 (Fig. 5c) and MR2 (Fig. 5d) compared to the ML training batches. Notably, we also observed significantly lower freezing on the adaptation day (Fig. 5b), suggesting that not only the AAC protocol modifications but also batch-specific variability influenced the freezing behavior of FC87 females.

To investigate the impact of these variations in our phenotyping procedure, we applied our standard clustering methods as well as our supervised ML classifiers to assign FC87 female mice into sustained and phasic responders. Interestingly, we observed high concordance of phenotyping results at MR1 with only one animal allocated differently between both approaches (Fig. 5e, f). However, clustering and ML classification differed drastically at MR2 (Fig. 5g, h). While batch-specific clustering produced two balanced groups, ML classification only assigned three animals to the sustained group. To further explore these discrepancies, we overlayed the confidence ellipses for the sustained and phasic groups derived from the female batches used for training the ML model (Fig. 5e–h, solid color confidence ellipses). FC87 batch-specific clustering at MR1 displayed similar distributions to the ML training batches. However, at MR2, FC87 clusters deviated substantially from the ML training batches, thereby highlighting the disparity between clustering and ML classification.

These results clearly demonstrated the benefits of using pre-trained supervised ML models for classifying animals into sustained and phasic responders. In the presence of protocol modifications or batch variability that cause shifts in the freezing response distributions, our ML classifiers provide robust assignment of mice to phenotypes that remain comparable across multiple animal cohorts.

R package

To facilitate a consistent and reproducible utilization of our methods in future studies, we developed an R package – anxiotraitR available at https://github.com/johannesmiedema/anxiotraitR. The package takes freezing data obtained from our behavioral pipeline [9] as input and uses our best performing ML classifiers depending on sex and MR (Table 1) to allocate animals to sustained and phasic responders. Furthermore, we implemented the option to include additional data if they become available and use them to retrain the ML classifiers. This feature enables the continuous improvement and further refinement of our proposed models.