An extended description of the methods can be found in the supplementary materials.
Application design, structure and technology stack
AIM Review is a client-side web application developed using Google’s Firebase platform. The application operates without a server-side connection, meaning that all the data and algorithms run in the user’s local browser. AIM Review has been tested across major browsers—including Google Chrome, Mozilla Firefox, and Microsoft Edge—and on multiple devices, such as Windows PCs, Android phones, and tablets.
The web application has a modular structure with three independent components. The first is the labeling application, which allows users to upload publications from a variety of databases in multiple formats (e.g., Excel spreadsheet, RIS files and text files), remove duplicates, manually screen them as relevant or irrelevant based on a given inclusion and exclusion criteria, and track screening progress with interactive plots. It includes an active learning algorithm that iteratively predicts relevance scores for unscreened publications and sorts them based on relevance. The second is the agreement application, used to assess inter-rater and document-level agreement between screeners by importing labels from the labeling application and resolving disputes among raters (see supplementary materials for a description of the agreement measures). The third is the ML prediction application, which applies NCV-based ML pipelines to classify a subset of manually labeled publications based on their relevance, and predict the relevance of unscreened publications. Importantly, each module of the application can be run in parallel. Publication screening, agreement measures and model predictions can be saved into a standardized spreadsheet file.
The AIM Review web application framework is built with HTML, CSS, and object-oriented JavaScript. ML computations run in the background using JavaScript web workers. Python-based ML libraries, such as scikit-learn38, are executed utilizing Pyodide’s WebAssembly implementation39. TensorFlow.js40 powers the training of sequential neural networks and Doc2Vec sentence encoders, while HuggingFace.js and the HuggingFace inference API41 were used to generate embeddings from pretrained sentence transformers.
Vectorization of text
AIM Review incorporates multiple text vectorization methods to convert titles and abstracts into numerical representations for ML classification:
-
Term frequency-inverse document frequency (TF-IDF; scikit-learn)42 — default option
-
Latent semantic analysis (LSA; scikit-learn) of TF-IDF using truncated singular value decomposition
-
Universal sentence encoder (univBERT; tensorflow.js)43
-
Document to vector embedding model (Doc2Vec; tensorflow.js native implementation)44
-
Pretrained sentence transformers (e.g., all-MiniLM-L6-v2 and SPECTER2; transformers.js)45,46.
AIM Review allows users to choose multiple vectorization methods and combine the embeddings either via concatenation of embeddings (fusion) or via stacked generalization47. Each method balances speed and complexity—simpler models like TF-IDF are faster, while deep learning models like sentence transformers are slower but able to capture more complex semantic relationships.
Classification models
AIM Review includes various classifier algorithms that can be trained on user labels to predict the relevance of unseen publications:
-
Logistic regression (LR; scikit-learn) — default option
-
Linear and non-linear support vector machines (SVM; scikit-learn)
-
Decision tree classifier (DT; scikit-learn)
-
Multi-layer perceptron (MLP; scikit-learn)
-
Sequential neural networks (SeqNN; tensorflow.js).
The application supports both shallow (e.g., LR) and deep learners (e.g., SeqNN), but shallow learners were prioritized in the benchmarking to maximize efficiency and minimize latency, in line with previously developed applications. The model parameters can be configured by users. Additionally, using the NCV framework, selected model hyperparameters can be optimized (see supplementary materials for further details on model parameters).
Active learning strategies
The active learning methodologies in AIM Review are inspired by state-of-the-art implementations of active learning for systematic review screening9,10,11, and applied into a web environment. Figure 4A illustrates the active learning strategies used in the labeling application of AIM Review. Briefly, once the user begins screening titles and abstracts, an active learning cycle can be activated in which a classifier model is trained using the numerical representations of the screened papers to predict the assigned label. The classifier is then applied to predict the relevance of unlabeled entries. The predicted relevance scores are displayed using a color-coded scheme. Users can choose to sort the publications based on their relevance score (high or low relevance first, or uncertain first; see supplementary materials), as well as add random percentage of publications to avoid unrepresentative prioritizations. This cycle can be reactivated every time the user clicks a button or, alternatively, at regular time intervals. Classifiers are applied without hyperparameter tuning; all parameters are pre-defined by the user.
A Active learning pipeline for labeling. Documents (titles and abstracts from publications) are screened by the researcher. Once active learning is activated, titles and abstracts are converted into numerical representations using selected vectorization methods, and a classifier is trained on publications screened by the user to predict relevance. The classifier is then applied to predict the relevance scores of unscreened publications, which are displayed in the application. The researcher can sort publications by relevance (heist or lowest relevance first) or uncertainty (closeness to 50% relevance). This process is repeated with each button press or at set intervals. Sorting by highest relevance allows researchers to stop screening once no relevant papers appear. B Supervised learning pipeline for ML prediction. A subset of documents (X%) is manually screened to train and optimize model parameters via nested cross-validation. After finding the optimal hyperparameters, the model is applied to all screened samples, and an ROC curve is created to assess sensitivity and specificity at different relevance thresholds. The remaining publications (100-X%) are labeled using a user-defined threshold, with higher thresholds (e.g., 0.5–0.75) labeling only highly probable relevant papers, and lower thresholds (e.g., 0.25-0.5) allowing more papers to be labeled as relevant. Artwork was created using Flaticon designs. Abbreviations: ROC receiver operating characteristic.
Nested cross-validated supervised machine learning strategies
The NCV methodologies implemented in AIM Review are inspired by NeuroMiner, a ML pattern recognition software developed to design, train and validate clinical diagnostic and prognostic models. NCV allows to obtain a reliable assessment of the models’ performance in unseen publications while simultaneously tuning hyperparameters of models (see supplementary methods in supplementary materials). Figure 4B illustrates the supervised learning pipeline in AIM Review. In short, a subset of screened titles and abstracts are used to train and optimize models to predict the relevance of publications using NCV. Data is split into folds within an outer cross-validation (CV2) for model evaluation and an inner cross-validation (CV1) for hyperparameter tuning. The dataset is split into N outer folds, where one fold serves as a validation set while the remaining N − 1 folds undergo inner cross-validation. Within CV1, the data is further split into M inner folds for training and testing, identifying optimal hyperparameters. Once these are found, models are retrained with the optimal parameters and are validated on the held-out outer fold. An ensemble prediction that aggregates results across all validation folds is generated to produce performance estimates. To enhance reliability, users can apply multiple permutations of inner and outer loops, averaging performance across iterations. This approach provides a more robust and unbiased performance estimate compared to conventional cross-validation, which lacks a dedicated loop for hyperparameter tuning and may lead to overfitting. Model performance is assessed via the receiver operating characteristic (ROC) curve generated using the aggregated validation predictions (predictions for publications when they serve as the validation sample). This enables users to set probability thresholds for classifying relevant publications in the unscreened subset, with the default threshold being 0.5. A higher threshold (e.g., 0.75) is more conservative, prioritizes specificity, reducing false positives but lowering recall. A lower threshold (e.g., 0.25) is less conservative, increases recall at the cost of specificity. Once a threshold is selected in the application, various performance metrics, including accuracy, BAC, sensitivity, specificity, precision, F1 score and P4 score (see metrics definition in supplementary materials) are provided, and the model predicts the relevance of the unscreened publications.
Ensemble strategies
AIM Review employs ensemble learning strategies when multiple text vectorization modalities are selected, allowing the combination of complementary textual representations to improve prediction robustness. Two ensemble approaches are supported: stacked generalization (default) and modality fusion.
In stacked generalization, an independent base classifier is trained for each selected vectorization modality. The predictions of these base models are then aggregated and used as input features for a meta-learner, which generates the final relevance prediction. This layered architecture enables the model to capture different semantic and lexical aspects of the data across modalities. Both base learners and the meta-learner are trained and evaluated within the NCV framework, ensuring consistent model selection and reliable performance estimation. Although stacking can enhance predictive performance, it increases computational demands due to the training of multiple models.
In contrast, modality fusion combines all selected vectorization modalities into a single feature space by concatenating their representations, upon which a single classifier is trained. This approach reduces model complexity and computational overhead, making it suitable when resources are limited or when the modalities provide complementary information that jointly capture diverse textual patterns. Figure S1 in the supplementary materials illustrates both ensemble strategies.
Case studies
We assessed the AIM Review pipelines using six systematic review case studies with varying search sizes and acceptance rates. Three case studies26,27,28 were accessed through professional networks from the fields of psychology/psychiatry. An additional three case studies29,30,31 were later accessed from Synergy32, a free and open dataset on study selection in systematic reviews, to evaluate the generalizability of AIM Review to other disciplinary fields, such as computer science, environmental health and endocrinology; the results of these analyses are provided in the supplementary materials. Case study 1 evaluated universal prevention strategies for affective and psychotic disorders (16,661 publications, 1.21% included for full-text screening)28. Case study 2 examined data-driven biological subtypes in schizophrenia and bipolar disorder (5536 publications, 7.03% included)27. Case study 3 identified studies using the Adult Attachment Interview (729 publications, 53.36% included)26. Case study 4 examined fault prediction performance in software engineering (8793 publications, 1.18% included)29. Case study 5 examined the effects of postmenopausal hormone replacement therapy (366 publications, 21.86% included)31. Case study 6 studied the transgenerational inheritance of health effects (48,375 publications, 1.57% included)30. The inclusion and exclusion criteria of each study are described in the supplementary materials. All tasks were performed using AIM Review on Google Chrome v136 in a PC equipped with an Intel i7-4770 processor and 32 GB RAM.
To ensure consistency in the evaluation of vectorization and classification methods in AIM Review, we maintained fixed configuration parameters for all three case studies across active learning and supervised learning pipelines. We utilized TF-IDF, LSA of TF-IDF, Doc2Vec, univBERT, all-MiniLM-L6-v2 and SPECTER2 as text vectorization methods, and LR, L-SVM and SeqNN as classifiers (SeqNN used only in supervised learning; see definitions and configuration parameters in the supplementary materials). Additionally, we also utilized stacked generalization (with a LR meta-model) and fusion of modalities to combine all vectorization methods. Within each case study, we also maintained the same NCV structure across all experiments to ensure fair and directly comparable performance evaluations. For case studies 4–6, conducted later to validate AIM Review in other disciplinary fields, we limited the evaluation to the combination of all vectorization methods.
For supervised learning, we used 20% of publications as our model training sample and 80% of publications as our OOCV sample. Within the training sample, a nested 5×5 cross-validation (CV2) framework with 5 outer permutations was used for model hyperparameter tuning. For LR and L-SVM, the regularization parameter C was optimized over the range [2-4, 24] to improve the BAC of predictions. The structure of SeqNN was optimized between three structures: a single 500-unit layer, two 50-unit layers, and two 100-unit layers. These neural network structures were selected to balance model complexity and overfitting risk, ranging from a large-capacity single-layer model to smaller, deeper architectures better suited for capturing non-linear patterns in data. Performance of the optimal model was assessed using the AUROC metric of the model applied to the ensembled validation CV2 sets (OOT) and on the 80% held-out set (OOCV). Additionally, BAC, sensitivity, specificity and P4 score are measured by applying a default threshold of 0.5. We did not use F1 scores as our model optimization metric since the score does not account for true negatives, and we aimed to optimize models with a robust ability to correctly reject irrelevant papers—reflecting a key functionality of the application in efficiently filtering out non-relevant literature.
For active learning, models were trained using all available screened publications and applied to the unscreened publications to predict a probability of relevance without optimization of parameters, as the goal was iterative low-latency ranking of records rather than identifying a globally optimal classifier. This pipeline was evaluated using the WSS metric, which quantifies the reduction in screening effort compared to random sampling while achieving a target recall level13. Specifically, we used the WWS that measures the reduction of workload while identifying 95% (WSS95%) and 100% (WSS100%) of the relevant records.
$${{\rm{WSS}}}_{{\rm{X}} \% }=\frac{{{\rm{N}}}_{{\rm{T}}}-{{\rm{N}}}_{{\rm{X}} \% {\rm{recall}}}}{{{\rm{N}}}_{{\rm{T}}}}\times 100$$
(1)
Where \({{\rm{N}}}_{{\rm{T}}}\) is the total number of publications in the case study and \({{\rm{N}}}_{{\rm{X}} \% {\rm{recall}}}\) is the number of publications that need to be reviewed to achieve X% recall. To obtain a reliable WSS, we simulated the title and abstract screening of all three case studies (n = 5 repetitions with shuffling) and sorted the publications by their relevance (highest relevance first) every five new publications were screened and assigned the correct label. The screening process was simulated by waiting 2 s and assigning the observed relevance of the publication. The 2-second delay simulated the time required for model retraining, relevance scoring, and sorting of publications—not human screening time, as labels were already known.
