Sampling bias caused by the structure of bacterial populations dramatically impacts the ability of machine learning (ML) to predict antimicrobial resistance (AMR), researchers warn.
The survey results are PLOS Biologyhighlights the importance of methods that account for this population structure and use more diverse datasets to improve prediction and monitoring of AMR.
“Addressing the interaction between population structure and AMR prediction will require a multifaceted approach that includes improved sampling, algorithmic innovation, and systematic evaluation of proposed prediction methods,” the researchers, led by Dr. Yanying Yu of Harvard Medical School, recommend.
“Only by facing these challenges can we unlock the full potential of machine learning to provide actionable insights into AMR and advance both surveillance capabilities and our understanding of resistance mechanisms.
“These efforts are critical to combating the global threat of AMR and ensuring the continued effectiveness of lifesaving antimicrobial therapies.”
AMR poses a serious threat and kills nearly 5 million people each year. ML techniques have emerged as a promising detection tool to uncover the determinants of resistance from genomic data.
However, most classical ML methods assume that the training data is independent and identically distributed. This is not the case for pathogen surveillance samples due to the underlying structure of the bacterial population.
During an epidemic, successful clones spread rapidly, but if part of this spread is due to acquisition of AMR determinants, this may result in associations between phenotypes and lineage markers that do not directly contribute to AMR.
These non-causal associations are further exacerbated by biased sampling focused on human diseases in high-income countries, which can leave large parts of the phylogeny unexplored.
By constructing real-world pathological scenarios, the researchers comprehensively evaluated the influence of bacterial population structure in predicting AMR.
They collected between 3,204 and 7,188 genomes of three Gram-negative and two Gram-positive species that represent current WHO priority pathogens.
These included gastrointestinal and urinary tract pathogens. Escherichia coli;opportunistic pathogens Klebsiella pneumoniae;Gastrointestinal pathogens salmonella enterica; commensal and opportunistic pathogens of the skin Staphylococcus aureus;Main pathogens of community-acquired pneumonia pneumococcus.
This dataset included a total of 27 antibiotic resistance phenotypes spanning multiple drug classes and diverse sequence types.
To limit the effects of sample size and class imbalance, the researchers excluded antibiotic-microbe combinations with less than 1,000 genomes or with more than 80% of resistant or susceptible strains in the dataset.
The average number of genomes was approximately 2,700, with approximately 44% resistant strains, and 80% of the data were 1,134 to 3,955 genomes, with 25% to 71% resistant.
Whole-genome alignments were constructed for each species, and distinct clades were identified based on deep divergence between branches of the phylogenetic tree.
The results showed that even in the presence of large training datasets, ML models continue to confuse lineage markers with true resistance indicators.
“Our results demonstrate that current ML approaches are particularly vulnerable to biased sampling and the confounding effects of population structure,” the researchers reported. “Even as sample size increases, the model struggles to generalize beyond the specific clades included in the training set, highlighting the limitations of scaling as a solution to bias.”
Their machine learning framework used a combination of single nucleotide polymorphisms in the core genome and presence-absence patterns of accessory genes to capture broad genomic differences beyond the strict core genome, but still failed to capture sequence variation within accessory genes.
“This limitation is particularly relevant for species with large and diverse accessory genomes, for example. Escherichia coli and Klebsiella pneumoniaeHere, mutations within genes may convey additional predictive signals for AMR,” the authors noted.
The researchers concluded that “expanding surveillance to underrepresented regions and settings, particularly low- and middle-income countries (LMICs), would provide a more balanced dataset and reduce bias toward high-income countries and outbreak scenarios.”
“Targeted sampling that prioritizes diversity both within and between clades is important for developing models that can be generalized across phylogenetic contexts.
“Beyond sampling, careful experimental design, including the use of balanced test sets and rigorous evaluation metrics such as those employed here, further ensures that the model is robust to confounding effects.”
