Abstract
Background. Genomic machine-learning models for antimicrobial resistance (AMR) prediction are usually evaluated by discrimination alone, even though a model’s usefulness depends on whether its predicted probabilities are also numerically reliable (well calibrated) outside the population it was trained on. Public AMR genome collections are dominated by high-income countries, so it is unclear whether trained models keep both properties when applied to less-represented regions such as South Asia. Methods. We assembled 7,274 quality-controlled Escherichia coli genomes with laboratory-confirmed ciprofloxacin susceptibility testing from the Bacterial and Viral Bioinformatics Resource Center (BVBRC). An eight-feature elastic-net logistic regression model, trained on 2,574 United Kingdom genomes (28.4% resistant), was evaluated by five-fold cross-validation in-domain and, without refitting, on two external populations: Norway (n = 3,153; 9.9% resistant) and a pooled, illustrative South Asian population (India, Pakistan, and Bangladesh; n = 238; 66.8% resistant). Discrimination (AUROC, AUPRC, sensitivity, specificity, MCC) and calibration (slope, intercept, Brier score, expected calibration error) were assessed separately, with bootstrap confidence intervals. Local recalibration was tested across a range of target-population sample sizes. Results. Discrimination was high and largely preserved outside the UK (AUROC 0.992 in-domain, 0.968 in Norway, 0.970 in South Asia). Calibration was not preserved in the full cohort (Norwegian calibration slope 0.726, 95% CI 0.677–0.796), but the slope was 1.003 (95% CI 0.814–5.303) within the 2007–2012 window shared by all three populations, so the shift cannot be attributed to geography alone. Calibration error concentrated among resistant genomes lacking the model’s dominant feature, a chromosomal parC mutation. A single-feature (parC-only) model reproduced the same shift, indicating that the shift does not depend on the other seven features. Local recalibration moved the calibration intercept toward zero as target-population sample size grew, but the unpenalized recalibration fit produced extreme estimates at the smallest recalibration sample sizes and, in South Asia, at the largest, when few genomes remained for held-out evaluation; a ridge penalty removed nearly all of these extreme estimates. Recalibration was evaluated only by calibration intercept and Brier score, in unpaired before-and-after comparisons. Collection period and, in South Asia, study composition were confounded with geography and could not be fully separated from the observed shift. Conclusions. Strong discrimination in a genomic AMR model does not guarantee that its predicted probabilities transfer to a new population. Calibration checks, mechanism-aware error analysis, and attention to database annotation quality should accompany discrimination metrics when genomic AMR models are evaluated for use outside their training population.