Abstract
Abstract
Background
Ambient AI scribes now entering clinical practice depend on speech recognition that transmits patient audio to cloud services, and no open benchmark exists for clinical conversation transcription. We built an open, reproducible benchmark and used it to characterize clinically consequential failure modes across deployment settings, and secondarily to determine whether on-device models reach cloud-level clinical term accuracy and whether multi-model fusion narrows any residual gap.
Methods
We evaluated 12 on-device ASR models and 5 cloud APIs on 400 simulated conversations from three public datasets (OSCE respiratory, n = 272; PriMock57 primary care, n = 57; Kazi et al. psychiatric, n = 71). WER was reported in standard (Whisper normalizer) and meaning-preserving regimes. Clinical term recall (CTR) used UMLS 2025AB spans via QuickUMLS with BCa bootstrap 95% CIs. ROVER and six advanced fusion algorithms were evaluated.
Results
Medication names were the highest-error clinical term category for most systems in both deployment modes; the highest-error drugs were misrecognized in 85% to 93% of on-device occurrences versus 3% to 30% in the cloud. A medical-dictation model cut wellbutrin errors to 5%, showing this deficit is addressable through domain adaptation. On-device clinical term recall was within 1–3 pp of the best cloud service per dataset; the standard-WER gap was 0.5–5.2 pp, narrowing to 0.7–3.3 pp under meaning-preserving normalization. ROVER gained ≤ 0.82 pp; no advanced algorithm exceeded naive voting by more than 0.4 pp.
Conclusions
Medication errors reflect look-alike/sound-alike confusability, and their persistence alongside near-parity recall shows that aggregate accuracy can mask clinically critical failure modes. Because this failure is independent of where transcription runs, any clinical ASR deployment requires explicit medication verification before note finalization. Within that constraint, on-device ASR is a clinically viable, lower-cost alternative that keeps patient audio local, and a single model suffices; fusion did not close the small residual gap.