Abstract
Objective: To develop and evaluate a framework for characterizing clinician editing of Artificial Intelligence (AI)-generated documentation and assess its feasibility for health system-level monitoring of AI scribes. Materials and Methods: We analyzed outpatient encounters in which an AI scribe was used at a single academic health system, examining the History of Present Illness (HPI) and Assessment and Plan (A&P) sections of notes. We characterized edits on three dimensions: lexical edit intensity based on Levenshtein distance, embedding edit intensity BERTScore, and clinical edit intensity based on removed and added UMLS concepts. We analyzed the Positive Predictive Value (PPV) of clinical edit intensity as a measure of clinically meaningful editing using clinicians as the gold standard, examined correlations among dimensions, and designed exponentially weighted moving-average control charts to monitor longitudinal changes in clinician editing behavior. Results: 268,379 encounters were included (267,654 HPI, 267,594 A&P). Clinical edit intensity ≥1 had an 88.9% PPV for clinically meaningful editing. Lexical and embedding edit intensity were highly correlated (Spearman rho 0.95), while clinical edit intensity was less strongly correlated with both (rho 0.77-0.81). Longitudinal monitoring detected changes coinciding with system-wide rollout. Discussion: Clinicians edited A&Ps more heavily than HPIs, potentially reflecting greater attention to content involving clinical decision-making. Over one-third of sections involved clinical concept changes, and clinical edit intensity identified clinically meaningful edits while providing information complementary to lexical editing measures. Conclusion: Clinician editing can be characterized at scale using complementary editing dimensions, providing a scalable signal for post-deployment surveillance of the human-AI documentation process