Abstract
Abstract
Objectives
To produce the first structured longitudinal dataset of symptom severity trajectories in vagus nerve stimulation (VNS) patients using a 2-stage large language model pipeline for automated symptom severity extraction from clinical notes, as part of the NIH-funded research evaluating vagal excitations and anatomical linkages (U54AT012307).
Materials and methods
We analyzed 1427 annotated clinical notes from 95 patients across 5 corpora. LLaMA models (3.2-1B, 3.2-3B, and 3.3-70B) were fine-tuned to classify 13 symptoms as present, absent, or negated (stage 1), then assessed severity (indeterminate, mild, moderate, or severe; stage 2). Longitudinal trajectories were tracked quarterly in 23 VNS patients against a 10-patient surgical control cohort (NSQIP).
Results
Severity F1-scores ranged from 0.76 to 0.96, and symptom classification reached 0.99 for high-prevalence symptoms, with the validation corpus (VNS vs NSQIP) as the dominant factor (29.4% of variance; no significant source-corpus effect), supporting cross-domain generalizability. Macro F1-scores were lower than weighted F1-scores for smaller models and NSQIP; suicidal-ideation sensitivity was lower on NSQIP than VNS (0.80 vs 1.00, largest model), limiting generalization for psychiatric risk detection. Equity analyses were exploratory given small subgroups (gender precision 0.95). The longitudinal dataset revealed a median time to clinically meaningful seizure improvement of 0.5 quarters, with 50% responders, 25% non-responders, and 25% worsened. Post-surgical decline was absent in the unmatched NSQIP control cohort, consistent with, but not confirming, VNS specificity.
Discussion
To our knowledge, this is the first system producing structured symptom trajectories from free-text VNS documentation, supporting individualized over population-level monitoring.
Conclusion
This dataset supports future treatment-response prediction, personalized device programming, and a reproducible clinical natural langauge processing benchmark for neuromodulation research.