Abstract
Abstract
Background
Large language models (LLMs) configured as virtual patients may expand structured practice of clinical reasoning in health professions education. Before testing effectiveness, it must be established whether students engage with them and whether the resulting scores are complete, reliable, and related to performance with real patients. Our earlier pilot randomized trial found very low engagement with a less structured LLM-based approach.
Objective
This study aimed to assess the feasibility (adherence, acceptability, and data completeness) of a revised, structured LLM-based clinical reasoning laboratory (Laboratorio de Inteligencia Artificial [LIA]) in undergraduate physical therapy education and to explore its measurement properties, including its association with clinical judgment with a real patient assessed with the Lasater Clinical Judgment Rubric (LCJR).
Methods
In this single-arm pilot feasibility study, 27 third-year physical therapy students at a Spanish university completed, in 2024-2025, 3 virtual patient cases (lumbar, cervical, and thoracic spine pain) simulated by ChatGPT (GPT-4) over 4 weeks, each twice. ChatGPT scored each attempt from 0 to 4 on the same 10 dimensions of the Lasater Clinical Judgment Rubric (LCJR; 0-40 points) that human assessors applied (1-4 per dimension; 10-40 points) during a real patient encounter. Feasibility outcomes were case completion, data completeness, and satisfaction (0-10). Measurement outcomes were internal consistency and convergent validity (Spearman correlations between model-generated and human LCJR scores). Variability estimates informed the sample size of a future trial.
Results
All 27 students were assessed with the LCJR; 22 (81%) completed at least 1 LIA case. Both attempts of the lumbar, cervical, and thoracic cases were completed by 78%, 63%, and 74% of students, respectively. Satisfaction data were available for 15 students (56%; mean 6.0, SD 1.3), and missing secondary data clustered in the third rotation. Internal consistency was excellent for the LCJR (α=.96) and ranged from .43 to .90 for model-generated ratings. Model-generated LIA scores clustered near the maximum (mean 36.0, SD 1.4). Among completers, LIA scores correlated weakly with the LCJR (ρ=0.25, 95% CI –0.20 to 0.61; ρ=0.44, 95% CI 0.05-0.71, when noncompleted attempts were scored as 0). Grade point average correlated almost perfectly with the LCJR (ρ=0.96; n=22); after adjustment for it, the LIA-LCJR association disappeared (partial ρ=–0.10). A future trial would require approximately 39 students per group (80% power, conservative variance).
Conclusions
Delivering the LIA within the curriculum was feasible, but clinical judgment ratings generated by ChatGPT did not reflect those of human assessors with a real patient: they clustered near the maximum and showed no convergent validity once noncompletion was separated from performance. LLM-generated ratings should not be used as outcome or assessment measures without validation against human raters. Case calibration, completion tracking, and clarification of the GPA-LCJR relationship are also needed before a definitive trial.
Trial Registration
ClinicalTrials.gov
NCT06809634
;
https://clinicaltrials.gov/study/NCT06809634