Abstract
language processing (NLP) and generative artificial intelligence.Our objective is to assess the system's agreement against human grading and analyse how response length influences grading concordance, addressing a gap in medical education assessment.We obtained 840 responses from 28 radiology residents who completed a test comprising 30 X-ray quiz questions.The system employed string-matching algorithms and rule-based logic for anatomical and laterality mismatches.A ChatGPT-created synonym dictionary was used to recognise acceptable alternative answers.Cohen's kappa statistics assessed inter-rater agreement across answer lengths, and logistic regression examined response length's relationship with grading disagreement.The system achieved 97.5% raw agreement (819/840) with the human grader.Perfect agreement (κ = 1.00) was observed for one-word responses (n=429), with near-perfect agreement for two-word responses (κ = 0.927, n=75).Agreement generally decreased as response length increased, with each additional word increasing the odds of disagreement by 47.5% (OR = 1.475, 95% CI: 1.220-1.783,p < 0.001).The automated grading system, enhanced by NLP techniques and ChatGPT-assisted synonym expansion, demonstrated strong concordance with expert human grading, particularly for one-and two-word answers.This hybrid approach offers a scalable, transparent solution for short-answer assessment, with strong potential in radiology education and other medical education contexts.