Leveraging large language models for accurate classification of liver lesions from MRI reports

Abstract Background & aims The rapid advancement of large language models (LLMs) has generated interest in their potential integration in clinical workflows. However, their effectiveness in interpreting complex (imaging) reports remains underexplored and has at times yielded suboptimal results. This study aims to assess the capability of state-of-the-art LLMs to classify liver lesions based solely on textual descriptions from MRI reports, challenging the models to interpret nuanced medical language and diagnostic criteria. Methods We evaluated multiple LLMs, including GPT-4o, Deepseek V3, Claude 3.5 Sonnet, and Gemini 2.0 Flash, on a physician-generated fictitious dataset of 88 MRI reports designed to resemble real clinical radiology documentation. The dataset included a representative spectrum of common liver lesions, such as hepatocellular carcinoma, cholangiocarcinoma, hemangiomas, metastases, and focal nodular hyperplasia. Model performance was assessed using micro and macro F1-scores benchmarked against ground truth labels. Results Claude 3.5 Sonnet demonstrated the highest diagnostic accuracy among the evaluated models, achieving a micro F1-score of 0.91, outperforming other LLMs in lesion classification. Conclusion These findings highlight the feasibility of LLMs for text-based diagnostic support, particularly in resource-limited or high-volume clinical settings. While LLMs show promise in medical diagnostics, further validation through prospective studies is necessary to ensure reliable clinical integration. The study emphasizes the importance of rigorous benchmarking to assess model performance comprehensively.