Abstract
Abstract
Background
Clinical pharmacists play an important role in managing pharmacotherapy safety. However, insufficient efficiency of manual drug consultation, heavy repetitive workloads, and delayed updating of professional knowledge commonly pose major challenges. Large language models (LLMs) have shown great capabilities in professional knowledge retrieval, text analysis, and intelligence-assisted decision making. Therefore, this study aimed to systematically evaluate the performance of LLMs (ChatGPT-5.5 Instant, Gemini-3.5 Flash-Lite and DeepSeek-V3) as a tool for clinical pharmacists in dental specialty hospital.
Methods
Thirty questions were curated from daily pharmacotherapy practice in stomatological hospital and classified into three levels, easy, moderate and difficult. The response quality of three LLMs was blindly and independently assessed by five senior clinical pharmacists by global quality scale (GQS). The scores were reported as median and interquartile range (IQR). Inter-model differences were assessed Kruskal–Wallis tests with Bonferroni-corrected post-hoc comparisons. The correlation between question difficulty level and response quality were evaluated by Fisher-Freeman-Halton exact test. The quality of information sources and readability of the responses were also evaluated.
Results
ChatGPT reached the highest response quality median score (4.30, IQR: 4.00-4.60), followed by DeepSeek (3.70, IQR: 3.25–4.20) and Gemini (3.40, IQR: 3.00-3.40). The difference between each LLMs reached statistical significance by Kruskal–Wallis tests with Bonferroni-corrected post-hoc comparisons. Fisher-Freeman-Halton exact test indicated that response quality was not significantly correlated with question difficulty level. And the quality of information sources and readability were not significant among three LLMs.
Conclusions
This study indicated that LLMs, especially ChatGPT, have the potential to serve as an assistive tool for clinical pharmacist in dental specialty hospital, with acceptable response quality across diverse pharmacotherapy domains. LLMs should be regarded as a promising assistive tool for clinical pharmacist rather than an independent decision maker. Further studies are still warranted since the rapid evolution and refinements of LLMs.