Abstract
Abstract
Objective
Secondary headaches require urgent recognition due to potentially devastating consequences if untreated. Despite established clinical “red flag” criteria, identifying patients needing immediate evaluation remains challenging in primary care. This study developed and evaluated a large language model (LLM)–based multi-agent clinical decision support system for interpretable secondary headache diagnosis.
Materials and Methods
We first established 7 clinically relevant secondary headache red flag domains through manual review and synthesis of clinical guidelines. Based on these domains, we designed an LLM-based system using an orchestrator–specialist multi-agent architecture that decomposes diagnostic reasoning into 7 guideline-aligned agents corresponding to key red flag features. Each agent generates structured, evidence-grounded reasoning, coordinated by a central orchestrator. The system was evaluated on 90 expert-validated secondary headache cases and compared with a single-LLM baseline under 2 prompting strategies: question-based prompting (QPrompt) and guideline-based prompting (GPrompt). Five open-source LLMs (Qwen-8b, Qwen-14b, Qwen-30b, GPT-OSS-20b, and Llama-3.1-8b) were tested.
Results
The orchestrated multi-agent system with GPrompt achieved the highest red flag classification performance across models, measured by F1 score. Performance gains were consistent and more pronounced in smaller LLMs, suggesting that structured reasoning improves efficiency and accuracy beyond prompt engineering alone. The framework also produced transparent and guideline-aligned intermediate reasoning.
Discussion
Decomposing clinical reasoning into specialized agents enhances interpretability and diagnostic reliability compared with monolithic LLM approaches. Multi-agent orchestration provides a clinically aligned framework for explainable decision support.
Conclusion
An orchestrator–specialist multi-agent LLM framework improves secondary headache diagnosis accuracy and transparency, supporting the development of explainable AI systems for time-constrained clinical decision-making in primary care.