Abstract
Introduction and hypothesis
ChatGPT Health is a health-focused conversational artificial intelligence (AI) environment, but its performance in patient-oriented urogynecology has not been directly compared with standard ChatGPT using both validated quality assessment and objective guideline-based criteria. We hypothesized that ChatGPT Health would provide higher patient-facing response quality while maintaining comparable guideline concordance and safety.Methods
Twenty-four guideline-informed patient queries covering common urogynecological and lower urinary tract concerns were submitted independently to ChatGPT Plus and ChatGPT Health on 14 August 2026. Forty-eight responses were anonymized and independently evaluated by five blinded clinicians using the validated Quality Analysis of Medical Artificial Intelligence (QAMAI) instrument, a prespecified five-element guideline-concordance checklist, and a safety scale. Blinded head-to-head preference was assessed secondarily. Prompt-level paired comparisons used Wilcoxon signed-rank tests.Results
Median guideline concordance was 100% for ChatGPT Health and 98% for ChatGPT Plus (p = 0.298). ChatGPT Health had a higher blinded five-domain QAMAI subtotal (25.0 vs 24.2; p = 0.026) and greater clarity (p = 0.020), completeness (p = 0.016), and usefulness (p = 0.033), with all three remaining significant after correction for false discovery rate. Accuracy did not differ (p = 0.453). No clinically meaningful safety concerns were identified in 240 ratings. ChatGPT Health was preferred in 47.5% of 120 assessments, ChatGPT Plus in 18.3%, and no meaningful difference was reported in 34.2%.Conclusions
Both platforms generated highly accurate, guideline-concordant, and safe responses. ChatGPT Health showed advantages primarily in clarity, completeness, usefulness, and overall quality of patient-facing communication rather than objective concordance of clinical content.