Abstract
Abstract
Background
Large language models (LLMs) are increasingly used to generate health education materials, yet questions remain about whether LLM-generated content can balance professional accuracy with public accessibility and what ethical challenges may arise during deployment.
Objective
This study aimed to evaluate Chinese pediatric asthma educational materials generated using LLMs, comparing AI-generated responses (AI-GRs) against published expert-authored responses (PEARs) on total scores, adoption intent among health care professionals, perceived usefulness among family members, and the effect of AI-simplified responses (AI-SRs), as well as to assess participants’ ability to correctly identify the source of materials.
Methods
In this randomized, double-blind evaluation study, participants (medical professionals and patient family members) were randomly assigned (1:1:1) to evaluate PEARs, AI-GRs, or AI-SRs. Each participant assessed 5 randomly selected items from their assigned set using a 19-item, 5-point Likert scale based on the information adoption model. Secondary outcomes included dimension-specific scores, source identification accuracy, and readability indices.
Results
AI-GRs had numerically higher mean total scores on the 19-item questionnaire than PEARs among both medical professionals (76.17, SD 12.59 vs 72.97, SD 14.88; Holm-adjusted
P
=.53) and pediatric patient families (68.77, SD 16.47 vs 64.18, SD 15.76; Holm-adjusted
P
=.15). The corresponding mean scores for AI-simplified responses were 72.98 (SD 12.64) and 65.47 (SD 16.52), respectively, with no statistically significant differences from PEARs after Holm adjustment (both adjusted
P
>.99). No statistically significant differences were detected between either LLM group and PEARs in any of the 4 questionnaire dimensions after Holm adjustment. The material group–population interaction was also not statistically significant (
F
2,513
=0.116;
P
=.89; partial η
2
Conclusions
This randomized, double-blind evaluation study provides preliminary participant-level evidence regarding the perceived quality and acceptance of LLM-assisted pediatric asthma education among medical professionals and pediatric patient families under controlled conditions. Additional language simplification did not improve acceptance in this predominantly highly educated sample, and the strong tendency to attribute materials to human authors highlights the importance of source transparency. Professional review and dedicated assessments of factual accuracy, clinical safety, comprehension, and behavioral outcomes remain necessary before practical implementation.