Convergence between AI chatbot-calculated microstructure measures and SALT analysis of child language samples

This study examined the association and agreement between microstructure calculations from four large language model (LLM) chatbots and those generated by Systematic Analysis of Language Transcripts (SALT) software. Eighty-four typically developing children (35 girls; ages 2–9) participated, with 30 Japanese-English bilinguals and 54 English monolinguals. Narrative language samples were collected and coded with SALT conventions. Transcripts were randomly ordered and analyzed using two prompts. Using a general prompt, transcripts were entered into each LLM chatbot (ChatGPT, Copilot, MagicSchool, and Gemini), microstructure outputs were recorded, then the process was repeated with the same order using the more specific prompt. Results showed that across LLM chatbots, measures requiring relatively simple word or morpheme counts (MLUw, MLUm, NTW, NDW) generally exhibited stronger associations with SALT than the subordination index (SI), which showed greater variability. While intraclass correlation coefficients within measures were more variable, association and agreement coefficients were generally consistent with each other. Children's age and language backgrounds were associated with differences in the magnitude of some LLM–SALT coefficients. Finally, prompt specificity did not strengthen microstructure calculations for any of the LLM chatbots. Findings provide preliminary methodological evidence that some SALT-calculated microstructure measures may be more amenable to LLM chatbot-assisted calculation than others, with substantial associations and generally strong agreement observed for several measures; however, due to the exploratory nature of the current study, their clinical utility, decision-level accuracy, and workflow implications require further investigation.