Development and psychometric evaluation of the AI emotional engagement scale

Background With the increasing integration of artificial intelligence (AI) technologies into daily life, the underlying mechanisms of emotional interactions between users and AI have emerged as a critical research topic in the field of human-computer interaction (HCI). Existing studies have predominantly adopted traditional technology acceptance models or satisfaction scales, which lack targeted applicability in capturing unique features of AI interactions, such as parasocial relationships and emotional dependence. The present study aims to develop a self-report instrument, the AI Emotional Engagement Scale (AEES), and examine its psychometric properties for assessing users’ level of AI Emotional Engagement with AI systems. Methods This study was conducted based on Norman’s three-level theory of emotional design. Study 1 (N = 24) extracted core information from interview data using qualitative analysis, and generated an initial item pool through multiple rounds of expert review. Study 2 (N = 103) conducted a pre-test of the initial items to examine indicators including item quality and relevance, optimized the item design, and developed the formal item set. Study 3 (N = 924) verified the factor structure of the AEES via exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), conducted reliability and validity tests of the scale, and finalized the factor structure of the 28-item, three-dimensional scale. Results Results of EFA indicated that AI Emotional Engagement consists of three dimensions: the visceral level, behavioral level, and reflective level, which was consistent with the theoretical framework, with a cumulative variance contribution rate of 68.32%. Results of CFA demonstrated that the three-factor model had a good fit ( χ²/df  = 3.765, CFI = 0.91, TLI = 0.90, RMSEA = 0.077, SRMR = 0.0496). The overall AEES and its three dimensions exhibited good internal consistency reliability, with reliability coefficients ranging from 0.90 to 0.97. Conclusion The AEES has favorable psychometric properties, and can serve as a reliable instrument for researchers and practitioners to assess users’ AI Emotional Engagement with AI systems.