Abstract
Detecting careless responding in experience sampling methodology (ESM) is essential for ensuring data quality. Although quantitative detection methods are increasingly available, their low alignment raises questions about what they capture. Evaluation is constrained by the lack of ground truth in real-world data, while simulations require assumptions about the form of careless responding. This methodological catch-22 leaves disagreements among methods unresolved. We demonstrate how qualitative information about response processes (i.e., cognitive processes involved in rating items) can triangulate quantitative classifications and provide insight into what different methods capture. Using 4,175 measurement occasions from 41 participants in a 28-day ESM study, we compared a measurement-model-based mixture model using five emotion items and an indicator-based mixture model using four careless responding indices within a mixture continuous-time latent Markov framework. The models classified 35% and 15% of occasions as careless, respectively, with moderate overlap (Jaccard similarity = .37). Across models, careless occasions had more coded response processes and longer descriptions than attentive occasions. A qualitative analysis of 350 occasions showed how inductively derived classifications and response behavior codes provide complementary information about convergence and divergence between quantitative approaches. Overall, our findings demonstrate the potential value of triangulation for understanding what careless responding detection methods capture.Keywords: experience sampling methodology, careless responding, response processes, qualitative triangulation, latent Markov model