Abstract
In human social cognition, understanding others' attention relies not only on explicit cues such as gaze and posture but also on observers' inferences of implicit factors, including emotions, task goals, and social roles. This belief-based ability, known as second-order attention modeling, is a key component of theory of mind (ToM), enabling individuals to infer others' attentional states during multiagent interactions. Existing vision systems mainly focus on first-order attention modeling, such as gaze estimation and gaze following, which predict physical gaze locations. Although effective at the perceptual level, these methods cannot capture belief formation or account for modulatory factors such as emotions, limiting their applicability to higher-level social cognition. To address this limitation, we propose a belief-driven framework for second-order attention modeling, formulating observer inference under uncertainty as conditional probability estimation and integrating emotional cues, task semantics, and social roles. The framework extends attention modeling from gaze perception to emotion-modulated belief reasoning, enabling prediction of observer-reported attentional beliefs from visual, visually inferred affective, and social-context cues when direct gaze evidence is limited or ambiguous. We further introduce a large-scale dataset with diverse agents, tasks, and emotional contexts, whose observer-reported labels are validated through test-retest analysis, together with an emotion-conditioned evaluation metric for social-cognitive assessment. Experiments demonstrate that the proposed framework outperforms existing methods and produces predictions consistent with observer reports, showing effectiveness for emotion-aware second-order attention modeling.