Abstract
How does the human mind organize dynamic social scenes? Existing accounts typically rely on pre-defined social-affective or action-based taxonomies, yet whether humans use these dimensions to structure their representations remains largely unknown. We collected 53,223 odd-one-out triplet similarity judgments from 232 participants on 250 naturalistic dyadic-interaction videos and trained a sparse-positive embedding model to recover the underlying latent dimensions. We show the learned dimensions explain twice as much variance in similarity judgments as pre-labeled annotations, capture interpretable fine-grained social-event structure (combining information about social relationships with actions), and fractionate broad semantic content (e.g., parent-child interactions separate into caregiving versus affectionate play). The same dimensions predict fMRI responses in the visual cortex, including lateral visual pathway and superior temporal sulcus, better than pre-labeled annotations. Together, these results suggest that human representations of social scenes combine social relationships and actions in ways that pre-defined taxonomies miss, offering a behaviorally grounded target for future studies of social vision.