Abstract
When people watch everyday activity or movies, the eyes are drawn by visually salient features, and also by features that are semantically meaningful. However, saliency and meaning often co-occur, making it difficult to isolate their individual influence on gaze. Here we leveraged a large language-image model (CLIP: Contrastive Language-Image Pretraining) to isolate these influences. Eighty-one participants watched four movies of everyday activities and eye movements were recorded. CLIP embeddings were used as a metric to measure semantic content and were compared with the gaze maps. The embeddings were generated for both upright and inverted frame versions, because frame inversion selectively disrupts semantic information while preserving saliency. Upright CLIP embeddings predicted viewers’ gaze patterns better than inverted embeddings, and this advantage persisted even after accounting for low-level saliency and other random-noise controls. Finally, temporally aligned action descriptions predicted gaze better than shuffled descriptions across multiple text-based representations, suggesting that action meaning itself carries information about where viewers look. Together, these findings suggest that gaze during naturalistic movie viewing is sensitive to semantic information at fine temporal scales, and that computational vision-language models provide a useful approach for studying cognitive processes in naturalistic environments.