Abstract
Abstract
Existing image retrieval methods mainly rely on global visual features or object-level semantic information, making it difficult to sufficiently capture spatial arrangements and social interaction structures that are important in human-centric scenes. In particular, even when two images contain a similar number of people and comparable backgrounds, the scene meaning and relational structure can vary substantially if nonverbal cues such as interpersonal distance, gaze direction, body orientation, and gestures differ. To alleviate this limitation, this paper proposes a scene-graph-based human-centric image retrieval method that combines person layout and nonverbal social relation cues.The proposed method detects people from an input image and estimates body keypoints, after which each person is represented as a node in a scene graph. In addition, monocular depth estimation is used to compute a 2.5D person layout that includes not only 2D image-plane positions but also relative front-back relationships. Between pairs of people, relation edges are constructed by reflecting gaze and head direction, body orientation alignment, arm gestures, proximity, and in-scene importance, thereby explicitly representing the nonverbal communication structure among people. The generated scene graph is converted into a fixed-length descriptor, and the final matching score is computed during retrieval by combining descriptor similarity, person-layout similarity, social-relation similarity, and person-count consistency.Through this process, the proposed method can structurally compare how people are arranged and interact within a scene, rather than relying only on global visual similarities such as color, background, or object appearance. Through qualitative retrieval results and component removal analysis, we analyze how the proposed method reflects person layout and nonverbal relation cues in the retrieval results. In particular, by comparing ranking changes when the social relation component is removed, we confirm that nonverbal relation edges contribute to adjusting the final ranking among candidates that are difficult to distinguish using only global appearance or simple person layout. This study presents a structural image retrieval framework that uses the social relation structure of human-centric scenes as a retrieval criterion, enabling effective retrieval of images that differ in appearance but share similar person layout and interaction context.