HSGG: Training-Free Hierarchical Scene Graph Generation with Geometry-Guided Relation Reasoning

Scene Graph Generation (SGG) connects visual perception with structured reasoning, but is limited by scarce annotations and the long-tailed distribution of relational predicates. Training-free methods based on vision-language models (VLMs) reduce supervision requirements, yet often rely on flat graph structures and produce hallucinated relations. We present HSGG, a training-free framework for open-world Hierarchical Scene Graph Generation, whose inference proceeds in two steps. First, bidirectional hierarchical entity perception combines top-down object expansion with bottom-up attribute reasoning to construct multi-level scene hierarchies capturing part–whole semantics. Second, geometry-guided relation reasoning infers valid relations from these structured entities: geometry-aware relation filtering first prunes spatially implausible object pairs using 2D proximity, depth cues, and object scale, and geometry-grounded contrastive relation decoding then suppresses hallucinated predicates by contrasting predictions from a visually grounded expert against a hallucination-prone geometric prior, ensuring relations are both geometrically consistent and semantically coherent. Experiments show that HSGG generalizes effectively to unseen objects and predicates without training, substantially reduces relational hallucinations, and consistently improves downstream reasoning performance.

Scene graph generation helps computers describe not only the objects in an image, but also how those objects are related. This is important for deeper image understanding, but existing methods often need many labeled examples and can make errors when predicting rare or unfamiliar relationships. We present HSGG, a training-free method for building hierarchical scene graphs in open-world settings. Our approach first organizes a scene into multiple levels, including objects, their parts, and their attributes. It then uses simple geometric information, such as distance, depth, and size, to rule out unlikely relationships and reduce hallucinated predictions. As a result, HSGG can better handle unseen objects and relationships while producing more accurate and reliable scene descriptions.