Learning Context-Conditioned Predicate Semantics via Prototype Feedback

In scene graph generation, a central challenge is modeling polysemous predicates whose meanings shift across contexts. Prior approaches address this issue by decomposing predicates into multiple static prototypes or retrieving semantically similar exemplars. However, these strategies keep predicate representations static and cannot reorganize semantics to reflect image-specific evidence, leading to systematic confusions in ambiguous contexts. We propose AlignG, which learns context-conditioned predicate semantics via prototype feedback. AlignG infers context-conditioned predicate semantics from the relation candidates within each image and feeds the adapted semantics back to recalibrate relation representations. The learning objective anchors this adaptation to global semantic centers, preventing semantic drift while still allowing selective reorganization when the scene provides consistent relational cues. Experiments on VG-150 and GQA-200 show consistent improvements over state-of-the-art baselines, with F@100 improvements of +1.4 on VG-150 and +2.7 on GQA-200 under SGDet. We further visualize per-image prototype similarity shifts and observe coherent context-dependent reorganization where prototypes selectively merge or separate predicates according to scene evidence. The code is available at https://github.com/Namgyu97/AlignG-SGG.pytorch.

Picture two photos on a ski slope, one of a skier racing downhill and one of people standing still on their skis. To a computer the two photos look nearly identical, yet only in the first is anyone actually riding the skis. The trouble is that relationship words are slippery, and most methods fix their meaning in advance. We proposed AlignG, a method that lets these meanings bend to fit the evidence in each photo. Instead of deciding in advance what a word like "riding" means, AlignG first reads the whole scene to work out what each relationship word should mean given that specific evidence. It then feeds those adjusted meanings back to re-judge each relationship, while a built-in safeguard keeps them from drifting too far, so the method stays reliable. AlignG describes how objects relate more accurately than earlier methods. The same idea of keeping meaning stable yet letting it bend to context could help any tool that needs to describe what is happening in a photo or answer questions about it.