SegPVSG: Panoptic Video Scene Graph Generation via Temporal Focusing and Generative Augmentation

Panoptic Video Scene Graph Generation (PVSG) aims to identify relations between pixel-level entities in a video, serving as a novel paradigm for structured video parsing. However, this task faces two key challenges. First, the interactions between entities are temporally fragmented and sparse, meaning videos are dominated by irrelevant content with limited salient information. Second, the distribution of relations exhibits a significant long-tailed pattern, making models struggle to perform well on tail categories with insufficient data. To address these issues, we propose SegPVSG, an innovative, temporal-segment-aware PVSG framework consisting of two key components: TempFocusNet (TFN) and Relation-centric Generative Video Augmentation (RGVA) module. TFN is a localization-then-recognition network that improves PVSG performance by explicitly localizing and focusing on salient segments before relation recognition. Meanwhile, RGVA is a novel augmentation module that generates realistic, context-consistent video segments for rare relations and coherently inserts them into original videos. Our method outperforms prior methods by +3.53 mR@20 and +5.9 mR@50, demonstrating its effectiveness. The code is available at https://github.com/ticatt/SegPVSG.

Videos are full of objects and actions, but the key interactions often happen only briefly. For example, a person may pick up a bottle for a few seconds while most of the video is unrelated. This paper studies how to identify who interacts with what, what the interaction is, and when it happens in a video. The challenge is that useful moments are sparse, and rare interactions have very few training examples. We propose SegPVSG, a method that first finds likely interaction segments and then focuses recognition on those moments. We also generate realistic short clips for rare interactions and insert them into training videos to provide more examples. This helps the model learn both common and uncommon video relations more reliably. Experiments show that SegPVSG improves video relation understanding, especially for rare relation types.