Beyond Detection: A Structure-Aware Framework for Scene Text Tracking

Modern visual object trackers show impressive results on general targets, yet their performance drops substantially when dealing with scene text. Although currently underexplored, tracking text in videos is essential for dynamic text manipulations such as segmentation, removal, and editing. To fill this gap, this paper formalizes this specific task as Scene Text Tracking and presents the first systematic work for it. We identify three primary challenges in this task: 1) severe geometric distortions from perspective shifts, 2) high visual ambiguity across different instances, and 3) high sensitivity to fine-grained structural details. To address these issues, we propose SymTrack, a unified detection-free framework with synergistic dual-branch design. It integrates a Cross-Expert Calibration mechanism to reduce semantic bias, along with a Predictive Token Rectification mechanism to correct structural imbalances, complemented by an Adaptive Inference Engine that stabilizes predictions under motion constraints. Considering the lack of dedicated benchmarks for this task, we utilize three datasets from video text spotting to construct a benchmark with high-quality annotations. Extensive experiments demonstrate that SymTrack sets the new state-of-the-art on all three benchmarks, outperforming previous best trackers by up to 11.97\% AUC on $ \text{BOVText}_{\text{SOT}} $. Overall, our work promotes efficient and thorough text tracking, paving the way toward more generalized video text manipulation.

Text often appears in videos, such as signs, subtitles, storefronts, and labels. Accurately following a specific piece of text across video frames is important for applications such as text editing, removal, and other forms of video understanding. However, this problem has received much less attention than recognizing text in individual images or detecting all text in a video. In this work, we define this problem as Scene Text Tracking and study it systematically. We show that tracking text is especially difficult because text can become distorted as the camera viewpoint changes, nearby words may look very similar, and even small tracking errors can affect the perceived characters. To address these challenges, we develop SymTrack, a new framework that directly follows a chosen text instance through a video without relying on repeated text detection in every frame. We also build three evaluation benchmarks from existing video text datasets so that future methods can be compared fairly. Experiments show that SymTrack consistently performs better than previous tracking and video text methods on all three benchmarks. Our study provides both a new research task and a strong solution for more reliable text tracking in videos.