Learning Generalized Trackers with Elastic Token Budgets

Visual tracking aims to estimate target states in video sequences, with applications spanning diverse computational requirements. Recent methods optimize trackers using manually pruned image tokens with a fixed budget to reduce computational costs. However, these trackers, once trained, are constrained to perform tracking under a fixed computational budget, limiting their adaptability to real-world computational diversity. To address the above limitation, we provide the first exploration of the elastic token budget training framework (ETBTrack), enabling trackers to perform robust tracking under varying computational budgets. It enjoys several merits. First, we present a novel result-driven importance criteria, in which we optimize a policy network guided by the localization precision of the tracker to estimate token importance, thereby aligning the objectives of importance estimation and tracking precision. Second, we develop a new budget-collaborative optimization strategy, in which we collaboratively optimize the tracker across varying budgets, thereby enabling the tracker to be compatible with diverse budgets. Two optimization processes are performed alternately to enhance the capability of elastic inference. Extensive experiments on large-scale benchmarks demonstrate the effectiveness of our method.

Video tracking identifies objects in videos but often demands heavy computing power. Current methods reduce this cost by limiting data processing to a fixed amount, making them rigid and inefficient for devices with varying capabilities. We introduce ETBTrack, a new framework that allows trackers to flexibly adjust their computational usage. Instead of a fixed limit, our method dynamically selects the most important visual information based on tracking accuracy. It also trains the system to perform well across many different computing budgets simultaneously. This ensures robust performance whether running on powerful servers or limited mobile devices. By adapting to available resources, ETBTrack makes video tracking more versatile and efficient. This breakthrough helps deploy advanced tracking technology in diverse real-world applications, from smartphones to autonomous vehicles, without sacrificing accuracy.