Timezone: »

 
Poster
End-to-end Active Object Tracking via Reinforcement Learning
Wenhan Luo · Peng Sun · Fangwei Zhong · Wei Liu · Tong Zhang · Yizhou Wang

Wed Jul 11 09:15 AM -- 12:00 PM (PDT) @ Hall B #64
in Posters Wed »

We study active object tracking, where a tracker takes as input the visual observation (\ie, frame sequence) and produces the camera control signal (\eg, move forward, turn left, \etc). Conventional methods tackle the tracking and the camera control separately, which is challenging to tune jointly. It also incurs many human efforts for labeling and many expensive trial-and-errors in real-world. To address these issues, we propose, in this paper, an end-to-end solution via deep reinforcement learning, where a ConvNet-LSTM function approximator is adopted for the direct frame-to-action prediction. We further propose an environment augmentation technique and a customized reward function, which are crucial for a successful training. The tracker trained in simulators (ViZDoom, Unreal Engine) shows good generalization in the case of unseen object moving path, unseen object appearance, unseen background, and distracting object. It can restore tracking when occasionally losing the target. With the experiments over the VOT dataset, we also find that the tracking ability, obtained solely from simulators, can potentially transfer to real-world scenarios.

[ PDF] 

Author Information

Wenhan Luo (Tencent AI Lab)
Peng Sun (Tencent AI Lab)
Fangwei Zhong (Peking University)
Wei Liu (Tencent AI Lab)
Tong Zhang (Tecent AI Lab)
Tong Zhang

Tong Zhang is a professor of Computer Science and Mathematics at the Hong Kong University of Science and Technology. His research interests are machine learning, big data and their applications. He obtained a BA in Mathematics and Computer Science from Cornell University, and a PhD in Computer Science from Stanford University. Before joining HKUST, Tong Zhang was a professor at Rutgers University, and worked previously at IBM, Yahoo as research scientists, Baidu as the director of Big Data Lab, and Tencent as the founding director of AI Lab. Tong Zhang was an ASA fellow and IMS fellow, and has served as the chair or area-chair in major machine learning conferences such as NIPS, ICML, and COLT, and has served as associate editors in top machine learning journals such as PAMI, JMLR, and Machine Learning Journal.

Yizhou Wang (Peking University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors