Timezone: »

 
Poster
Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration
Jing Lin · Xiaowan Hu · Yuanhao Cai · Haoqian Wang · Youliang Yan · Xueyi Zou · Yulun Zhang · Luc Van Gool

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #104
in Poster Session 2 »

How to properly model the inter-frame relation within the video sequence is an important but unsolved challenge for video restoration (VR). In this work, we propose an unsupervised flow-aligned sequence-to-sequence model (S2SVR) to address this problem. On the one hand, the sequence-to-sequence model, which has proven capable of sequence modeling in the field of natural language processing, is explored for the first time in VR. Optimized serialization modeling shows potential in capturing long-range dependencies among frames. On the other hand, we equip the sequence-to-sequence model with an unsupervised optical flow estimator to maximize its potential. The flow estimator is trained with our proposed unsupervised distillation loss, which can alleviate the data discrepancy and inaccurate degraded optical flow issues of previous flow-based methods. With reliable optical flow, we can establish accurate correspondence among multiple frames, narrowing the domain difference between 1D language and 2D misaligned frames and improving the potential of the sequence-to-sequence model. S2SVR shows superior performance in multiple VR tasks, including video deblurring, video super-resolution, and compressed video quality enhancement. https://github.com/linjing7/VR-Baseline

Keywords: APP: Computer Vision · DL: Generative Models and Autoencoders
[ Slides]  [ Poster]  [ Paper PDF] 

Author Information

Jing Lin (Tsinghua Univisity, Tsinghua Shenzhen International Graduate School)
Xiaowan Hu (Tsinghua Univisity, Tsinghua Shenzhen International Graduate School)
Yuanhao Cai (Tsinghua Univisity, Tsinghua Shenzhen International Graduate School)
Haoqian Wang (Tsinghua Shenzhen International Graduate School, Tsinghua University)
Youliang Yan (Huawei Noah's Ark Lab)
Xueyi Zou (Huawei Noah's Ark Lab)
Yulun Zhang (ETH Zurich)

I am a postdoctoral researcher at Computer Vision Lab, ETH Zürich, Switzerland, working with Prof. Luc Van Gool. Previously, I obtained my PhD degree at Department of Electrical & Computer Engineering, Northeastern University, USA, in Aug. 2021. Before that I received my master degree in the Department of Automation, Tsinghua University, China, in Jul. 2017 and B.E degree from School of Electronic Engineering, Xidian University, China, in Jul. 2013. My research interest broadly includes machine learning and computer vision. Specifically, I focus on image/video restoration (e.g., super-resolution, denoising, deblurring), synthesis (e.g., style transfer, texture transfer), biomedical image enhancement and analysis,deep model compression, computational imaging (e.g., spectral compressive imaging), etc.

Luc Van Gool (ETH Zurich)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors