Timezone: »

 
Poster
Channel Equilibrium Networks for Learning Deep Representation
Wenqi Shao · Shitao Tang · Xingang Pan · Ping Tan · Xiaogang Wang · Ping Luo

Wed Jul 15 05:00 AM -- 05:45 AM & Wed Jul 15 07:00 PM -- 07:45 PM (PDT) @
in Poster Session 21 »

Convolutional Neural Networks (CNNs) are typically constructed by stacking multiple building blocks, each of which contains a normalization layer such as batch normalization (BN) and a rectified linear function such as ReLU. However, this work shows that the combination of normalization and rectified linear function leads to inhibited channels, which have small magnitude and contribute little to the learned feature representation, impeding the generalization ability of CNNs. Unlike prior arts that simply removed the inhibited channels, we propose to ``wake them up'' during training by designing a novel neural building block, termed Channel Equilibrium (CE) block, which enables channels at the same layer to contribute equally to the learned representation. We show that CE is able to prevent inhibited channels both empirically and theoretically. CE has several appealing benefits. (1) It can be integrated into many advanced CNN architectures such as ResNet and MobileNet, outperforming their original networks. (2) CE has an interesting connection with the Nash Equilibrium, a well-known solution of a non-cooperative game. (3) Extensive experiments show that CE achieves state-of-the-art performance on various challenging benchmarks such as ImageNet and COCO.

Keywords: Architectures · Computer Vision · Networks and Relational Learning · Deep Learning - General

Author Information

Wenqi Shao (The Chinese University of HongKong)
Shitao Tang (Simon Fraser University)
Xingang Pan (The Chinese University of Hong Kong)
Ping Tan (Simon Fraser University)
Xiaogang Wang (Chinese University of Hong Kong, Hong Kong)
Ping Luo (The University of Hong Kong)

More from the Same Authors