Activation with Intrinsic-Extrinsic Consensus

Artificial Neural Networks (ANNs) are powerful tools for complex decision-making tasks. While existing activation mechanisms often promote sparsity through thresholding, they lack explicit awareness of feature channel relevance, causing networks to continuously suffer from interference by noisy channels. Such irrelevant activation signals can propagate through the network and adversely affect the final decision. Inspired by observations that channel relevance can be reflected in both intrinsic activity levels and extrinsic decision weights, and that there is strong consensus between these two aspects, we propose AIEC (Activation with Intrinsic-Extrinsic Consensus), a novel activation mechanism that has the ability to identify and suppress irrelevant feature channels during training. With a basic threshold activation, AIEC integrates an intrinsic Activation-Counting Unit that tracks channel activation statistics, an extrinsic Decision-Making Unit that learns channel decision weights, and a Consensus Gatekeeping Unit that suppresses irrelevant channels based on the agreement between intrinsic and extrinsic channel relevance assessments. Extensive experiments demonstrate that AIEC can effectively suppress irrelevant channels and encourage sparser representations. Furthermore, AIEC is compatible with a wide range of mainstream ANN architectures and achieves superior performance compared to existing activation mechanisms across multiple tasks and domains. Code is available at https://github.com/horrible-dong/AIEC.

Artificial neural networks rely on many information channels when making decisions. Some of these channels are useful, while others may be noisy or irrelevant, and keeping the irrelevant ones active may distract the network and reduce its reliability. We propose AIEC, a new way for neural networks to decide which information channels should stay active during learning. AIEC looks at each channel from two viewpoints: how often it becomes active inside the network, and how much it helps the network make final decisions. When these two viewpoints agree that a channel is not useful, AIEC suppresses it so that the network can focus more on the information that matters. The experiments show that this approach helps neural networks build cleaner and more compact representations of information. Furthermore, AIEC can be added to many common neural network designs and improves performance across different tasks and application areas.