Hamiltonian Asymmetric Fusion: One-Way Safe Directed Refinement under Modality Imbalance

In RGB--D salient object detection, multimodal fusion is commonly implemented via symmetric token interaction, implicitly allowing information to flow in both directions. Under modality imbalance---when an auxiliary stream is substantially noisier than a designated primary stream---such symmetry creates a backflow channel that injects auxiliary noise into the primary representation and amplifies errors across iterative refinement stages. We formulate fusion in this regime as directed refinement with one-way safety: the primary modality defines a guidance field, while only auxiliary representations are iteratively purified, and primary perturbations induced by the auxiliary stream are explicitly bounded. We propose Hamiltonian Asymmetric Fusion (HAF), a lightweight unrolled refinement block that updates auxiliary tokens with momentum regularization and gated driving. The refinement force is instantiated by FFT-based spectral global correlation and modulated by a shared learnable spectral response to emphasize reliable frequency components with minimal parameters; a leaky momentum gate and a stable integrator improve multi-step refinement stability. We provide guarantees of auxiliary error contraction and bounded primary perturbation, which symmetric fusion operators do not satisfy under imbalance. Experiments on six RGB--D SOD benchmarks show consistent gains and substantially more graceful degradation under controlled auxiliary corruption.

Many systems use both color images and depth maps to find the most important object in a scene. But these two sources are not always equally trustworthy: color images can be confused by clutter or lighting, while depth maps often provide more stable shape information. Current fusion methods usually mix information in both directions, which can let noise from the weaker source damage the stronger one. We introduce Hamiltonian Asymmetric Fusion, a method that combines the two sources more carefully. It uses the more reliable source as guidance and gradually cleans up the noisier source, while limiting how much noise can flow back. This makes the fusion process more stable over multiple refinement steps. Our work helps multimodal vision systems remain accurate when one input is unreliable. Experiments on six RGB-D salient object detection benchmarks show better performance and stronger robustness under corrupted inputs.