Hamiltonian Asymmetric Fusion: One-Way Safe Directed Refinement under Modality Imbalance

Multimodal fusion is commonly implemented via symmetric token interaction, implicitly allowing information to flow in both directions. Under modality imbalance---when an auxiliary stream is substantially noisier than a designated primary stream---such symmetry creates a backflow channel that injects auxiliary noise into the primary representation and amplifies errors across iterative refinement stages. We formulate fusion in this regime as directed refinement with one-way safety: the primary modality defines a guidance field, while only auxiliary representations are iteratively purified, and primary perturbations induced by the auxiliary stream are explicitly bounded. We propose Hamiltonian Asymmetric Fusion (HAF), a lightweight unrolled refinement block that updates auxiliary tokens with momentum regularization and gated driving. The refinement force is instantiated by FFT-based spectral global correlation and modulated by a shared learnable spectral response to emphasize reliable frequency components with minimal parameters; a leaky momentum gate and a stable integrator improve multi-step refinement stability. We provide guarantees of auxiliary error contraction and bounded primary perturbation, which symmetric fusion operators do not satisfy under imbalance. Experiments on six RGB--D SOD benchmarks show consistent gains and substantially more graceful degradation under controlled auxiliary corruption.