Deformba: Vision State Space Model with Adaptive State Fusion

State Space Models (SSMs) have emerged as a powerful and efficient alternative to Transformers, demonstrating linear-time complexity and exceptional sequence modeling capabilities. However, their application to vision tasks remains challenging. First, existing vision SSMs largely depend on manually designed fixed scanning methods to flatten image patches into sequences, which imposes predefined geometric structures and increases the complexity. Second, the broader adoption of vision SSMs is hindered in domains that require query-based interactions between distinct information streams. This is a result of the inherently causal and self-referential nature of SSMs designed for 1D sequence modeling tasks. This fusion mechanism is indispensable for critical perception tasks such as multi-view 3D fusion. To address these limitations, we propose Deformba, a context adaptive method that dynamically augments the spatial structural information while maintaining the linear complexity of SSMs. Deformba also allows multi-modal fusion, analogous to standard cross attention. To demonstrate the effectiveness and general applicability of Deformba, we test its performance on general 2D vision tasks such as image classification, object detection, instance segmentation, and semantic segmentation, as well as 3D vision tasks like BEV perception. Extensive experiments show that Deformba achieves strong performance across various visual perception benchmarks.

Many efficient AI models process images by turning them into a long sequence, much like reading a picture line by line. This works well for speed, but it can miss the natural two-dimensional structure of images and makes it hard to combine information from different sources, such as multiple camera views in autonomous driving. Deformba addresses this by first writing image information into a compact memory and then learning where to read from that memory based on the current visual context, which is similar to how a person may look back and forth across an image to understand important regions. This adaptive reading replaces rigid scan patterns and also supports query-based fusion for multi-view 3D perception like bird’s-eye-view. Experiments on classification, detection, segmentation, and autonomous-driving perception show strong accuracy with efficient, linear-time computation which makes state space models attractive for practical visual perception systems.