WBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution

Large kernel depthwise convolutions achieve strong performance but suffer from significant degradation as kernel size grows due to irregular memory access from gather-based computation; while Large Kernel Acceleration (LKA) helps on small feature maps, it becomes counterproductive on large feature maps, even slower than non-accelerated implementations. We propose Windowed Batch Matrix Multiplication (WBMM), which partitions input into contiguous windows and indexes a compact relative position bias table to construct weight matrices, enabling regular memory access via batched matrix multiplication. This yields a unique property: WBMM's throughput improves with larger windows, opposite to depthwise convolutions that degrade with larger kernels. Operator-level benchmarks show WBMM with $14 \times 14$ windows outperforms $5 \times 5$ depthwise convolution baselines in speed while providing a $7.8\times$ larger per-layer receptive field. Combined with inter-block cross-window communication and hierarchical window reparameterization, WBMM achieves comparable or higher accuracy on ImageNet-1K, COCO, and ADE20K with $1.31$--$1.88\times$ training speedup, and demonstrates consistent advantages across GPU, CPU, and edge devices without requiring specialized acceleration kernels. Our code is available at https://github.com/wansong-s/WBMM.

Modern AI vision systems work better when they can take in larger areas of an image at once, which requires larger filters. But large filters run slowly: to compute each result, the computer fetches pixels scattered across its memory, like collecting items from shelves spread all over a warehouse. The bigger the filter, the more scattered these trips — and existing speed-up tricks backfire on high-resolution images. We developed a method called WBMM that reorganizes this computation. Instead of chasing scattered pixels, it splits the image into small contiguous tiles stored together in memory, like grabbing one pre-packed box instead of roaming the warehouse. It looks up the filter weights from a compact table and processes them in one efficient bulk operation. Surprisingly, this gets faster as the tiles grow larger — the opposite of normal filters. WBMM matches or beats leading methods' accuracy on standard image benchmarks while training up to nearly twice as fast, running efficiently on everything from data-center GPUs to small edge devices without special hardware. This makes powerful, wide-viewing vision models cheaper and greener to build, putting them within reach of researchers with limited computing budgets.