Robust Multi-View Fusion via Prototype-Anchored Unbalanced Optimal Transport

Multi-view classifiers typically fuse all observed views into a single representation, which becomes fragile when some views are missing or corrupted.We propose a prototype-anchored fusion module based on an entropically regularized unbalanced optimal transport (UOT) barycenter.Each view is summarized into a small set of learned atoms and is matched to a shared prototype support; fusion outputs a probability measure over prototypes with fixed dimension.By relaxing marginal constraints with a generalized KL penalty, the UOT objective can leave a fraction of view mass unmatched when matching is geometrically costly, yielding a simple differentiable trimming mechanism without hand-tuned thresholds.We provide a basic theoretical result showing that discarding an arbitrary subset of atom mass incurs a penalty bounded by its total mass, independent of transport distances.Experiments on multi-view action recognition benchmarks under simulated missing views, missing-rate shift, and feature-space corruption demonstrate consistently improved stability under severe missingness with modest overhead on top of strong backbones.

Can an AI still recognize what a person is doing when some of its sensors fail? We wanted to build action-recognition systems — the kind that could watch over an elderly person at home or flag unsafe behavior in a factory — that keep working even in such cases. These systems usually combine signals from several sensors at once, like regular cameras, depth sensors, and skeleton trackers. But in the real world, sensors break, get blocked, or send garbled data, and today's methods often fall apart when this happens.Instead of treating each sensor's output as a single answer, we treat it as a "cloud" of possible answers with confidence scores attached. Our AI then searches for the best agreement between these clouds — even if some sensors go completely silent. The key idea comes from a mathematical theory originally invented to efficiently move goods between warehouses — technically called optimal transport. Surprisingly, even when 80\% of sensor data is randomly removed, our method still recognizes actions accurately, making reliable multi-sensor AI practical for real-world healthcare and safety applications.