Certified Robustness under Heterogeneous Perturbations via Hybrid Randomized Smoothing

Randomized smoothing provides strong, model-agnostic robustness certificates, but existing guarantees are limited to single modalities, treating continuous and discrete inputs in isolation. This limitation becomes critical in multimodal models, where decisions depend on cross-modal semantics and adversaries can jointly perturb heterogeneous inputs, rendering unimodal certificates insufficient. We introduce a unified randomized smoothing framework for mixed discrete--continuous inputs based on an analytically tractable Neyman--Pearson formulation of the joint worst-case problem. By analyzing the joint likelihood ordering induced by factorized discrete and continuous noise, our approach yields a closed-form, one-dimensional certificate that strictly generalizes both Gaussian (image-only) and discrete (text-only) randomized smoothing. We validate the framework on multimodal safety filtering, providing, to our knowledge, the first model-agnostic Neyman--Pearson certificate for joint discrete-token and continuous-image perturbations in interaction-dependent text--image safety filtering.

Artificial intelligence systems increasingly make decisions from several types of input at once, such as an image together with a text prompt. This creates a robustness problem: an input may look harmless when the image and text are checked separately, but become unsafe when they are interpreted together. Existing robustness checks usually handle only one type of change at a time, such as small image changes or word changes, and therefore miss these joint attacks. We propose a general method to give mathematical guarantees for AI systems facing both kinds of changes at once. The method estimates whether the system’s decision remains stable even if an attacker modifies the text and slightly changes the image. We test it on multimodal safety filtering, where the goal is to keep unsafe image–text combinations detected despite adversarial changes. Our results show that joint guarantees are possible and more appropriate than checking each input type independently.