Poster
in
Workshop: Methods and Opportunities at Small Scale (MOSS)

Encoding Domain Insights into Multi-modal Fusion: Improved Performance at the Cost of Robustness

Jackson Michaels ⋅ Sidong Zhang ⋅ Madalina Fiterau

Keywords: Interpretability Robustness Small-Scale Experiments Synthetic Tasks Inductive Priors Multi-modal Fusion

Project Page [ OpenReview]

Abstract

Using small-scale experiments with real and synthetic tasks, we compare multi-modal fusion methods, including a proposed `Product Fusion', to demonstrate how encoding task-specific priors affects performance. Our results highlight a crucial trade-off: aligning fusion design with priors boosts clean-data accuracy with limited data but significantly diminishes robustness to noisy inputs.

Chat is not available.