Calibrated Knowledge Aggregation in Bayesian Mixture-of-Experts for Continual VQA

Continual learning for visual question answering (VQA) is typically implemented by training one expert per task and routing each query using task-ID supervision. Yet continual VQA tasks overlap substantially: on the VQA-v2 task stream, a non-native expert outperforms the task’s own expert on $49.9\%$ of queries, so hard routing both wastes transferable knowledge and can be confidently wrong when mismatched. We propose a calibrated Bayesian mixture-of-experts that trains parameter-efficient per-task adapters, learns routing by directly maximizing expected VQA utility, and marginalizes expert identity at inference via Bayesian aggregation in a unified answer space; an entropy penalty prevents the utility objective from collapsing to one-hot routing, enabling evidence pooling across plausible experts. We reach $64.16$ accuracy with $0.63$ forgetting on VQA-v2 CL-LS ($+5.74\%$ accuracy, $-2.99$ forgetting vs. the strongest prior method), $78.81$ with $0.40$ forgetting on TDIUC CL-LS ($+3.10$, $-1.74$), and $83.41$ with $3.21$ forgetting on TDIUC CL-VS ($+1.58$, $-0.82$). Calibration also improves on VQA-v2, reducing ECE from $0.15$ to $0.07$.

Many AI systems can answer questions about images, such as identifying objects, actions, or scene details. In real use, these systems may need to learn new kinds of questions or images over time, but learning new tasks can make them forget older ones or become too confident in wrong answers. This paper studies how to make image-question answering systems more reliable as they keep learning. Instead of forcing the system to choose only one task-specific model for each question, we let several specialized models contribute when they seem useful. The method learns which specialists are likely to help for each question, then combines their answers so that agreement raises confidence and disagreement lowers it. This matters because image-question tasks often overlap: a model trained for one kind of question can still help answer another. Across several benchmarks, the approach improved accuracy, reduced forgetting, and made the system’s confidence better match whether its answers were correct. These results suggest a practical way to build visual question answering systems that adapt over time while making fewer overconfident mistakes.