Adaptive Trimodal Fusion for Mental-Health Symptom Classification in Memes

Public social-media memes often express anxiety and depression through irony, visual metaphor, and figurative reference, carrying psychological signal that is frequently missed by purely textual analysis. Models for this task may support aggregate public-health research and moderation triage, provided their outputs are treated as uncertain signals rather than individual-level assessments. We study fine-grained anxiety and depression symptom classification on the AxiOM and RESTORE benchmarks and propose TCAF, a trimodal classifier that operates on top of three frozen pretrained encoders: a vision encoder over the meme image and a frozen MentalBART text encoder applied separately to OCR text and to figurative reasoning triples. The trainable head combines an Adaptive Gated Trimodal Fusion (AGTF) block that learns a per-dimension soft mask over the concatenated modalities, an additional three-way modality gate, a dual classifier head that blends a linear MLP with learnable class prototypes, and a supervised contrastive auxiliary loss on the fused representation. With only $1.7$M trainable parameters, TCAF reaches a test macro-F1 of $0.7067$ on AxiOM and $0.6857$ on RESTORE, exceeding the prior trimodal state of the art on both benchmarks. We further provide a per-class analysis of modality weighting and a geometric study of the learned class prototypes.