Skip to yearly menu bar Skip to main content


Poster

Auto-Encoding Morph-Tokens for Multimodal LLM

Kaihang Pan · Siliang Tang · Juncheng Li · Zhaoyu Fan · Wei Chow · Shuicheng YAN · Tat-Seng Chua · Yueting Zhuang · Hanwang Zhang


Abstract:

For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge. This is due to a conflicting objective: for comprehension, an MLLM needs to abstract the visuals; for generation, it needs to preserve the visuals as much as possible. Thus, the objective is a dilemma for visual-tokens. To resolve the conflict, we propose encoding images into \emph{morph-tokens} to serve a dual purpose: for comprehension, they act as visual prompts instructing MLLM to generate texts; for generation, they take on a different, non-conflicting role as complete visual-tokens for image reconstruction, where the missing visual cues are recovered by the MLLM. Extensive experiments show that morph-tokens can achieve a new SOTA for multimodal comprehension and generation simultaneously. The anonymous project is available at https://anonymous.4open.science/r/morph-tokens-498F.

Live content is unavailable. Log in and register to view live content