CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models

Large Language Models(LLMs) have revolutionized text generation and multimodal perception, but their capabilities in 3D content generation remain underexplored. Existing methods compromise by producing either low-resolution meshes or coarse structural proxies, failing to capture fine-grained geometry natively. In this paper, we propose CG-MLLM, a novel Multi-modal Large Language Model (MLLM) capable of 3D captioning and high-resolution 3D generation in a single framework. Leveraging the Mixture-of-Transformer architecture, CG-MLLM decouples disparate modeling needs, where the Token-level Autoregressive (TokenAR) Transformer handles token-level content, and the Block-level Autoregressive (BlockAR) Transformer handles block-level content. By integrating a pre-trained vision-language backbone with a specialized 3D VAE latent space, CG-MLLM facilitates long-context interactions between standard tokens and spatial blocks within a single integrated architecture. Experimental results show that CG-MLLM significantly outperforms existing MLLMs in generating high-fidelity 3D objects, effectively bringing high-resolution 3D content creation into the mainstream LLM paradigm.

We present CG-MLLM, a multimodal LLM that unifies 3D captioning and high-resolution 3D generation in a single framework. Built on a Mixture-of-Transformer design, it decouples token-level content (TokenAR) from block-level spatial content (BlockAR), and couples a pretrained vision-language backbone with a dedicated 3D VAE latent space, enabling long-context interaction between tokens and spatial blocks. Experiments show CG-MLLM markedly outperforms prior MLLMs in 3D fidelity, bringing high-resolution 3D creation into the mainstream LLM paradigm.