Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs

Current 3D spatial reasoning methods face a fundamental trade-off: neuro-symbolic 3D (NS3D) concept learners achieve interpretable reasoning through compositional programs but are constrained to closed-set concept vocabularies and simple programs; end-to-end 3D multi-modal LLMs (3D MLLMs) could handle complex natural language and open-vocabulary concepts but suffer from black-box reasoning without explicit spatial verification. We introduce APEIRIA, a neuro-symbolic 3D MLLM to bridge two paradigms by distilling symbolic reasoning patterns into MLLMs with natural language chain-of-thought. Our three-stage curriculum progressively builds reasoning capabilities: a) 3D perception alignment grounds object visual-geometric features to the LLM, b) CoT-SFT teaches query decomposition and stepwise verification from symbolic program traces, and c) CoT-RL extends reasoning patterns to open-set concepts and deeply nested instructions. By transferring reasoning patterns rather than concept-specific knowledge, APEIRIA preserves key NS3D virtues: transparent reasoning and modular interchangeability of planning and perception components. Evaluations on grounding, question answering, and captioning show that APEIRIA surpasses prior NS3D methods and matches state-of-the-art 3D MLLMs on 3D spatial reasoning datasets, unifying symbolic methods' systematic reasoning with MLLMs' flexibility. Code is available at https://github.com/oceanflowlab/APEIRIA.

Robots and AI assistants that work in homes or offices need to understand 3D spaces, not just recognize objects. For example, when asked to find “the chair next to the coat rack and left of the table,” they must identify objects, compare their positions, and explain why an object matches the description. Existing systems face a trade-off: some reason step by step but only understand a small fixed set of words, while newer large AI models understand flexible language but often make decisions as black boxes. We introduce APEIRIA, a system that teaches a 3D language model to reason more like a transparent step-by-step program. It first learns to recognize objects and their locations in 3D scenes, then learns from verified symbolic reasoning examples, and finally adapts these skills to more natural and complex instructions. This lets the model handle descriptions such as “comfortable furniture” while still showing the steps it used to reach an answer. In experiments, APEIRIA outperforms previous neuro-symbolic methods and matches or improves strong 3D language models on several 3D scene understanding tasks. By making spatial reasoning more explainable, this work could help build safer and easier-to-debug AI systems for robotics, navigation, and other embodied applications.