PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs

Despite remarkable progress in Multimodal Large Language Models (MLLMs), these models still struggle with fine-grained understanding tasks. In this work, we propose Procedurally Generated Tasks (PGT) a simple data-driven framework that serves a dual purpose: inducing fine-grained visual understanding and acting as a low-cost diagnostic tool to identify the source of perception failures. By overlaying unambiguous geometric primitives on images, PGT generate additional dense supervision that disentangles visual grounding capability from semantic priors. Extensive experiments on relational, quantitative, and 3D/depth understanding benchmarks show that PGT yields remarkable gains across diverse architectures. Instruction tuning MLLMs on LLaVA-v1.5-Instruct augmented with PGT data results in improvements of up to +20\% on the What’sUp benchmark and +13.3\% on CV-Bench-2D, while maintaining general perception capabilities. Moreover, finetuning state-of-the-art MLLMs on PGT data leads to boosts of up to +5.5\% on What’sUp and +8.3\% on CV-Bench-2D. These findings demonstrate that PGT effectively address the bottleneck of fine-grained perception, revealing that many spatial reasoning deficits stem from inadequate supervision signals rather than inherent architectural or resolution limitations.

Modern AI models are great at looking at a picture and telling you what is in it, but they often struggle with the fine details. For example, they frequently make mistakes when trying to judge distances, count objects accurately, or understand exactly how different items in an image relate to one another spatially. To address this, we created a simple, automated system called Procedurally Generated Tasks (PGT). Instead of relying on expensive human labels, we propose to automatically draw basic shapes—like circles and boxes—directly onto existing images to create clear, visual puzzles for the AI to solve. This forces the AI to look closely at the actual visual details rather than just guessing based on context clues (like assuming a bowl is on a table rather than under it). Our training method significantly boosts the AI's ability to understand spatial details without hurting its general capabilities. It proves that AI spatial mistakes aren't caused by flaws in model design, but rather by the lack of precise training data.