CRAG: Can 3D Generative Models Help 3D Assembly?

Most existing 3D assembly methods treat the problem as pure pose estimation, rearranging observed parts via rigid transformations. In contrast, human assembly naturally couples structural reasoning with holistic shape inference. Inspired by this intuition, we reformulate 3D assembly as a joint problem of assembly and generation. We show that these two processes are mutually reinforcing: assembly provides part-level structural priors for generation, while generation injects holistic shape context that resolves ambiguities in assembly. Unlike prior methods that cannot synthesize missing geometry, we propose CRAG, which simultaneously generates plausible complete shapes and predicts poses for input parts. Extensive experiments demonstrate state-of-the-art performance across in-the-wild objects with diverse geometries, varying part counts, and missing pieces. Our code and models will be released.

Imagine trying to put a broken vase back together when some of the pieces are gone for good. This shows up everywhere: archaeologists rebuilding pottery, scientists piecing together fossil bones, robots repairing parts on a factory line. Today's programs treat it as a jigsaw puzzle: they only match edges between pieces, so they get stuck when pieces are missing, worn, or don't fit. Our work, CRAG, asks something different: what if the program could also picture in its head what the finished object should look like, and use that picture to decide where each piece goes? CRAG combines two AI models that usually work apart, one that figures out where each piece belongs in 3D space, and one that imagines the whole object, and lets them share what they learn. Placing the pieces makes the imagined object clearer, and the clearer picture helps decide where the pieces go. Across everyday furniture parts, broken pottery, and real fossil bones, CRAG does much better than earlier methods, and keeps working even when 40% of the pieces are missing. The same idea could help robots repair things, museums restore artifacts, and scientists rebuild specimens no human could finish by hand.