Locate and Replace: A Diffusion-Transformer Hybrid Architecture for Language Generation

Continuous diffusion language models lag behind autoregressive transformers, partly because diffusion is applied in spaces poorly suited to language denoising and token recovery. We propose DiHAL, a geometry-guided diffusion-transformer hybrid that asks where diffusion should enter a pretrained transformer. By scoring layers with geometry-based proxies, DiHAL selects a diffusion-friendly hidden-state interface and replaces the lower transformer prefix with a diffusion bridge, retaining the upper layers and original LM head to avoid direct continuous-to-discrete recovery. Experiments on 8B-scale backbones show that the geometry score predicts effective shallow insertion layers under a fixed bridge-training protocol, and that hidden-state recovery improves over continuous diffusion baselines in a diagnostic comparison matching the training budget. These results suggest that hidden-state geometry helps identify where diffusion-based replacement is feasible inside pretrained language models.