Fisher-Preserving Guidance: Training-Free Manifold Constraints for Safe Diffusion Control

Diffusion models are effective for waypoint prediction in visual navigation, but standard sampling and test time guidance can produce unreliable or inefficient trajectories when updates drift off the training manifold. We propose Fisher Preserving Guidance with Outer Product Span Projection, a training-free inference method that avoids large Fisher drift associated with off-distribution actions while optimizing a task objective. Our method computes the Fisher-preserving update via a low-rank Jacobian factorization, requiring only a single backward pass per step and enabling real-time use. We further introduce Truncated Fisher Denoising Sensitivity as an uncertainty signal and use it for robust multi-sample action blending. Experiments on toy and realistic navigation benchmarks, including Maze2D with TSDF-based guidance, PushT with official Diffusion Policy weights, and visual navigation in simulation and on real robots, demonstrate consistent improvements in performance over strong diffusion-policy baselines without additional training.

Robots that use diffusion models can predict useful navigation waypoints and actions, but their predictions can become unreliable when extra guidance pushes them toward behavior they have not learned before. This paper introduces a training-free method that helps guide robot actions toward the task goal while keeping them close to familiar behavior. It also estimates when predicted actions are uncertain and uses this information to combine several possible actions more reliably. Across simulated and real-world navigation and control tasks, the method improves performance over strong existing diffusion-policy approaches without requiring additional training.