Diffusion models are a promising approach to generate molecules in 3D. However, challenges remain in generating realistic 3D molecules with refined geometry that respect the quantum physical laws governing the atom configurations. In this work, we generalized the neural predictor used in diffusion guidance to a non-differentiable expert oracle, GFN2-xTB, a semi-empirical quantum mechanical method for accurate and efficient quantum chemistry calculations. With an off-the-shelf diffusion model, we guide it to generate molecules that are valid and more stable with less net force. By prompting atoms to move to lower molecular energy with the estimated gradient from GFN2-xTB, we show that our method generates molecules that are more stable and favored in energy with better optimized geometries than existing literature.