In principle, quantum chemistry allows us to quantify all electronic and geometric properties of molecules and their interactions. Thus, incorporating pre-calculated quantum mechanical properties into deep learning models could improve their ability to predict important pharmacological properties of small molecules and potential drugs. However, this opportunity has been under-exploited in the recent wave of AI-driven drug discovery. We show that by pre-training Equivariant Graph Neural Network (EGNN) models to predict atom-centered partial charges, that have been pre-calculated using quantum mechanical methods, we can obtain more accurate models to predict absorption, distribution, metabolism, excretion, and toxicological (ADMET) properties. We compared the performance of quantum chemistry pre-training against non-quantum mechanics-based pre-training and with no pre-training at all, and found quantum chemistry pre-training to produce the most accurate models for lipophilicity, blood-brain barrier penetration, metabolism by CYP2D6, and toxicity; and very similar performance to non-pre-trained models for the much more challenging task of hepatocyte clearance prediction. By using our quantum chemistry-based pre-training approach to predict both atom-level and molecule-level properties, we obtain richer representations of the molecules than without pre-training, helping our models to learn from the underlying physics and chemistry.