Fine-tuning zero-shot foundation models often compromises their robustness to downstream distribution shifts. We propose dual risk minimization (DRM) which combines empirical risk minimization with worst-case risk minimization to better preserve core features conducive to downstream robustness. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of robustness: expected and worst-case performance over all possible domains, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of fine-tuned CLIP ViT-L/14@336 on ImageNet (75.9 to 77.1), WILDS-iWildCam (47.1 to 51.8), and WILDS-FMoW (50.7 to 53.1); opening up new avenues for achieving next-level robustness in fine-tuning zero-shot models.