Speedrunning GPT3: Training an (Almost-) GPT3-175B-Quality Model in Under 10K USD

We describe a system to pretrain a 4B-parameter model, called CloverLM, aimed at zero-shot performance similar to the standard GPT3-175 / OPT-175B models, in a highly cost-effective manner. Our approach works by combining multiple known techniques: 1) Accurate native NVFP4 training via the Quartet II algorithm, 2) High-quality data training on the CLIMB dataset; 3) Several model- and framework-specific optimizations. While we claim no technical novelty, it is notable that we can reach OPT-175B-level of accuracy on multi-choice zero-shots in pure NVFP4 using approximately 1600 B300 GPU hours, for an estimated cost between USD 5'600 (spot) and USD 10'000 (on-demand), for the main run, on a sustainable neocloud provider. Our code is openly available.