We develop a transformer-based conditional generative model for discrete point-objects and their properties and use to to build a model for populating cosmological simulations with gravitationally collapsed structures called dark matter halos.Specifically, we condition our model with dark matter distribution obtained from fast, approximate simulations to recover the correct three-dimensional positions and masses of individual halos. This leads to a first model that can recover the statistical properties of the halos at small scales to better than 3\% level using an accelerated dark matter simulation. This trained model can then be applied to simulations with significantly larger volume which would otherwise be computationally prohibitive with traditional simulations, and also provides a crucial missing link in making end-to-end differentiable cosmological simulations. The code, named GOTHAM (Generative Conditional Transformer for Halos And their Masses) will be made publicly available.