We introduce a diffusion-generative model to describe the distribution of galaxies in our Universe directly as a collection of points in 3-D space, without resorting to binning or voxelization. The custom diffusion model, which employs graph neural networks as the backbone score function, can be used as an emulator that accurately reproduces essential summary statistics of the galaxy distribution and enables cosmological parameter estimation using gradient-based inference techniques. This approach allows for a comprehensive analysis of cosmological data by circumventing limitations inherent to summary statistics-based as well as likelihood-free methods.