Abstract: We present a method to infer galaxy properties and redshifts at the population level from photometric data using normalizing flows. Our method PopSED can reliably recover the redshift and stellar mass distribution of $10^{5}$ galaxies using SDSS ugriz photometry with <1 GPU-hour, being $10^{6}$ times faster than the traditional SED modeling method. The approach can also be applied to spectroscopic data including DESI and Gaia XP spectra. Our method provides an efficient and self-consistent way to learn the population posterior without deriving the posteriors for every individual object and then combining them.