We introduce a generative model for galaxy spectra based on an autoencoder architecture. Our encoder combines convolutional and attentive elements to identify important spectral features. The decoder is a fully-connected network, tasked with generating restframe spectra, which are then explicitly redshifted to the observed redshifts and resampled to match the spectral resolution and coverage of the instrument. The architecture thus reflects the astrophysical dependencies of a data-generating process that exhibits two fundamental degrees of freedom for each galaxy, namely its redshift and the characteristics of its restframe spectrum, and learns a compressed data-driven parameterization of the latter. We train this model on 100,000 optical spectra from SDSS, and find that it generates highly realistic galaxy spectra and excellent representations of the inputs. However, the desired redshift-independent encoding is possible only by augmenting the training spectra with artificially altered redshifts. Doing so establishes redshift invariance at the price of restricting the utilized spectral features to a consensus set that is accessible for any redshift covered by the training data, thereby limiting the information extracted from all spectra.