Deep learning techniques have revolutionized the field of computational biology, however it is often difficult to assign biological meaning to their results. To improve interpretability, methods have incorporated biological priors, like pathway definitions, directly into the learning task. However, due to the correlated and redundant structure of pathways, it is difficult to determine an appropriate computational representation. Here, we present \textbf{pathway module Variational Autoencoder} (pmVAE). Our method utilizes pathway information by restricting the structure of our VAE to mirror gene-pathway memberships. Its architecture is composed of a set of subnetworks, refered to as pathway modules, that learn interpretable multi-dimensional latent representations by factorizing the latent space according to pathway gene sets. We directly address correlations between pathways by balancing a module-specific local loss and a global reconstruction loss. We demonstrate that these representations are directly interpretable and reveal underlying biology, such as perturbation effects and cell type interactions. We compare pmVAE against two other state-of-the-art methods on a single-cell RNA-seq case-control dataset, and show that our representations are both more discriminative and specific in detecting the perturbed pathways.