Recent empirical studies have found inductive biases in supervised learning toward simple features that may be spuriously correlated with the label, resulting in suboptimal performance on certain subgroups. In this work, we explore whether recent Self-Supervised Learning (SSL) methods would produce representations which exhibit similar behaviour. First, we show that classical approaches in combating spurious correlations, such as re-sampling the dataset, do not necessarily lead to invariant representations during SSL. Second, we discover that spurious information is represented disproportionately heavily in the later layers of the encoder. Motivated by these findings, we propose a method to remove spurious information from SSL representations during pretraining, by pruning or re-initializing later layers of the encoder. We find that our method produces representations which outperform the baseline on 5 datasets, without the need for group or label information during SSL.