We require systems to monitor species in real time and in greater detail to quickly understand which conservation and sustainability efforts are most effective and take corrective action. Current ecological monitoring systems generate data far faster than researchers can analyze it, making scaling up impossible without automated data processing. Pre-training, particularly methods that require minimal human supervision, is clearly well-aligned with this problem setting where large amounts of unlabeled data are available. However, ecological data collected in the field presents a number of challenges that current pre-training methods are often not designed to tackle. These include strong spatiotemporal correlations and domain shifts, imperfect data quality, fine-grained categories, and long-tailed distributions. I will discuss gaps between the current pre-training paradigm and what is needed for usable, impactful computer vision based environmental monitoring systems, and outline several interesting future directions at the intersection of pre-training and environmental monitoring.