Reliable uncertainty quantification is crucial for applying machine learning in safety critical systems such as in healthcare and autonomous vehicles, since it enables decision-making to account for risk. One effective strategy is to construct prediction sets, which modifies a model to output sets of labels rather than individual labels. In this talk, we describe our work on prediction sets that come with probably approximately correct (PAC) guarantees. First, we propose an algorithm for constructing prediction sets that come with PAC guarantees in the i.i.d. setting. Then, we show how our algorithm and its guarantees can be adapted to the covariate shift setting (which is precisely when reliable uncertainty quantification can be most critical). Furthermore, we describe how to adapt our algorithm to the meta learning setting, where a model is adapted to novel tasks with just a handful of examples. Finally, we demonstrate the practical value of PAC prediction sets in a variety of applications, including object classification, detection, and tracking, anomaly detection, natural language entity prediction, detecting oxygen saturation false alarms in pediatric intensive care units, and heart attack prediction.