As machine learning algorithms continue to expand into healthcare domains that affect decision making systems, new strategies will need to be incorporated to effectively detect and evaluate subgroup disparities in order to ensure accountability and generalizablility in clinical machine learning workflows. In this paper, we explore how uncertainty can be used as one way to evaluate disparity in both patient demographics (race) and data acquisition (scanner) subgroups for breast density assessment on a dataset of 108,190 mammograms collected from over 33 clinical sites. Our results show that the choice of uncertainty quantification varies significantly at the subgroup level even if aggregate performance is comparable. We hope this analysis can promote future work on how uncertainty can be incorporated into clinical workflows to increase transparency in machine learning. The integration of predictive uncertainty can have implications for both regulation and generalizability of machine learning applications in healthcare.