Distribution shifts pose significant challenges for model calibration and model selection tasks in the unsupervised domain adaptation problem--a scenario where the goal is to perform well in a distribution shifted domain without labels. In this work, we tackle difficulties coming from distribution shifts by developing a novel importance weighted group accuracy estimator. Specifically, we present a new perspective of addressing the model calibration and model selection tasks by estimating the group accuracy. In addition, we formulate an optimization problem for finding an importance weight that leads to an accurate group accuracy estimation in the distribution shifted domain with theoretical analyses. Our strong empirical results from extensive experiments emphasize the significance of group accuracy estimation for addressing the challenges in unsupervised domain adaptation, as an orthogonal improvement direction with improving transferability of accuracy.