With the rapid development of decision aids that are driven by AI models, the practice of human-AI joint decision making has become increasingly prevalent. To improve the human-AI team performance in decision making, earlier studies mostly focus on enhancing humans' capability in better utilizing a given AI-driven decision aid. In this paper, we tackle this challenge through a complementary approach---we aim to adjust the designs of the AI model underlying the decision aid by taking humans' reaction to AI into consideration. In particular, as humans are observed to accept AI advice more when their confidence in their own decision is low, we propose to train AI models with a human-confidence-based instance weighting strategy, instead of solving the standard empirical risk minimization problem. Under an assumed, threshold-based model characterizing when humans will adopt the AI advice, we first derive the optimal instance weighting strategy for training AI models. We then validate the efficacy of our proposed method in improving the human-AI joint decision making performance through systematic experimentation on both synthetic and real-world datasets.