Algorithmic case-based decision support provides examples to help humans make sense of predicted labels and aid humans in decision-making tasks. However, representations learned by supervised models may not align well with human intuitions: what models consider as similar examples can be perceived as distinct by humans. In this work, we incorporate ideas from metric learning with supervised learning to examine the importance of alignment for effective decision support. In addition to instance-level labels, we use human-provided triplet judgments to learn human-compatible decision-focused representation. Using human subject experiments, we demonstrate that such representation is better aligned with human perception than representation solely optimized for classification. Human-compatible representations identify nearest neighbors that are perceived as more similar by humans and allow humans to make more accurate predictions.