Medical object detection revolves around discovering and rating lesions and other objects, with the most common way of measuring performance being FROC (Free-response Receiver Operating Characteristic), which calculates sensitivity at predefined thresholds of false positives per case. However, in a diagnosis or screening setting not all lesions are equally important, because small indeterminate lesions have limited clinical significance, while failing to detect and correctly classify high risk lesions can potentially hinder clinical prognosis and treatment options. It is therefore cardinal to correctly account for this risk imbalance in the way machine learning models are developed and evaluated. In this work, we propose risk-adjusted FROC (raFROC), an adaptation of FROC that constitutes a first step on reflecting the underlying clinical need more accurately. Experiments on two different breast cancer datasets with a total of 1535 lesions in 1735 subjects showcase the clinical relevance of the proposed metric and its advantages over traditional evaluation methods. Additionally, by utilizing a risk-adjusted adaptation of focal loss (raFocal) we are able to improve the raFROC results and patient-level performance of nnDetection, a state-of-the-art medical object detection framework, at no expense of the regular FROC.