Quantifying the uncertainties associated with predictive models can facilitate optimal decision-making and accelerate workflows where time and resource efficiency are essential. Computational tools exist that estimate the predictive uncertainty, which is useful for assessing the costs and risks involved with deploying machine learning models. In drug discovery, these tools can provide valuable insights into the efficient allocation of resources by identifying promising experiments, thereby reducing the overall costs associated with the development of therapeutic agents. We address the pressing need for a comprehensive, large-scale temporal evaluation of probability calibration methods, specifically focusing on drug-target interactions. We investigate the performance of several calibration-free uncertainty estimation and post-hoc probability calibration methods. Furthermore, we systematically compare the effect of different training set sizes and shifts in active ratios on the capability of the uncertainty estimation methods.