Benchmark datasets are used to show the performance of an algorithm,~e.g. its accuracy, computational speed, or versatility. In the majority of cases, benchmark datasets currently have no external use,~i.e.~an improvement on the benchmark doesn't directly translate to a real-world impact. In this paper, we explore why this is the case, weigh benefits and harms, and propose ways in which benchmark datasets could make a more direct positive impact.