Efficient molecular design is one of the fundamental goals of computer-aided drug or material design. In recent years, significant progress has been made in solving challenging problems across various aspects of computational molecular optimizations, with an emphasis on achieving high validity, diversity, novelty, and most recently synthesizability. However, a crucial aspect that is rarely discussed is the budget spent on the optimization. If candidates are evaluated by experiment or high-fidelity simulation, as they are in realistic discovery settings, sample efficiency is paramount. In this paper, we thoroughly investigate 13 molecular design algorithms across 21 tasks within a limited oracle setting, allowing at most 10000 queries. We illustrate that most ``state-of-the-art'' methods fail to outperform some classic algorithms. Our results also highlight the influence of the generative action space (e.g., token-by-token, atom-by-atom, fragment-by-fragment) on performance and the necessity of multiple independent runs and hyperparameter tuning. We suggest a standard experimental benchmark to minimize the wasted effort caused by non-reproducibility, artificially poor baselines, and easily misinterpreted results.