The astronomy community is experiencing a lack of benchmark datasets tailored towards machine learning and computer vision problems. The overall goal of this software is to fill this need. We introduce the python library DeepBench, which is designed to provide a method of producing highly reproducible datasets at varying levels of complexity, size, and content. The software includes simulation of basic geometric shapes and astronomical structures, such as stars and ellipse galaxies, as well as tools to collect and store the dataset for consumption by a machine learning algorithm. We also present a trained ResNet50 model as an illustration of the expected use of the software as a benchmarking tool for different architectures’ suitability to scientifically motivated problems. We envision this tool to be useful in a suite of contexts at the intersection of astronomy and machine learning. For example, this could be useful for those new to machine learning principles and software as a way to build their skills and tools with a toy-model data set that looks like astronomical data. Also, experts can use this tool to build simple data sets that allow them to check their models. Finally, the geometric/polygon images can be used a highly simplified version of astronomical objects: this could be used for addressing a spectrum of problems from object classification to deblending.