Crop yield prediction typically involves the utilization of either theory-driven process-based crop growth models, which have proven to be difficult to calibrate for local conditions, or data-driven machine learning methods, which are known to require large data sets. In this work we investigate potato yield prediction using a hybrid modeling approach. A crop growth model is employed to generate synthetic data for (pre)training a convolutional neural net, which is then fine-tuned with observational data. When applied in silico, our hybrid approach yields better predictions than a baseline comprising a purely data-driven approach. When tested on real world data from field trials (n=303) and commercial fields (n=77), our hybrid approach yields competitive results with respect to the crop growth model. In the latter set, however, both models perform worse than a simple linear regression with a hand-picked feature set and dedicated preprocessing designed by domain experts. Our findings indicate the potential of hybrid modeling for accurate crop yield prediction; however, further advancements and validation using extensive real-world data sets is recommended to solidify its practical effectiveness.