De novo molecule design has become a highly ac-tive research area, advanced significantly throughthe use of state-of-the-art generative models. De-spite these advances, several fundamental ques-tions remain unanswered as the field increasinglyfocuses on more complex generative models andsophisticated molecular representations as an an-swer to the challenges of drug design. In thispaper, we return to the simplest representationof molecules, and investigate overlooked limita-tions of classical generative approaches, particu-larly Variational Autoencoders (VAEs) and auto-regressive models. We propose a hybrid modelin the form of a novel regularizer that leveragesthe strengths of both to improve validity, condi-tional generation, and style transfer of molecularsequences. Additionally, we provide an in depthdiscussion of overlooked assumptions of thesemodels’ behaviour.