Effects of Structural Reward Shaping on Biophysical Properties in RL-Trained Plasmid Generators

We compare the efficacy and distributional effects of supervised fine-tuning (SFT) and reinforcement learning (RL) post-training for PlasmidGPT, a foundation model for whole-plasmid generation, using Group Relative Policy Optimization (GRPO) for the RL model. Using a biologically motivated reward function encoding functional annotations, length constraints, and repeat penalties, the RL model achieves a 71.6% quality-control pass rate across 8 prompts on 4,000 sequences, compared to 4.3% for the pretrained baseline and 11.0% for SFT. A five-model reward ablation identifies the cassette arrangement bonus, which rewards correct promoter→CDS→terminator ordering, as the critical reward component. Rejectionsampling baselines indicate that the gain is not recovered by sampling more heavily from the base model. Beyond directly optimized features, RLgenerated sequences converge toward real plasmid distributions in 3-mer composition and minimum free energy density, neither of which is directly optimized by the reward function. Minimum free energy density independently converges to the real-plasmid regime under both SFT and RL despite these being parallel post-training paths. On a small curated hold-out set, RL improves continuation log-likelihood over the pretrained baseline on all 29 held-out sequences (mean ∆ = +0.83 nats).

Plasmid design is an essential step in many vital industries like pharmaceuticals, vaccine development, and gene therapy, but computational design remains a largely unsolved problem. Designing a new one is slow and expert-heavy: many genetic parts must be arranged in just the right order while avoiding sequences that destabilize the molecule. Existing AI generators produce mostly invalid designs. We applied reinforcement learning to an existing plasmid AI model, where the reward is inspired by how biologists design and evaluate plasmids: do the right components appear, in the right order, at the right length? Our model produces valid designs about 17 times more often than the starting model (72% vs. 4%). It also matches real plasmids on properties we never explicitly rewarded, such as fine-scale DNA patterns and folding stability, suggesting it absorbed real biology rather than just our scoring rules. This is a step toward AI that can quickly design custom plasmids for vaccines, therapies, and research.