PLaID++: A Preference Aligned Language Model for Targeted Inorganic Materials Design

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising approach to improve correctness in LLMs, however, in many scientific problems, the objective is not necessarily to produce *the* correct answer, but instead to produce a diverse array of candidates which satisfy a set of constraints. We study this challenge in the context of materials generation. To this end, we introduce PLaID++, an LLM post-trained for stable and property-guided crystal generation. We find that applying naive preference optimization to a coordinate-based crystal representation leads to mode collapse. Hence, we introduce a compact, symmetry-informed Wyckoff text representation which improves computational efficiency and encourages generalization from physical priors. By encoding symmetry constraints directly into text and guiding model outputs towards desirable chemical space, PLaID++ generates structures that are thermodynamically stable, unique, and novel at a $>$50\% greater rate than prior methods. We further demonstrate that unified training across conditional and unconditional tasks are mutually beneficial in data-sparse regimes. Our work demonstrates the potential of adapting post-training techniques from natural language processing to materials design, paving the way for targeted and efficient discovery of novel materials.

Recently, a machine learning technique called Reinforcement Learning has been adopted in LLM training because it can teach a model to generate more correct answers by rewarding right solutions and penalizing wrong ones. However, in many scientific problems, the goal is not to generate a single correct answer, but rather many solutions that satisfy a set of constraints. This challenge is particularly relevant in materials discovery, where the goal is not to find a singular "correct" material, but rather many candidate materials with desired physical properties that can be tested in the lab. Our paper finds that naively applying reinforcement learning to this problem does not work: the model memorizes the specific crystals it is told are good, rather than the underlying physical principles of why those crystals were good. To fix this, we introduce a new format for representing crystals as text that incorporates symmetry—similar to how a small wedge of a snowflake can be unfolded to describe its entire pattern. Our second result is that training a single model across many crystallographic objectives outperforms training separate specialized model for each task. Our model PLaID++ outperforms many other models that use more mathematically complex and materials-specific methods. These results indicate that with smart design of how crystals are formatted and what types of crystals are rewarded, general-purpose language models with feedback from other AI models can be used to discover brand-new crystals with interesting properties.