Compositional Evolutionary Probing of LLM Safety Alignment

Evolutionary search against aligned language models has been studied mostly as an attack optimizer, with limited success when neither gradients nor a helper LLM are available. We take a different view and use evolutionary search as a probe of how safety-aligned models behave. Our method is a stack-based genetic program that composes four typed primitives (phrases, token blocks, separators, and target instructions) into candidate payloads, queries a frozen Llama 3 8B Instruct, and promotes token blocks that repeatedly score well back into the primitive pool as new named phrases. We run six experiments that differ only in which primitive pool the search draws from. Our results indicate that pool composition matters more than pool size: a curated 5,000-token pool produces 40 promoted primitives and 53.3% coverage of the token-exploit category, while a size-matched random pool produces zero. The primitives that emerge are mostly code and markup tokens such as sprintf, urlencode, componentWillUnmount, and -->}). This points to safety alignment in Llama 3 interacting with code-related regions of the model's pretraining distribution.