Unnatural Languages Are Not Bugs but Features for LLMs

Large Language Models (LLMs) have been observed to process non-human-readable text sequences, such as jailbreak prompts, often viewed as a bug for aligned LLMs. In this work, we present a systematic investigation challenging this perception, demonstrating that unnatural languages - strings that appear incomprehensible to humans but maintain semantic meanings for LLMs - contain latent features usable by models. Notably, unnatural languages possess latent features that can be generalized across different models and tasks during inference. Furthermore, models fine-tuned on unnatural versions of instruction datasets perform on-par with those trained on natural language, achieving (49.71) win rates in Length-controlled AlpacaEval 2.0 in average across various base models. In addition, through comprehensive analysis, we demonstrate that LLMs process unnatural languages by filtering noise and inferring contextual meaning from filtered words. Our code is publicly available at https://github.com/John-AI-Lab/Unnatural_Language.

Large language models (AI systems like ChatGPT) are usually trained on human-readable text, but they can also understand and process text that looks like gibberish to us. This paper explores how these models handle such "unnatural" language—strings of words or symbols that don’t make sense to humans but still carry meaning for the AI. Surprisingly, the models can extract useful information from these unnatural inputs, even performing as well as when trained on normal language in some tasks. The study also shows that AI processes these strange inputs by filtering out noise and piecing together meaning from the remaining clues. This discovery challenges the assumption that AI only works well with human-like text and opens up new ways to think about how these models understand language.[Code available here: https://github.com/John-AI-Lab/Unnatural_Language]