Learning syntax without semantics: Disentangled tiny language models

Language models acquire syntax and world knowledge together, entangling the two in ways that limit efficiency and controllability. We show that syntax can be learned while suppressing semantic plausibility and world‑knowledge cues, yielding more efficient and controllable models. We train tiny LMs on grammatical nonsense — syntactically well-formed text with semantic content ablated via constrained relexicalization (SAMBAL). Models trained on this data perform comparably to standard pretraining on syntactic benchmarks (BLiMP, SyntaxGym) while scoring at chance on world knowledge probes (EWoK). On targeted grammar-plausibility conflict probes, content-neutral models prefer grammaticality where standard models prefer plausibility, and their representations show more syntactic vs lexical alignment. On efficiency, disentanglement yields substantial sample and parameter gains: in low‑resource regimes, a 5M‑parameter model matches a 30M‑parameter baseline at the same data budget. On controllability, content-neutral models adapt rapidly to a new domain with minimal exposure, suggesting the feasibility of modular post‑hoc knowledge specialization.

Today's language models are trained on enormous amounts of text — thousands of times more words than a child needs to learn to speak. During training they learn grammar and world knowledge together, with no way to separate the two afterward. We asked whether a model can instead learn the rules of language without picking up facts or plausibility about the world. We trained small language models on "grammatical nonsense" — ordinary text rewritten so that each sentence stays grammatically well-formed but no longer makes sense, like "Colorless green ideas sleep furiously." These models learn grammar as well as standard models while using less data, and knowing essentially nothing about the world. Specific knowledge can then be added on demand, giving you a capable model that knows only the facts you choose.