Can LLMs Test Their Own Code? Cross-Model Property-Based Testing of Code

Code-generation benchmarks evaluate LLM outputs against small sets of unit tests, but programs that pass every test can still violate their specification. We introduce X-PBT, a lightweight evaluation method that uses LLM-generated property-based tests (PBTs) to detect these specification-compliance failures. Across MBPP+ and HumanEval, we find that $2.8\%$ to $15.3\%$ of unit-passing programs generated by Claude Sonnet~4.5, GPT-4, and GPT-5.1 fail an LLM-generated PBT. On MBPP+, $3.6\%$ to $7.4\%$ of programs that also pass the held-out private test suite still fail, demonstrating that PBTs and example-based tests often catch complementary errors. A round-robin evaluation across PBT generators confirms these rates are not artifacts of generator self-bias. A manual audit further reveals that $26\%$ of PBT failures expose genuine bugs in benchmark reference solutions themselves.