Selective Code Generation under Correctness and Security Risk
Abstract
LLM-generated code can pass every unit test and still contain known-vulnerable patterns. Existing selective-generation methods -- pass@1, best-of-K on tests, and one-objective conformal abstention -- accept it silently. We extend Conformal Risk Control (CRC) to the joint event of accepting code that fails a test verifier or triggers a static analyzer. Our wrapper, joint-CRC, samples K candidates per problem, clusters by joint test-pass / static-analysis-clean behavior, and accepts when the largest good cluster clears a calibrated threshold; one CRC call bounds the joint silent-failure rate at a user-chosen alpha. On thirteen function-level CWE prompts (K=8, two generators), joint-CRC accepts correct-and-clean code in 82.6 +/- 1.4% of cases with no observed joint silent failures, while pass@1 and pass@1+Semgrep gate leak joint-bad code at 42% and 26%. A security-only CRC baseline achieves zero security silent failures yet still leaks 34% joint-bad code, directly showing that controlling either axis alone does not control the joint event. Our experiments demonstrate calibrated control of the verifier-defined joint event; controlling hidden correctness failures beyond the supplied tests requires a held-out test split, left to future work.