The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generation

Large language models (LLMs) now generate substantial production code, often for tasks with multiple valid algorithmic solutions. Incidental prompt cues—contextual words or metadata outside the task specification—can steer which algorithm the model selects, even when all outputs pass the same tests. Prompt sensitivity is well studied as a mechanism for improving output quality; here, we instead study output policy: algorithm choice under fixed correctness. We define algorithm steering as cue-induced shifts in algorithm-family distributions and run 46,535 controlled experiments across 11 tasks, 19 cue types (18 channels plus a memoization semantic-vs-surface ablation that preserves meaning while changing typography and punctuation), and 15 model configurations. We find large, systematic shifts in algorithm-family distributions (up to 100 percentage points, pp), largely consistent with cue semantics, including in applied tasks such as rate limiting. Direct algorithm naming is the most reliable mitigation we tested. These results reveal an “invisible lottery” in which accidental context alters downstream properties such as performance, security, and maintainability.

When a developer asks an AI coding assistant for help, the assistant often has several valid ways to solve the same problem; one solution might be fast but use lots of memory, another might be slow but simple to read, and a third might quietly use a function that is convenient but unsafe on real-world inputs. All of these solutions can pass the same correctness tests, so a developer running unit tests cannot tell which one they shipped. We show that small, seemingly irrelevant pieces of the prompt — the name of a team, an imported library, a project label, or whether the assistant is told to act like a junior or an academic — can flip the assistant from one valid solution to another. We measure this effect across fifteen models and eleven tasks, find shifts of up to 100 percentage points, and show that simply asking the model directly for the algorithm we want eliminates the lottery. The practical takeaway is that AI-generated code should be evaluated for algorithm choice in addition to functional correctness, especially when performance, memory, or security depend on the choice.