Evaluating Contextual Illegality: AI Compliance in Corporate Law Scenarios

While AI models often refuse explicitly unlawful requests, in real-world scenarios illegality often depends on context. We evaluate frontier models on contextual illegality across four corporate law scenarios in which routine actions—editing documents, trading stock, requesting payment, approving communications—become unlawful due to circumstances such as pending investigations or bankruptcy filings. We study both chat and agentic settings and compare results to a human baseline. The best-performing models consistently followed lawful requests and refused unlawful requests, though performance varied substantially between different scenarios and models. We also identify distinct failure modes, such as excessive refusal of lawful requests, and find higher performance in reasoning models and agentic environments. By studying contextual illegality in these controlled environments, we develop a methodology that can be extended to evaluate the legal compliance of AI models in additional scenarios and domains.

While AI models often refuse explicitly unlawful requests, in real-world scenarios illegality often depends on context. We evaluate frontier models on contextual illegality across four corporate law scenarios in which routine actions—editing documents, trading stock, requesting payment, approving communications—become unlawful due to circumstances such as pending investigations or bankruptcy filings. We study both chat and agentic settings and compare results to a human baseline. The best-performing models consistently followed lawful requests and refused unlawful requests, though performance varied substantially between different scenarios and models. We also identify distinct failure modes, such as excessive refusal of lawful requests, and find higher performance in reasoning models and agentic environments. By studying contextual illegality in these controlled environments, we develop a methodology that can be extended to evaluate the legal compliance of AI models in additional scenarios and domains.