Black-Box Adversarial Attacks on LLM-Based Code Completion

Modern code completion engines, powered by large language models (LLMs), assist millions of developers with their strong capabilities to generate functionally correct code. Due to this popularity, it is crucial to investigate the security implications of relying on LLM-based code completion. In this work, we demonstrate that state-of-the-art black-box LLM-based code completion engines can be stealthily biased by adversaries to significantly increase their rate of insecure code generation. We present the first attack, named INSEC, that achieves this goal. INSEC works by injecting an attack string as a short comment in the completion input. The attack string is crafted through a query-based optimization procedure starting from a set of carefully designed initialization schemes. We demonstrate INSEC's broad applicability and effectiveness by evaluating it on various state-of-the-art open-source models and black-box commercial services (e.g., OpenAI API and GitHub Copilot). On a diverse set of security-critical test cases, covering 16 CWEs across 5 programming languages, INSEC increases the rate of generated insecure code by more than 50%, while maintaining the functional correctness of generated code. We consider INSEC practical - it requires low resources and costs less than 10 US dollars to develop on commodity hardware. Moreover, we showcase the attack's real-world deployability, by developing an IDE plug-in that stealthily injects INSEC into the GitHub Copilot extension.

Modern code completion tools, which use advanced AI models called large language models (LLMs), help millions of programmers by automatically suggesting code that works correctly. Because these tools are so popular, it's important to understand if they can introduce security risks.In this study, we show that attackers can secretly influence these AI-powered code completion tools to produce insecure code more frequently. This insecure code could then be exploited by an attacker when the product based on this code is released publicly. We developed the first attack of this kind, called INSEC. INSEC works by inserting a specially crafted short comment into the code input, which tricks the AI into generating insecure code. We create this special comment using a method that optimizes it through trial-and-error queries, starting from carefully chosen initial examples.We tested INSEC on several popular open-source AI models and commercial services like OpenAI's API and GitHub Copilot. Our tests covered a wide range of security issues (16 different types of vulnerabilities) across five programming languages. We found that INSEC increased the rate of insecure code generation by more than 50%, without affecting the functionality of the generated code.INSEC is easy and inexpensive to carry out—it can be developed using common hardware for less than $10. To demonstrate its real-world impact, we even created a plugin for a popular developer tool that secretly inserts the INSEC attack into GitHub Copilot, showing how easily this attack could be used in practice.