Benchmarking LLM Susceptibility to Generating Vulnerable Code via Few-Shot Model Inversion | HackerNoon

Table of Links

Abstract and I. Introduction

II. Related Work

III. Technical Background

IV. Systematic Security Vulnerability Discovery of Code Generation Models

V. Experiments

VI. Discussion

VII. Conclusion, Acknowledgments, and References

Appendix

A. Details of Code Language Models

B. Finding Security Vulnerabilities in GitHub Copilot

C. Other Baselines Using ChatGPT

D. Effect of Different Number of Few-shot Examples

E. Effectiveness in Generating Specific Vulnerabilities for C Codes

F. Security Vulnerability Results after Fuzzy Code Deduplication

G. Detailed Results of Transferability of the Generated Nonsecure Prompts

H. Details of Generating non-secure prompts Dataset

I. Detailed Results of Evaluating CodeLMs using Non-secure Dataset

J. Effect of Sampling Temperature

K. Effectiveness of the Model Inversion Scheme in Reconstructing the Vulnerable Codes

L. Qualitative Examples Generated by CodeGen and ChatGPT

M. Qualitative Examples Generated by GitHub Copilot

Abstract—Large language models (LLMs) for automatic code generation have achieved breakthroughs in several programming tasks. Their advances in competition-level programming problems have made them an essential pillar of AI-assisted pair programming, and tools such as GitHub Copilot have emerged as part of the daily programming workflow used by millions of developers. The training data for these models is usually collected from the Internet (e.g., from open-source repositories) and is likely to contain faults and security vulnerabilities. This unsanitized training data can cause the language models to learn these vulnerabilities and propagate them during the code generation procedure. While these models have been extensively assessed for their ability to produce functionally correct programs, there remains a lack of comprehensive investigations and benchmarks addressing the security aspects of these models.

In this work, we propose a method to systematically study the security issues of code language models to assess their susceptibility to generating vulnerable code. To this end, we introduce the first approach to automatically find generated code that contains vulnerabilities in black-box code generation models. To achieve this, we present an approach to approximate inversion of the black-box code generation models based on few-shot prompting. We evaluate the effectiveness of our approach by examining code language models in generating high-risk security weaknesses. Furthermore, we establish a collection of diverse non-secure prompts for various vulnerability scenarios using our method. This dataset forms a benchmark for evaluating and comparing the security weaknesses in code language models.

I. INTRODUCTION

Large language models represent a major advancement in current deep learning developments. With increasing size, their learning capacity allows them to be applied to a wide range of tasks, such as text translation [1], [2] and summarization [3], [2], chatbots like ChatGPT [4], and also for code generation and code understanding tasks [5], [6], [7], [8]. A prominent example is GitHub Copilot [9], an AI pair programmer based on OpenAI Codex [5], [10] that is already used by more than a million developers [11]. ChatGPT [4], Codex [5] and open models such as Code Llama [12], CodeGen [6] and InCoder [7] are trained on a large-scale corpus of natural language and code data and enable powerful and effortless code generation. Given a text prompt describing a desired function and a function header (first few lines of the desired code), these models generate suitable code in various programming languages and automatically complete the code based on the user-provided

context description. These models can dramatically increase the productivity of the software developer. As an example, according to GitHub, developers who use GitHub Copilot implement the desired programs 55% faster [11], and nearly 40 % of the code written by programmers who use Copilot is generated by the model [9].

Like any other deep learning model, large language models such as ChatGPT, Codex, Code Llama, and CodeGen exhibit undesirable behavior in some edge cases due to inherent properties of the model itself and the massive amount of unsanitized training data [13], [14]. In fact, these models are trained on unmodified source code hosted on GitHub. While the model is trained, it also learns the training data’s coding styles and—even more critical—bugs that can lead to securityrelated vulnerabilities [15], [16]. Pearce et al. [15] have shown that minor changes in the text prompt (i.e., inputs of the model) can lead to software faults that can cause potential harm if the generated code is used unaltered. The authors use manually modified prompts and do not provide a way to find the vulnerabilities of the code generation models automatically.

In this work, we propose an automated approach to test the potential of code models in generating vulnerable codes and to benchmark a model’s generated code security. For arXiv:2302.04012v2 [cs.CR] 23 Oct 2023 this, we propose an automated approach for finding prompts that systematically trigger the generation of codes containing a specific vulnerability and enable to examine the models’ behavior on a large scale that can be easily extended to the new type of vulnerabilities. More specifically, we formulate the problem of finding a set of prompts that cause the models to potentially generate vulnerable codes as an inversion task. Our goal is to have an approximation of the inverse of the target model, which can generate prompts that can lead the target model to generate code with a specific vulnerability. However, it is unclear how the inverse in the black-box setting can be accessed. To solve this problem, we propose an approach to formulate the inversion using the model itself and fewshot prompting (in-context examples) [1], which has recently shown a surprising ability to generalize to novel tasks. A fewshot prompt contains a few examples (input and expected output) of a specific task to teach a pre-trained model to generate the desired output. In this work, we use few-shot prompting to guide the target black-box model to act as an approximation of its own inverse. Specifically, we direct the model to generate prompts that will generate code containing a specific vulnerability by providing a few examples of vulnerable codes and their corresponding prompts.

We use this approximation to generate prompts that potentially lead the models to generate codes with specific vulnerabilities and reveal their security vulnerability issues. Figure 1 overviews our black-box model inversion approach. In our experiments, we show that these generated prompts are transferable across different models, and in contrast to previous work, our prompts can be automatically generated for the targeted vulnerabilities. Leveraging this evidence, we employ our approach to generate a set of non-secure prompts using state-of-the-art code models. These prompts form a benchmark to assess and compare different models in generating codes with security weaknesses.

In summary, we make the following key contributions in this paper:

1. We propose an approach for testing a model’s potential in generating vulnerable code. We achieve this by approximating an inversion of the target black-box model via few-shot prompting.

2. Our approach found a diverse set of non-secure prompts, leading the state-of-the-art code generation models to generate more than 2k Python and C codes with specific vulnerabilities.

3. We propose a dataset of diverse non-secure prompts to evaluate and compare the susceptibility of code models in generating vulnerable codes. These prompts were automatically generated by applying our approach to evaluate security issues in the state-of-the-art models.

4. With the publication of this work, we will release our approach and generated dataset as an open-source tool that can be used to benchmark the security of the black-box code generation models. This tool can be easily extended to newly discovered potential security vulnerabilities.