Program synthesis, the automatic generation of programs that meet user specifications, has been a long-standing challenge in computer science. Traditional approaches focused on techniques like deductive and inductive synthesis, often guided by neural methods. However, the recent emergence of large language models (LLMs) and the abundance of open codebases have shifted the focus towards directly utilizing LLMs for code generation. LLMs like CODEX and CodeGen demonstrate impressive performance in code synthesis, leveraging natural language understanding and generative capabilities to predict code tokens based on context. The primary concern regarding LLM-generated code is its correctness. Traditional NLP metrics like BLEU scores are not reliable for evaluating program synthesis due to the possibility of semantically equivalent but structurally different code snippets. Formal verification, while ideal, proves challenging for general problems and is even difficult for domain-specific tasks. Consequently, existing code benchmarks heavily rely on manually constructed test cases to evaluate LLM solutions, which often fall short in capturing all possible scenarios. The paper argues that current benchmarks are inadequate for assessing the true correctness of LLM-generated code, leading to an overestimation of their performance. It identifies two key limitations of existing benchmarks: 1. **Insufficient Testing:** Existing benchmarks typically include a limited number of test cases (often less than 10) for each coding problem. These tests are often too simplistic, failing to fully explore the code's functionality or corner cases. As a result, logically flawed solutions can pass these basic tests, giving a false impression of correctness. The paper provides an example where ChatGPT generates seemingly correct code to return sorted unique common elements from two lists. However, it incorrectly converts the intermediate list to a set, compromising the intended order preservation. 2. **Imprecise Problem Descriptions:** The inputs for code generation often include natural language descriptions alongside the function signature. These descriptions, however, can be vague and ambiguous, failing to fully clarify the expected program behaviors. For instance, the input docstring might not specify the expected input domain or exception handling, leading to different interpretations by LLMs. Consequently, capable LLMs might be misjudged as incapable due to misinterpretations. These limitations, prevalent in various popular code generation benchmarks, call into question the validity of prior work and the claimed impressive performance of LLMs. The paper aims to address this fundamental evaluation challenge by asking the introspective question: "Is the code generated by LLMs really correct?"

The use of LLMs for code has gained immense popularity in recent years, driven by the availability of vast open codebases and the desire to improve developer efficiency. LLMs have demonstrated state-of-the-art performance in various code-related tasks, including code generation, program repair, automated testing, code translation, and code summarization. Notable LLMs like CODEX, CodeGen, INCODER, and PolyCoder have been developed and extensively evaluated for code generation, a long-standing challenge in computer science. These models generate code snippets based on natural language descriptions (e.g., docstrings) of the desired functionality. LLM-based code synthesis is primarily evaluated based on functional correctness, assessed by running test cases to check the desired outputs. HUMANEVAL is a pioneering and widely studied human-written benchmark for LLM-based code synthesis, consisting of 164 Python function signatures with docstrings and associated test cases for correctness checking. Each problem also includes a reference solution. Other datasets like MBPP, Spider, HUMANEVAL-X, CodeContests, and MultiPL-E extend the scope to other programming languages. Researchers have also created more realistic benchmarks based on GitHub issues, incorporating code bases and tests to assess LLMs' ability to perform real-world software engineering tasks. The paper's focus on test adequacy and automatic test generation stands out in this field.

To address the limitations of existing code generation benchmarks and evaluate the true correctness of LLM-generated code, the paper introduces EvalPlus, a comprehensive evaluation framework. EvalPlus focuses on automatically improving code benchmarks for LLMs by generating additional high-quality test cases. It combines both LLM-based and traditional mutation-based test input generation strategies to comprehensively test the functional correctness of LLM-synthesized code. The framework works as follows: 1. **Input:** EvalPlus takes as input an existing code generation dataset, including ground-truth implementations and base test inputs. It leverages the power of ChatGPT to generate a set of high-quality seed inputs, which are then further expanded through type-aware mutation to create a comprehensive set of test inputs. 2. **Seed Initialization via ChatGPT:** EvalPlus constructs a specialized prompt for ChatGPT, incorporating the ground-truth solution, exemplary test inputs, and instructions to generate interesting inputs. This prompt guides ChatGPT to learn the valid input formats and desired functionality of the ground-truth solution. The resulting inputs are filtered to ensure they comply with the program's input preconditions and semantic constraints. This step leverages ChatGPT's natural language understanding and generative capabilities to produce meaningful test cases. 3. **Type-Aware Input Mutation:** To generate large amounts of test inputs efficiently, EvalPlus performs type-aware mutation. This technique inspects the data types of valid seed inputs and generates new inputs structurally similar to the seeds. The mutations vary based on the input types (primitive, compound, string, etc.), ensuring the newly generated inputs are valid and cover different scenarios. The type-aware mutation approach utilizes the high-quality seed inputs from ChatGPT to generate diverse test inputs for evaluating the LLM-synthesized code. 4. **Test-Suite Reduction:** EvalPlus offers an optional test-suite reduction step to streamline the evaluation process. This step uses a greedy set covering algorithm to minimize the generated test suite while maintaining the same level of testing effectiveness. The algorithm prioritizes the coverage of different testing requirements, including code coverage, mutant killings, and empirical LLM sample killings. 5. **Program Input Contracts:** To further clarify the intended functionality and reduce ambiguity in problem descriptions, EvalPlus adopts a programming by contract approach. It systematically annotates function pre-conditions as code assertions, ensuring the test inputs are well-formed and valid. This approach helps filter out invalid inputs that might lead to undefined behaviors during testing. It also serves as an orthogonal descriptor alongside the natural language description, providing more comprehensive guidance for LLMs. 6. **Differential Testing:** EvalPlus uses differential testing to compare the outputs of the LLM-generated code and the ground-truth implementation for each test input. The test inputs generated by EvalPlus are used to rigorously evaluate the functional correctness of the LLM-synthesized code. The authors propose EvalPlus as a robust and versatile evaluation framework that can be applied to different code generation benchmarks. The paper focuses on evaluating its effectiveness on HUMANEVAL, a widely-used benchmark for code generation. It augments HUMANEVAL with additional test cases, resulting in HUMANEVAL+, and also creates a reduced version, HUMANEVAL+-MINI, for faster evaluation.

The paper's extensive evaluation, conducted over 26 popular and state-of-the-art LLMs, yielded several key findings: 1. **Significant Drop in Pass@k Rates:** Using HUMANEVAL+, the pass@k results for all evaluated LLMs consistently dropped compared to using the original HUMANEVAL benchmark. This significant drop in pass rates (up to 28.9% reduction) for different k values reveals the prevalence of undetected errors in LLM-generated code. This finding highlights the insufficiency of previous evaluation methods and the importance of using robust benchmarks like HUMANEVAL+. 2. **Impact on LLM Ranking:** The study surprisingly found that test insufficiency can lead to mis-ranking of LLMs. For example, both WizardCoder-CodeLlama and Phind-CodeLlama outperform ChatGPT on HUMANEVAL+, whereas they were deemed no better than ChatGPT on the original HUMANEVAL. This finding suggests that existing rankings of LLMs might not accurately reflect their true performance. It emphasizes the need for robust evaluation to obtain reliable rankings. 3. **Effectiveness of Test-Suite Reduction:** EvalPlus's test-suite reduction strategy (using greedy set covering) successfully minimized the generated test suite by 47 times while preserving a similar level of testing effectiveness. The reduced test suite achieved almost the same pass@1* drop as HUMANEVAL+, demonstrating its efficiency. The study also observed that using empirical LLM sample killings proved to be the most effective reduction strategy, while code coverage and mutation analysis provided a theoretical foundation for test adequacy. 4. **Incorrect Ground-Truth in HUMANEVAL:** The authors discovered 18 defects (11% of problems) in the ground-truth implementations of HUMANEVAL, further highlighting the limitations of existing benchmarks. The paper suggests that the quality of code synthesis benchmarks should be carefully examined and ensured to prevent inaccuracies from affecting the evaluation results. 5. **Consistent Best-Performing Temperatures:** While the optimal temperatures for different k values varied slightly after using HUMANEVAL+, they generally remained consistent across different k values. This observation suggests that temperature remains an important factor in LLM performance and that future research should investigate its impact on the robustness of evaluation tests, especially those targeting edge cases. Overall, the study provides strong evidence that existing code synthesis benchmarks are insufficient and that a more rigorous evaluation framework like EvalPlus is essential for accurately assessing the performance and correctness of LLM-generated code. The findings also emphasize the need for robust evaluation procedures to avoid misleading conclusions about the capabilities of LLMs.

The paper's findings have significant implications for the field of LLM-based code generation. The widespread adoption of LLMs for code generation necessitates reliable evaluation methods that can accurately assess the correctness and performance of these models. The study's demonstration of the insufficiency of existing code benchmarks underscores the importance of developing robust and automated evaluation frameworks like EvalPlus. The research provides a more realistic assessment of LLM-generated code by revealing the prevalence of undetected errors and highlighting the impact of test insufficiency on LLM rankings. The authors' discovery of inaccuracies in the ground-truth implementations of HUMANEVAL further emphasizes the need for careful validation and quality assurance of benchmarks used for evaluating LLMs. The findings of this study encourage future research to address the challenge of generating robust and comprehensive tests for code synthesis. The paper's approach of combining LLM-based and mutation-based test input generation offers a promising direction for developing effective test generation techniques. The research also opens up opportunities for further investigation into the role of temperature in LLM performance and its impact on the robustness of evaluation tests. By addressing these challenges, researchers can develop more reliable and accurate methods for evaluating LLM-generated code, contributing to the development of trustworthy and robust AI-assisted software development tools.

This paper introduces EvalPlus, a rigorous evaluation framework for program synthesis, driven by automated test generation. EvalPlus effectively leverages LLM-based and mutation-based strategies to generate diverse test inputs, resulting in significantly improved code benchmarks. The framework's effectiveness is demonstrated by its application to HUMANEVAL, producing HUMANEVAL+ and HUMANEVAL+-MINI, which uncover previously undetected errors and provide more accurate assessments of LLM performance. The findings highlight the inadequacy of existing code synthesis benchmarks and emphasize the importance of robust evaluation for trustworthy AI-assisted code generation. Future research directions include applying EvalPlus to other code benchmarks and exploring its integration with formal verification and validation techniques. The core test generation technique can also be applied to assist developers in identifying potential flaws in LLM-generated code during pair-programming.

The study primarily focuses on the HUMANEVAL benchmark, which limits the generalizability of the findings to other code generation benchmarks. The paper acknowledges the need for further exploration of EvalPlus's effectiveness on other datasets. The evaluation relies on differential testing, which might not be suitable for all programming tasks. Furthermore, the study only considers a specific set of LLMs, and the findings might not generalize to all LLMs. Additional research is needed to investigate the performance of EvalPlus on a broader range of LLMs and code generation benchmarks. The paper also highlights the need for further research to address the challenges of generating robust and comprehensive tests for code synthesis, particularly in the context of dynamically-typed languages.