Pith. sign in

REVIEW 3 major objections 8 minor 161 references

Reviewed by Pith at T0; open to challenge.

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Only 6% of LLM Code Benchmarks Show Real Speed Gains

2026-07-09 04:51 UTC pith:ZVBRUEUO

load-bearing objection Solid benchmark audit with a useful but imperfect test-generation framework; the core finding holds, the remediation has a validation gap. the 3 major comments →

arxiv 2607.07619 v1 pith:ZVBRUEUO submitted 2026-07-08 cs.SE

Rethinking Code Performance Benchmarks for LLMs

Nhat Minh Le , Yisen Xu , Zhijie Wang , Tse-Hsun (Peter) Chen This is my paper
classification cs.SE
keywords LLM code generationperformance benchmarksEffiBenchEnamelEvalPerfMercuryMann-Whitney U testmulti-agent framework
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that four popular benchmarks for evaluating whether LLMs generate efficient code are fundamentally unreliable under their default settings. The authors re-evaluated 1,538 tasks from EffiBench, Enamel, EvalPerf, and Mercury using 30 repeated runs per task and Mann-Whitney U statistical testing (p<0.05). They found that only 6.11% of benchmark-provided performant implementations are actually statistically faster than canonical baseline solutions. Through manual analysis of 308 non-significant tasks, they identified two root causes: (1) about 21% of performant implementations contain no meaningful performance change (e.g., refactoring, variable renaming), and (2) about 68% contain genuine algorithmic or data-structure improvements (e.g., O(n^3) to O(n^2)) whose benefits are invisible because the benchmark test inputs are too small to stress the performance-critical code paths. To address the second cause, the authors built a multi-agent LLM framework with three agents (generation, diagnosis, repair) that produces deterministic, performance-oriented test cases with larger and more demanding inputs. These generated tests revealed statistically significant performance improvements in 24.01% (DeepSeek-v3.1) and 25.43% (GPT-4o) of previously non-significant tasks, compared to 0.45-2% for the prior method COFFE. When applied to LLM-generated code from three models, the enhanced test suites uncovered an additional 22.19% of statistically significant performance improvements that original tests missed.

Core claim

The central discovery is that the failure of existing LLM code-performance benchmarks to show efficiency gains is primarily a measurement and test-design problem, not solely a model-capability problem. The benchmark-provided performant implementations---which are supposed to demonstrate that a task has optimization opportunities---are statistically indistinguishable from baseline solutions 93.89% of the time under rigorous 30-run statistical testing. This is because test suites designed for functional correctness use inputs too small to expose asymptotic differences between algorithms (e.g., an input of length 6 cannot distinguish O(n^3) from O(n^2)). A multi-agent framework that generates,診

What carries the argument

Multi-agent LLM test generation framework with three agents: Test Generation Agent (creates deterministic performance-stressing test inputs), Test Diagnosis Agent (analyzes failures and produces structured root-cause diagnostics), and Test Repair Agent (fixes failed tests using diagnostics). Combined with 30-run repeated execution and Mann-Whitney U statistical testing protocol.

Load-bearing premise

The paper assumes that the LLM-generated performance tests are functionally correct and deterministic. The framework validates tests only against the canonical solution, so if both the canonical solution and the generated test share the same bug, the test passes validation but produces incorrect performance measurements. This assumption is load-bearing because the entire evaluation of whether stronger tests reveal hidden performance gains depends on those tests being valid.

What would settle it

If an independent oracle or human-verified test suite found that a substantial fraction of the LLM-generated performance tests produce incorrect expected outputs or test the wrong function, the 24-25% of newly significant results could be artifacts of invalid tests rather than genuine performance differences being exposed.

If this is right

  • Published rankings of LLMs on code efficiency benchmarks may be unreliable if they use single-run or small-sample execution without statistical significance testing, as observed differences could be measurement noise rather than genuine efficiency gains.
  • Benchmark designers should select tasks with meaningful asymptotic optimization opportunities and construct test inputs large enough to expose algorithmic differences, rather than reusing correctness-oriented test suites from HumanEval/MBPP.
  • The multi-agent test generation approach could be applied to other domains where benchmark test inputs are insufficient to stress performance-critical behavior, such as database query optimization or compiler benchmarking.
  • Future performance benchmarks should extend beyond isolated function-level tasks to class-level or repository-level settings where performance bottlenecks arise from component interactions, I/O, concurrency, or long-running workloads.
  • The finding that 21% of benchmark performant implementations contain no meaningful performance change suggests existing benchmarks need data cleaning and quality auditing before they are used for evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the generated performance tests are validated only against the canonical solution (as described in Section 4.3), any bug shared between the canonical solution and the generated test would pass validation while producing incorrect performance measurements. This is a systematic blind spot: the test oracle inherits the baseline's errors.
  • The 24-25% success rate of the multi-agent framework means that 75-76% of previously non-significant tasks remain non-significant even with enhanced tests. This could indicate either that those tasks genuinely lack optimization opportunities, or that the LLM-generated tests still fail to find the right input sizes or structures to expose the performance differences---the ceiling of this approach i
  • If stronger test suites reveal that LLM-generated code is more efficient than previously thought, prior studies concluding that LLMs cannot match human-written code efficiency may need to be revisited with improved test infrastructure.
  • The framework's limitation in handling programs with multiple nested functions (Example 5) suggests that performance test generation for real-world code with complex function interfaces remains an open problem beyond the controlled benchmark setting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. This paper revisits four function-level Python performance benchmarks (EffiBench, Enamel, EvalPerf, Mercury) and evaluates whether benchmark-provided performant implementations are statistically distinguishable from canonical solutions under rigorous measurement (30 runs, Mann-Whitney U test, Cliff's delta). The key finding is that only 6.11% of 1,538 tasks show significant improvement, primarily due to insufficient test inputs. A manual analysis of 308 non-significant tasks (Cohen's Kappa 0.93) identifies two root causes: no meaningful performance change (99 tasks) and potential improvements not exposed by tests (209 tasks). The authors then propose a multi-agent LLM framework for generating performance-oriented tests, which reveals significant improvements in 24.01% (DeepSeek-v3.1) and 25.43% (GPT-4o) of previously non-significant tasks, substantially outperforming COFFE. The paper concludes with recommendations for benchmark design.

Significance. The paper makes a timely and valuable contribution to the software engineering community by systematically demonstrating that widely-used performance benchmarks for LLM-generated code often cannot reliably distinguish efficient implementations from canonical solutions. The experimental methodology is generally sound: 30-run repeated execution, non-parametric statistical testing, effect-size analysis, and a well-executed manual analysis with high inter-rater agreement (Kappa 0.93). The multi-agent test-generation framework is a practical contribution, and the replication package with generated code and measured data is publicly available. The finding that only 6.11% of benchmark-provided performant implementations are significantly faster under rigorous testing is important and actionable for benchmark designers. The comparison against COFFE provides a useful baseline. The paper also provides falsifiable predictions about which types of code changes (algorithmic vs. refactoring) are likely to be exposed by stronger test suites.

major comments (3)
  1. §4.3, validation loop: The framework validates generated tests by executing them against the canonical implementation only. This creates a self-referential oracle: if the canonical solution has a bug on certain edge-case inputs (e.g., integer overflow, hash collision patterns), the generated test may exploit that bug to create an artificial performance difference rather than measuring genuine algorithmic improvement. The paper acknowledges this risk partially in Example 5 (function misidentification) but does not systematically verify test correctness against independent oracles (e.g., the problem specification, the performant implementation, or differential testing). Since the RQ3/RQ4 claims (24-25% improvement rates) are load-bearing on these generated tests measuring genuine performance differences, the authors should either (a) validate a sample of generated tests against an oracle独立
  2. §4.3, missing test quality analysis: The paper reports that the framework generates 10 tests per task and filters out failing ones, but does not report what fraction of generated tests pass validation, how many tasks end up with zero valid tests, or the distribution of final test-suite sizes. Without these numbers, it is unclear whether the 24-25% significance rate reflects strong test quality or simply the fact that any sufficiently large input will create a runtime difference. A table reporting pass rates, final test counts, and input-size distributions would strengthen the evaluation.
  3. §4.4, RQ4 setup: The paper states that LLM-generated implementations from GPT-4o-mini, Claude-Sonnet-4.5, and Gemini-2.5-Flash are compared against canonical solutions under both original and generated test suites. However, it is unclear whether the LLM-generated implementations are first validated for functional correctness against the generated tests before performance measurement. If an LLM-generated implementation fails some generated tests, is it excluded from performance comparison? The paper should clarify the correctness filtering protocol used for RQ4.
minor comments (8)
  1. §4.2, Table 3: The 'Changes with performance impact' row lists 209 (67.86%) for the manual analysis, but the subcategory counts sum to 183 (algorithm strategy) + 26 (data structure) = 209, which is consistent. However, the LLM-as-a-Judge columns list 864 (76.06%) for DeepSeek and 840 (73.94%) for GPT-4o, but the subcategory percentages for DeepSeek sum to approximately 76.06% only if 'No changes' (6%) and 'Changes without performance impact' (17.96%) are added. The table should clarify whether the subcategory percentages are column-conditional or row-conditional.
  2. §4.2, p. 15: The text states '72.1% and 68.31% of the tasks are identified as involving algorithm strategy changes by DeepSeek-v3.1 and GPT-4o-2024-08-06, respectively.' However, Table 3 shows 76.06% and 73.94% for 'Changes with performance impact,' which includes both algorithm strategy changes and data structure replacements. The 72.1% and 68.31% figures appear to refer to algorithm strategy changes only, but this is not clearly stated.
  3. §3.2: The one-sided Mann-Whitney U test is used with the alternative hypothesis that the performant implementation is faster. The paper should clarify how tasks where the performant implementation is significantly slower are handled — are they counted in the p≥0.05 category, or separately?
  4. §4.1, Table 2: The total row shows 1,444 (93.89%) non-significant tasks, but 1,538 - 94 = 1,444 is correct. However, the effect-size column header uses 'Neg.' as an abbreviation, while the text uses 'negligible.' Consistent terminology would improve readability.
  5. Example 4 (p. 20): The generated test 'assert lengthOfLongestSubstring('ab'*500000) == 2' uses a very large input. While this effectively exposes the performance difference, the paper should discuss whether such inputs reflect realistic usage patterns or represent pathological cases that would not occur in practice.
  6. §4.3, Fig. 2: The prompt template for Test Generation contains a placeholder 'ystandalone' which appears to be a formatting artifact ('exactly y standalone'). This should be corrected.
  7. §5, p. 24: The text states 'an average of 32.66% more performance differences' are revealed by enhanced test suites, but this figure is not directly derivable from the results in Table 4 or Table 5. The paper should clarify how this number is computed or correct it.
  8. References: Several entries have formatting issues (e.g., 'art, 2026' for the replication package, 'Demˇ sar' for Demšar). These should be cleaned up to match the journal's formatting requirements.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful and constructive review. The referee identifies three major concerns: (1) the self-referential oracle problem in test validation, (2) missing test-quality metrics for the generated test suites, and (3) unclear correctness filtering for LLM-generated implementations in RQ4. All three comments are valid and actionable. We will address each through manuscript revisions and, where feasible, additional analysis.

read point-by-point responses

  1. Referee: §4.3, validation loop: The framework validates generated tests by executing them against the canonical implementation only. This creates a self-referential oracle: if the canonical solution has a bug on certain edge-case inputs, the generated test may exploit that bug to create an artificial performance difference rather than measuring genuine algorithmic improvement. The paper acknowledges this risk partially in Example 5 but does not systematically verify test correctness against independent oracles. The authors should either (a) validate a sample of generated tests against an oracle or (b) use differential testing.

    Authors: The referee raises a legitimate concern. Our current validation loop executes generated tests against the canonical implementation only, which means that if the canonical solution has a latent bug on certain edge-case inputs, a generated test could exploit that bug to produce an artificial performance difference. We agree that this is a threat to the validity of the RQ3/RQ4 results and that the manuscript does not currently address it systematically enough. In revision, we will take two steps. First, we will conduct a differential-testing validation on a random sample of at least 100 tasks from the 1,345 previously non-significant tasks: for each sampled task, we will execute the generated tests against both the canonical solution and the benchmark-provided performant implementation, and flag any test where the two implementations produce different outputs. Such cases would indicate either a genuine algorithmic difference (expected) or a test exploiting a canonical-solution bug (problematic). We will report the fraction of tests that produce divergent outputs and manually inspect those cases to determine whether they reflect genuine performance differences or oracle artifacts. Second, we will add a paragraph to Section 7 (Threats to Validity) explicitly discussing the self-referential oracle risk and the mitigation we performed. We note that the risk is partially bounded by the fact that our Stage 1 evaluation (Table 4) focuses on 209 tasks where the code changes were manually verified to have genuine performance impact, and the significance rates on this subset (44.98% and 48.80%) are consistent with the overall rates, suggesting that the results are not primarily driven by oracle artifacts. However, we agree that a systematic differential-testing check is needed and承诺 revision: yes

  2. Referee: §4.3, missing test quality analysis: The paper reports that the framework generates 10 tests per task and filters out failing ones, but does not report what fraction of generated tests pass validation, how many tasks end up with zero valid tests, or the distribution of final test-suite sizes. Without these numbers, it is unclear whether the 24-25% significance rate reflects strong test quality or simply the fact that any sufficiently large input will create a runtime difference. A table reporting pass rates, final test counts, and input-size distributions would strengthen the evaluation.

    Authors: This is a fair and important point. The manuscript currently does not report the test-generation pass rates, the distribution of final test-suite sizes, or the input-size characteristics of the generated tests. These metrics are necessary for the reader to assess whether the 24-25% significance rate reflects meaningful test quality or is an artifact of trivially large inputs. We will add a new table to Section 4.3 reporting: (1) the fraction of generated tests that pass validation on the first attempt versus after repair iterations, (2) the number of tasks that end up with zero valid tests after the iteration limit is reached, (3) the distribution of final test-suite sizes (mean, median, min, max), and (4) the distribution of input sizes in the generated tests (e.g., string lengths, list sizes) compared to the original benchmark tests. Regarding the concern that any sufficiently large input will create a runtime difference: we note that all statistically significant cases in Table 4 have large effect sizes (Cliff's delta >= 0.474), and the comparison against COFFE (which also generates large inputs but achieves only 2.00% and 0.45% significance) suggests that input size alone is not sufficient — the test must also exercise the correct performance-critical code path. We will make this argument explicit in the revised text and support it with the input-size distribution data. revision: yes

  3. Referee: §4.4, RQ4 setup: It is unclear whether the LLM-generated implementations are first validated for functional correctness against the generated tests before performance measurement. If an LLM-generated implementation fails some generated tests, is it excluded from performance comparison? The paper should clarify the correctness filtering protocol used for RQ4.

    Authors: The referee is correct that the manuscript does not clearly state the correctness filtering protocol for RQ4. To clarify: in RQ4, LLM-generated implementations are first validated for functional correctness against the benchmark-provided test suites (the same correctness check used in the original benchmarks). Only implementations that pass all benchmark-provided tests are included in the performance comparison. The generated performance-oriented tests from RQ3 are then used solely for runtime measurement, not for correctness filtering. This design choice was made to ensure consistency with the original benchmark evaluation protocols, where correctness is determined by the benchmark-provided tests and performance is measured separately. However, we agree that this should be stated explicitly. We will add a paragraph to Section 4.4 (Approach) describing the correctness filtering protocol: which tests are used for correctness validation, which are used for performance measurement, and how implementations that fail correctness checks are handled. We will also discuss in Threats to Validity the potential concern that an LLM-generated implementation could be functionally correct on the benchmark tests but incorrect on the generated performance-oriented tests, which could affect the runtime comparison. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; derivation chain is self-contained against external benchmarks, baselines, and standard statistical methods.

full rationale

The paper's main claims are derived from independent inputs: benchmark-provided implementations (external), standard statistical tests (Mann-Whitney U, Cliff's delta, Wilcoxon), and an external baseline (COFFE). RQ1 measures runtime differences using external implementations and tests with a standard protocol. RQ2 establishes categories through manual open coding (Kappa 0.93) and validates LLM-as-a-Judge against human labels (Kappa 0.72–0.75), not against itself. RQ3's multi-agent framework generates tests validated against the canonical solution for functional correctness, then independently measures execution time of both canonical and performant implementations on those tests. The skeptic's concern—that the canonical solution serves as both correctness oracle and performance baseline—is a validity threat (if canonical has a bug, tests may measure unintended behavior), but it is not circularity: the timing measurement does not reduce to the correctness validation by construction. The test provides inputs; correctness checking ensures both implementations agree on output; performance comparison measures wall-clock time. These are separate operations. The paper is transparent that tests are designed to stress performance-critical paths (prompt: 'make inefficient code run longer'), which is the stated research goal. No load-bearing self-citations are present. Replication package is publicly available. Score 1 reflects the minor validity concern about the test-validation oracle, which is acknowledged by the paper (Example 5, §7) but does not constitute circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The paper relies on standard statistical methods (Mann-Whitney U, Cliff's delta) and established thresholds from the software engineering literature. The main ad-hoc assumptions are the sufficiency of 30 runs (standard but unverified for these benchmarks), the reliability of LLM-as-a-Judge at Kappa 0.72-0.75, and the correctness of generated tests validated only against the canonical solution. The multi-agent framework is a new invented entity with independent evaluation.

free parameters (6)
  • Number of repeated executions = 30
    Chosen based on prior literature (Arcuri and Briand, 2011) as a standard threshold for statistical inference, not fitted to the data.
  • Statistical significance threshold (alpha) = 0.05
    Standard convention, not fitted.
  • Execution timeout = 300 seconds
    Chosen as a practical upper bound; not fitted but could affect results for borderline tasks.
  • Number of generated tests per task = 10
    Fixed by design (two batches of five), constrained by LLM context window. Not fitted to data but limits test suite diversity.
  • Max refinement iterations per batch = 3
    Chosen to bound computational cost; affects how many failing tests get repaired.
  • Max validation iterations = 5
    Chosen to bound computational cost; affects final test suite quality.
axioms (5)
  • standard math Mann-Whitney U test is appropriate for comparing two independent runtime samples without normality assumptions.
    §3.2, standard non-parametric test widely used in software engineering performance studies.
  • domain assumption Cliff's delta thresholds (negligible <0.147, small 0.147-0.33, medium 0.33-0.474, large ≥0.474) are meaningful for interpreting effect sizes in this context.
    §3.2, standard thresholds from Kampenes et al. 2007.
  • domain assumption 30 repeated executions provide sufficient observations for stable statistical analysis while maintaining practical computational cost.
    §3.1, following Arcuri and Briand 2011; the paper does not empirically verify that 30 is sufficient for these specific benchmarks.
  • ad hoc to paper LLM-as-a-Judge with Cohen's Kappa 0.72-0.75 (substantial agreement) is reliable enough to extend manual analysis to 1,136 tasks.
    §4.2, the paper uses this to scale labeling but the agreement level is lower than the manual inter-rater agreement (0.93).
  • ad hoc to paper Generated tests validated against the canonical solution are functionally correct for performance evaluation.
    §4.3, the framework validates tests by executing against the canonical implementation, assuming it is a correct oracle.
invented entities (1)
  • Multi-agent test-generation framework (Test Generation Agent, Test Diagnosis Agent, Test Repair Agent) independent evidence
    purpose: Generate deterministic performance-oriented tests that expose runtime differences
    The framework is evaluated against an external baseline (COFFE) and its output is tested using external statistical methods. The generated tests are available in the replication package. The framework makes falsifiable predictions about which tasks will show significant differences.

pith-pipeline@v1.1.0-glm · 30386 in / 3425 out tokens · 607274 ms · 2026-07-09T04:51:20.168886+00:00 · methodology

0 comments
read the original abstract

Many function-level performance benchmarks have been proposed to evaluate whether large language models (LLMs) can generate efficient programs. However, results on these benchmarks often show that LLM-generated implementations have little or no execution-time difference from canonical solutions. In this paper, we revisit four popular benchmarks: EffiBench, Enamel, EvalPerf, and Mercury. We evaluate 1,538 tasks under more rigorous setting by running each task 30 times and assessing the runtime differences between the canonical solutions and benchmark-provided performant implementations with statistical testing. With the benchmark-provided test suites, only 6.11% of the performant implementations are significantly faster than the canonical solutions. In a manual analysis of 308 non-significant tasks, 99 performant implementations contain no meaningful performance change, while 209 contain potential performance improvements that are not exposed by the original tests. These results suggest that the main limitation is not only the evaluation method, but also the limited sufficiency of the benchmark-provided performance tests. To address this limitation, we propose an LLM-based multi-agent framework to generate performance-oriented tests that expose runtime differences more effectively than the original tests. The framework uses three separate agents to generate, diagnose, and repair deterministic tests that preserve functional correctness while better exposing performance differences. Across 1,345 benchmark tasks for which the original tests found no significant performance difference, tests generated by our framework with DeepSeek-v3.1 and GPT-4o reveal statistically significant improvements in 24.01% and 25.43% of the tasks, respectively, outperforming the SOTA LLM-based performance test generation method.

Figures

Figures reproduced from arXiv: 2607.07619 by Nhat Minh Le, Tse-Hsun (Peter) Chen, Yisen Xu, Zhijie Wang.

Figure 1
Figure 1. Figure 1: Overview of our multi-agent framework for generating and repairing [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Simplified prompt templates used by the Test Generation, Test Diag [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

161 extracted references · 161 canonical work pages · 25 internal anchors

  1. [1]

    Breakthroughs in statistics: Methodology and distribution , pages=

    On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection , author=. Breakthroughs in statistics: Methodology and distribution , pages=. 1992 , publisher=

    work page 1992

  2. [2]

    Sebastian Baltes and Paul Ralph , title =. Empir. Softw. Eng. , volume =. 2022 , url =. doi:10.1007/S10664-021-10072-8 , timestamp =

    work page doi:10.1007/s10664-021-10072-8 2022

  3. [3]

    Educational and psychological measurement , volume=

    A coefficient of agreement for nominal scales , author=. Educational and psychological measurement , volume=. 1960 , publisher=

    work page 1960

  4. [4]

    Biometrika , pages=

    The probable error of a mean , author=. Biometrika , pages=. 1908 , publisher=

    work page 1908

  5. [5]

    Breakthroughs in statistics: Methodology and distribution , pages=

    Statistical methods for research workers , author=. Breakthroughs in statistics: Methodology and distribution , pages=. 1970 , publisher=

    work page 1970

  6. [6]

    X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling , author=. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , volume=. 1900 , publisher=

    work page 1900

  7. [7]

    IOWA , author=

    Statistical methods. IOWA , author=. Iowa State University Press. Starkstein, SE, & Robinson, RG (1989). Affective disorders and cerebral vascular disease. The British Journal of Psychiatry , volume=

    work page 1989

  8. [8]

    Biometrika , volume=

    The generalization of ‘STUDENT'S’problem when several different population varlances are involved , author=. Biometrika , volume=. 1947 , publisher=

    work page 1947

  9. [9]

    2009 , publisher=

    Introduction to the Practice of Statistics , author=. 2009 , publisher=

    work page 2009

  10. [10]

    Information and Software Technology , volume=

    A systematic review of effect size in software engineering experiments , author=. Information and Software Technology , volume=. 2007 , publisher=

    work page 2007

  11. [11]

    , author=

    Dominance statistics: Ordinal analyses to answer ordinal questions. , author=. Psychological bulletin , volume=. 1993 , publisher=

    work page 1993

  12. [12]

    Biometrics bulletin , volume=

    Individual comparisons by ranking methods , author=. Biometrics bulletin , volume=. 1945 , publisher=

    work page 1945

  13. [13]

    The Journal of Nervous and Mental Disease , volume=

    Nonparametric statistics for the behavioral sciences , author=. The Journal of Nervous and Mental Disease , volume=. 1957 , publisher=

    work page 1957

  14. [14]

    Devanbu and others , title =

    Toufique Ahmed and Premkumar T. Devanbu and others , title =. 22nd. 2025 , url =. doi:10.1109/MSR66628.2025.00086 , timestamp =

    work page doi:10.1109/msr66628.2025.00086 2025

  15. [15]

    CoRR , volume =

    Dawei Li and Bohan Jiang and others , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2411.16594 , eprinttype =. 2411.16594 , timestamp =

    work page doi:10.48550/arxiv.2411.16594 2024

  16. [16]

    Proceedings of the 18th

    Xiaoyu Zhang and Yishan Li and others , title =. Proceedings of the 18th. 2024 , url =. doi:10.1145/3640457.3688075 , timestamp =

    work page doi:10.1145/3640457.3688075 2024

  17. [17]

    DeepSeek Chat , howpublished =

    DeepSeek , year =. DeepSeek Chat , howpublished =

    work page

  18. [18]

    DeepSeek Model , howpublished =

    DeepSeek , year =. DeepSeek Model , howpublished =

    work page

  19. [19]

    ChatGPT , howpublished =

    OpenAI , year =. ChatGPT , howpublished =

    work page

  20. [20]

    Gemini25 , howpublished =

    Google , year =. Gemini25 , howpublished =

    work page

  21. [21]

    claudesonnet45 , howpublished =

    Anthropic , year =. claudesonnet45 , howpublished =

    work page

  22. [22]

    , author =

    An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. , author =. International Symposium on Software Testing and Analysis , month =

    work page

  23. [23]

    Journal of Machine learning research , volume=

    Statistical comparisons of classifiers over multiple data sets , author=. Journal of Machine learning research , volume=

    work page

  24. [24]

    The annals of mathematical statistics , pages=

    On a test of whether one of two random variables is stochastically larger than the other , author=. The annals of mathematical statistics , pages=. 1947 , publisher=

    work page 1947

  25. [25]

    Replication Package , year=

    work page

  26. [26]

    Prompt LLM-As-A-Judge , year=

    work page

  27. [27]

    Findings of the Association for Computational Linguistics:

    Wenhan Wang and others , editor =. Findings of the Association for Computational Linguistics:. 2025 , url =. doi:10.18653/V1/2025.FINDINGS-NAACL.197 , timestamp =

    work page doi:10.18653/v1/2025.findings-naacl.197 2025

  28. [28]

    Robust benchmarking in noisy environments

    Jiahao Chen and Jarrett Revels , title =. CoRR , volume =. 2016 , url =. 1608.04295 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  29. [29]

    Proceedings of the 19th Annual International Conference on Supercomputing,

    Dan Tsafrir and Yoav Etsion and others , title =. Proceedings of the 19th Annual International Conference on Supercomputing,. 2005 , url =. doi:10.1145/1088149.1088190 , timestamp =

    work page doi:10.1145/1088149.1088190 2005

  30. [30]

    Mogul and Anita Borg , title =

    Jeffrey C. Mogul and Anita Borg , title =. 1991 , url =. doi:10.1145/106972.106982 , timestamp =

    work page doi:10.1145/106972.106982 1991

  31. [31]

    Tyson and Matthew K

    Gary S. Tyson and Matthew K. Farrens , title =. Int. J. Parallel Program. , volume =. 1996 , url =. doi:10.1007/BF03356746 , timestamp =

    work page doi:10.1007/bf03356746 1996

  32. [32]

    Tyson and others , title =

    Matt Postiff and Gary S. Tyson and others , title =. J. Instr. Level Parallelism , volume =. 1999 , url =

    work page 1999

  33. [33]

    1999 , url =

    Eric Rotenberg and Steve Bennett and others , title =. 1999 , url =. doi:10.1109/12.752652 , timestamp =

    work page doi:10.1109/12.752652 1999

  34. [34]

    The Twelfth International Conference on Learning Representations,

    Alexander Shypula and Aman Madaan and others , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    work page 2024

  35. [35]

    Yun Peng and Jun Wan and others , title =. Proc. 2025 , url =. doi:10.1145/3715727 , timestamp =

    work page doi:10.1145/3715727 2025

  36. [36]

    NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness

    Manav Singhal and Tushar Aggarwal and others , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2401.15963 , eprinttype =. 2401.15963 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2401.15963 2024

  37. [37]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  38. [38]

    Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , pages=

    Towards the use of the readily available tests from the release pipeline as performance tests: Are we there yet? , author=. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , pages=

    work page

  39. [39]

    IEEE Transactions on Software Engineering , volume=

    PerfJIT: Test-level just-in-time prediction for performance regression introducing commits , author=. IEEE Transactions on Software Engineering , volume=. 2020 , publisher=

    work page 2020

  40. [40]

    IEEE Transactions on Software Engineering , volume=

    A large-scale empirical study of real-life performance issues in open source projects , author=. IEEE Transactions on Software Engineering , volume=. 2022 , publisher=

    work page 2022

  41. [41]

    Proceedings of the 38th international conference on software engineering , pages=

    Performance issues and optimizations in javascript: an empirical study , author=. Proceedings of the 38th international conference on software engineering , pages=

    work page

  42. [42]

    Journal of Systems and Software , volume=

    A case study on the stability of performance tests for serverless applications , author=. Journal of Systems and Software , volume=. 2022 , publisher=

    work page 2022

  43. [43]

    Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , pages=

    Trace-based multi-dimensional root cause localization of performance issues in microservice systems , author=. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , pages=

    work page

  44. [44]

    IEEE Access , volume=

    Detecting performance bottlenecks guided by resource usage , author=. IEEE Access , volume=. 2019 , publisher=

    work page 2019

  45. [45]

    IEEE Access , volume=

    Understanding and statically detecting synchronization performance bugs in distributed cloud systems , author=. IEEE Access , volume=. 2019 , publisher=

    work page 2019

  46. [46]

    2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages=

    Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models , author=. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages=. 2023 , organization=

    work page 2023

  47. [47]

    Information and Software Technology , volume=

    A3test: Assertion-augmented automated test case generation , author=. Information and Software Technology , volume=. 2024 , publisher=

    work page 2024

  48. [48]

    2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages=

    CAT-LM training language models on aligned code and tests , author=. 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages=. 2023 , organization=

    work page 2023

  49. [49]

    Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages=

    Domain adaptation for code model-based unit test case generation , author=. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages=

    work page

  50. [50]

    IEEE Transactions on Software Engineering , volume=

    An empirical evaluation of using large language models for automated unit test generation , author=. IEEE Transactions on Software Engineering , volume=. 2023 , publisher=

    work page 2023

  51. [51]

    Proceedings of the ACM on Software Engineering , volume=

    Evaluating and improving chatgpt for unit test generation , author=. Proceedings of the ACM on Software Engineering , volume=. 2024 , publisher=

    work page 2024

  52. [52]

    Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis , pages=

    Perffuzz: Automatically generating pathological inputs , author=. Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis , pages=

    work page

  53. [53]

    Proceedings of the 2017 ACM SIGSAC conference on computer and communications security , pages=

    Slowfuzz: Automated domain-independent detection of algorithmic complexity vulnerabilities , author=. Proceedings of the 2017 ACM SIGSAC conference on computer and communications security , pages=

    work page 2017

  54. [54]

    Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings , pages=

    Pynguin: Automated unit test generation for python , author=. Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings , pages=

    work page

  55. [55]

    Dong Huang and Jianbo Dai and others , title =. Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024 , year =

    work page 2024

  56. [56]

    Forty-second International Conference on Machine Learning,

    Dong Huang and Guangtao Zeng and others , title =. Forty-second International Conference on Machine Learning,. 2025 , url =

    work page 2025

  57. [57]

    LLM4EFFI: Leveraging Large Language Models to Enhance Code Efficiency and Correctness

    Tong Ye and Weigang Huang and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2502.18489 , eprinttype =. 2502.18489 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.18489 2025

  58. [58]

    Towards Better Correctness and Efficiency in Code Generation

    Yunlong Feng and Yang Xu and others , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2508.20124 , eprinttype =. 2508.20124 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.20124 2025

  59. [59]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),

    Siming Huang and Tianhao Cheng and others , title =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2025 , url =

    work page 2025

  60. [60]

    2025 , url =

    Ramya Jonnala and Jeong Yang and others , title =. 2025 , url =. doi:10.1109/ACCESS.2025.3585742 , timestamp =

    work page doi:10.1109/access.2025.3585742 2025

  61. [61]

    Ensuring Functional Correctness of Large Code Models with Selective Generation , author=

    work page

  62. [62]

    Yuxiang Wei and Federico Cassano and others , title =. Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024 , year =

    work page 2024

  63. [63]

    2025 , booktitle=

    Do LLMs Understand Code Preference? Training Code Preference Models via Synthetic Code Evolution , author=. 2025 , booktitle=

    work page 2025

  64. [64]

    Simon , title =

    Herbert A. Simon , title =. J. 1963 , url =. doi:10.1145/321186.321192 , timestamp =

    work page doi:10.1145/321186.321192 1963

  65. [65]

    Shaw and others , title =

    David E. Shaw and others , title =. Advance Papers of the Fourth International Joint Conference on Artificial Intelligence, Tbilisi, Georgia, USSR, September 3-8, 1975 , pages =. 1975 , url =

    work page 1975

  66. [66]

    Proceedings of the 38th

    Sumit Gulwani , title =. Proceedings of the 38th. 2011 , url =. doi:10.1145/1926385.1926423 , timestamp =

    work page doi:10.1145/1926385.1926423 2011

  67. [67]

    Waldinger and Richard C

    Richard J. Waldinger and Richard C. T. Lee , title =. Proceedings of the 1st International Joint Conference on Artificial Intelligence, Washington, DC, USA, May 7-9, 1969 , pages =. 1969 , url =

    work page 1969

  68. [68]

    Waldinger , title =

    Zohar Manna and Richard J. Waldinger , title =. Commun. 1971 , url =. doi:10.1145/362566.362568 , timestamp =

    work page doi:10.1145/362566.362568 1971

  69. [69]

    Cordell Green , title =

    C. Cordell Green , title =. Proceedings of the 1st International Joint Conference on Artificial Intelligence, Washington, DC, USA, May 7-9, 1969 , pages =. 1969 , url =

    work page 1969

  70. [70]

    6th International Conference on Learning Representations,

    Ashwin Kalyan and Abhishek Mohta and others , title =. 6th International Conference on Learning Representations,. 2018 , url =

    work page 2018

  71. [71]

    Yu, Zishun and Tao, Yunzhe and others , journal=

    work page

  72. [72]

    Foundations and Trends

    Program synthesis , author=. Foundations and Trends. 2017 , publisher=

    work page 2017

  73. [73]

    Competition-Level Code Generation with AlphaCode

    Yujia Li and others , title =. CoRR , volume =. 2022 , url =. doi:10.48550/ARXIV.2203.07814 , eprinttype =. 2203.07814 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2203.07814 2022

  74. [74]

    CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

    Codegen: An open large language model for code with multi-turn program synthesis , author=. arXiv preprint arXiv:2203.13474 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  75. [75]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    Yue Wang and others , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,. 2023 , url =. doi:10.18653/V1/2023.EMNLP-MAIN.68 , timestamp =

    work page doi:10.18653/v1/2023.emnlp-main.68 2023

  76. [76]

    Raymond Li and others , title =. Trans. Mach. Learn. Res. , volume =. 2023 , url =

    work page 2023

  77. [77]

    Aakanksha Chowdhery and others , title =. J. Mach. Learn. Res. , volume =. 2023 , url =

    work page 2023

  78. [78]

    InCoder: A Generative Model for Code Infilling and Synthesis

    Incoder: A generative model for code infilling and synthesis , author=. arXiv preprint arXiv:2204.05999 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  79. [79]

    SantaCoder: don't reach for the stars!

    Loubna Ben Allal and others , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2301.03988 , eprinttype =. 2301.03988 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2301.03988 2023

  80. [80]

    Language Models are Few-Shot Learners

    Tom B. Brown and others , title =. CoRR , volume =. 2020 , url =. 2005.14165 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2020

Showing first 80 references.