Pith. sign in

REVIEW 3 major objections 7 minor 51 references

Re-Evaluating Code LLM Benchmarks Under Semantic Mutation

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Semantically similar prompt template mutations cause large swings in code-benchmark scores and can reorder model rankings, so single-template evaluations are unreliable.

desk verdict A solid empirical study showing code-benchmark scores and model rankings shift substantially under semantically similar prompt mutations; the main caveat is that 'semantically similar' rests on a validation pipeline that would be much stronger with released templates. read the letter →

arxiv 2506.17369 v1 pith:MD3HXVF4 submitted 2025-06-20 cs.SE cs.AI

Zhiyuan Pan , Xing Hu , Xin Xia , Xiaohu Yang This is my paper · ORCID
classification cs.SEcs.AI
keywords promptsensitivitycodebenchmarksLLMevaluationsemanticmutationmodelrankingtemplatesbenchmarkrobustnessempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Code benchmarks typically score each model with a single prompt template per task. This paper argues that the choice of that template matters: across 100 semantically similar mutations per task, eight code tasks, and ten open-source models, 24% of model-task pairs moved at least one standard deviation from the mean, and 39% showed a possible improvement above 10% over the original template. For leaderboards, the mutations also reorder models, especially within the same family, and the templates that make one model look best barely overlap with those that make another look best. The upshot is that a benchmark's reported rankings may partly reflect arbitrary wording, so single-template evaluation does not reliably estimate a model's true capabilities.

What carries the argument

The central object is the prompt-template syntax tree, built by parsing a benchmark's prompt into Text, Format, Tag, and Delimiter nodes. Mutations are atomic operations that modify only existing nodes — paraphrasing text, paraphrasing tags, changing tag casing, changing format strings, or changing delimiters — so the overall structure is preserved by construction. An LLM proposes varied argument values for these operations, a refinement step repairs references between nodes (for example, renaming a tag without updating the text that mentions it), and validation rejects mutations whose embedding similarity falls below a cosine threshold of 0.85. The resulting set of 100 templates per task turns 'prompt sensitivity' into a measurable quantity: the spread of metric values across templates is the evidence.

What would settle it

Have independent raters blind-compare a sample of the 100 mutated templates against the original for each task and label any that change meaning, hint at answer format, or bias toward particular test cases. If a substantial share (e.g., more than a few percent) are judged meaning-altering, or if the size of the performance swings fails to correlate with template quality as judged by those raters, the observed variation would be explained by semantic drift rather than prompt sensitivity.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that prompt sensitivity is a common and quantifiable property of modern code benchmarks: minor, meaning-preserving edits to a task's prompt template produce substantial shifts in both absolute and relative model performance. The evidence is a controlled mutation pipeline in which the original prompt is parsed into a syntax tree, changed only at existing nodes through five operation types, and validated so that wording, formatting, tags, and delimiters stay semantically and structurally close to the original. Repeating this for eight tasks and ten models, the paper finds that 24% of model-task pairs show $|Z|>1$, 39% show a maximum possible performance improvement above 10%, Kendall's W drops below the 0.85 strong-agreement threshold on several tasks (most sharply within model families), and the top-performing templates for different models are nearly disjoint. The paper concludes that relying on a single arbitrary prompt template can fail to provide robust capability estimates and robust capability rankings across models.

Load-bearing premise

The load-bearing premise is that the mutated prompts really do mean the same thing and follow the same structure as the original; if the mutations quietly change what answer format is suggested or nudge models toward certain outputs, the measured shifts are semantic drift rather than sensitivity. That premise rests on the authors' manual syntax-tree decomposition, a cosine-similarity cutoff chosen and applied by the authors, and a manual check, none of which was independently verified.

Editorial extensions

If this is right

  • A benchmark that reports only one prompt template per task gives a noisy estimate of a model's absolute ability; reporting the mean, spread, and worst case over several semantically equivalent templates would be a direct remedy.
  • Leaderboards that compare models within the same family should be treated as provisional, since within-family rankings shifted substantially across templates on several tasks.
  • Small wording changes such as capitalization or delimiter style can move scores by more than 10% on some tasks, so reported prompt-engineering gains need to be measured against this baseline variability.
  • Because top-performing templates barely overlap across models, choosing one template after seeing model outputs can systematically favor some models over others.
  • Benchmark designers should report sensitivity statistics, such as Kendall's W across a validation set of templates, before claiming that a benchmark measures a stable capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mutation suite could be reused as a benchmark-hardening check: a proposed benchmark could be required to show stable rankings across, say, twenty semantically equivalent templates before release.
  • The low cross-model overlap of top templates suggests prompt choice interacts with model-specific training; that makes per-model prompt optimization a legitimate part of evaluation, at the cost of making cross-model comparison harder.
  • An untested extension is whether template stability predicts other reliability properties, such as a model's consistency across rephrased inputs or across random seeds; the data here do not address that.
  • The framework only covers benchmarks where a single model answers alone; multi-LLM interaction benchmarks could show even stronger sensitivity, since one agent's prompt-induced variation could cascade through the interaction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. Pan et al. study prompt sensitivity of code LLM benchmarks. They propose a mutation framework that parses a benchmark's prompt template into a syntax tree with text, format, tag, and delimiter node types; defines five atomic operation types; uses GPT-4o to generate operation arguments; refines inconsistent node references; and validates mutations with rule-based checks including a cosine-similarity filter (C2). Applying this to eight tasks from CRUXEval, TestEval, and CoderUJB, they generate 100 prompt templates per task and evaluate 10 open-source LLMs, with smaller supplementary studies on GPT-3.5-Turbo, GPT-4o-mini, and DeepSeek-R1-Distill-Qwen-32B. Using Z-scores, maximum performance improvement (MPI), Kendall's W, and top-k Intersection-over-Union, they report that semantically similar prompt mutations cause substantial variation in absolute scores (24% of model-task pairs with |Z| > 1, 39% with MPI > 10%), that sensitivity is not reliably predicted by model capability, and that rankings (especially within model families) and top-prompt transferability are unstable. They conclude that single-template code benchmarks may not provide robust absolute or relative evaluations and recommend multi-prompt evaluation.

Significance. If the central premise holds, this is a timely and substantial empirical contribution: it extends prompt-sensitivity research from NLP benchmarks to code benchmarks with structured, referential prompts and execution-based metrics, at a scale (eight tasks, 100 templates each, ten models, roughly 1.84B input tokens) that makes the headline statistics falsifiable and re-testable on other benchmarks. The framework is a reusable methodological template, and the two-way ANOVA control study (Section 6.1) is a genuine strength, showing a statistically significant template effect even at high temperature. I do not see circularity: the outcome variables are externally defined benchmark scores, and the mutation framework is a measurement instrument rather than a fitted target. The main weakness is that the instrument's validity is asserted rather than independently verified: the semantic-equivalence premise rests on a thresholded embedding filter that covers only long text paraphrases and on an author-only manual check, and the mutated templates are not released. This is a reproducibility gap that should be fixed before publication rather than an internal inconsistency.

major comments (3)
  1. [Sec. 3.3.3 (C2), Sec. 4.2.2, Sec. 6.3] The load-bearing premise that the 100 mutated templates per task preserve the original semantics and structure rests on the manually constructed syntax trees (Section 3.2), the cosine-similarity filter C2 (Section 3.3.3), which applies only to paraphrases of text nodes longer than 10 words, and the manual verification in Section 4.2.2, which inspects paraphrase operation arguments but reports neither the number of items checked nor any inter-rater reliability. Format, tag, delimiter, casing, and short-paraphrase operations receive no automatic semantic check, and the cumulative effect of a sequence of individually similar operations on the final template is not validated; Section 6.3 does not list semantic equivalence among the threats to validity. The manuscript provides no access to the 800 generated prompt templates (100 per task) or the per-task meta-templates, so an independent reader cannot verify the central premise, and if subtle meaning drift (e.g., answer-format hints or shifts of emphasis) slipped through, the observed variation would reflect semantic change rather than sensitivity to semantically equivalent wording. I ask the authors to release the artifacts, to have multiple annotators judge semantic equivalence on a representative sample with reported agreement, to check all operation types automatically and report the similarity-score distribution rather than a single 0.85 threshold, or, as an alternative, to restrict the claims to form-preserving mutations and state that limitation.
  2. [Sec. 4.2.4, Fig. 7a] The Z-score Z=(x0-x)/sigma defined in Section 4.2.4 is computed with the original template included in the mean x and the standard deviation sigma over all 100 templates, so a large |Z| measures the position of the original prompt within the template-level distribution rather than the magnitude of the variation across templates. Under an exchangeable null in which a randomly chosen template is designated as the original, roughly 32% of the 80 model-task cells would be expected to show |Z| > 1, so the reported 24% in Figure 7a is not evidence of unusual sensitivity and is actually below that expectation; Finding 1 should instead be supported by comparing the spread and MPI values against a noise or permutation baseline, and by reporting the sign and effect size of the deviations. Relatedly, the MPI in Figure 7b is a maximum over 99 candidates and is reported without uncertainty bounds or multiple-comparison awareness, which is especially relevant for the CRUXEval tasks whose pass@5 values are noisy estimates from 10 generations at temperature 0.8.
  3. [Sec. 5.2.2, Table 8] The IoU values in Table 8 are described as consistently low, but no chance baseline is provided: for independent random k-subsets of 100 templates the expected IoU is about 0.01 for k=1, 0.026 for k=5, 0.053 for k=10, and 0.11 for k=20, and several reported means are close to or only modestly above these values (e.g., T3 at k=5 is 0.02 and T1 at k=5 is 0.05). Because the top-k sets for T1 and T2 are defined from noisy pass@5 estimates, the near-chance overlap could reflect selection noise rather than a genuine model-template interaction, so I recommend reporting the chance IoU for each k, adding bootstrap intervals or restricting the analysis to the deterministic tasks (T3-T8), and then re-assessing the wording of Finding 4.
minor comments (7)
  1. [Sec. 6.2] In Section 6.2, the nine mutated templates used for the proprietary and reasoning models are selected by a diversity rule that sorts templates by the performance deviation averaged over the ten open-source models and picks the largest deviations, so the Z-scores and MPI values in Table 10 are likely inflated relative to what a random sample of templates would yield; the case study should either be framed as an upper-bound estimate or the selection bias should be disclosed alongside the results.
  2. [Table 6] Table 6 reports Pearson correlation coefficients computed over only ten models per task without p-values or confidence intervals; with n=10 the confidence intervals are very wide (e.g., the 95% interval for r=0.71 spans roughly 0.1 to 0.9), so the labels strong, weak, and negligible used in Section 5.1.3 should be supported by interval estimates or at least by p-values.
  3. [Sec. 1, Contributions] The contribution statement claims the first comprehensive analysis of prompt sensitivity in software engineering, but Section 2.2 positions the work only against NLP prompt-sensitivity studies; the authors should survey recent work on prompt robustness of code-generation benchmarks and either cite the closest work or hedge the novelty claim accordingly.
  4. [Fig. 7] The color bars in Figure 7 appear to end well below the maximum cell values in the heatmaps (panel (a) ends at 1.4 while the T2/M4 cell is 2.80, and panel (b) ends at 14 while several cells exceed 100), so the visual encoding truncates the largest values; the scale limits and any intentional truncation should be stated in the caption.
  5. [Sec. 6.1] In Section 6.1, observation 2 concludes that temperature has a higher influence than prompt template mutation by comparing F-values of the two factors, but F-statistics with different degrees of freedom are not directly comparable; reporting effect sizes such as partial eta-squared would make this comparison and the conclusion defensible.
  6. [Sec. 6.3.1] Section 6.3.1 states that only a few unhandled postprocessing edge cases were identified and fixed, but the number of affected instances across the eight tasks is not reported; a brief quantification would let readers assess the risk that postprocessing artifacts contribute to the template-level differences.
  7. [Table 7] In Section 5.2.1, the Kendall's W values for All Models are at or above 0.85 for six of eight tasks (and 0.83 for a seventh), which by the paper's own threshold means strong agreement, so the conclusion that rankings are prone to mutations rests largely on T7 and on the within-family rows; the discussion should make this asymmetry explicit and should also report whether the within-family W drops are driven by a few templates or by a broad pattern.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the observed performance metrics are external measurements, and the semantic-equivalence premise is a validity threat rather than a circular reduction.

full rationale

This paper is an empirical measurement study rather than a derivation. The outcome variables (pass@5, test pass rate, accuracy, compile@1) are produced by external benchmark harnesses, and no result is computed from fitted constants or from the mutation framework itself. The mutation pipeline (manual syntax-tree decomposition, LLM-generated atomic operations, the C2 embedding filter, and the manual check in Section 4.2.2) is an instrument for constructing the treatment set; the conclusions about performance variation are obtained by actually running ten models and reading the resulting scores. I find no equation or claim in which a predicted quantity is equivalent by construction to an input: Z-score, MPI, Kendall's W, and IoU are descriptive or rank-based statistics of the observed score distributions. The semantic-similarity premise (R1/R2, Section 3.2, C2 in Section 3.3.3, Section 4.2.2) is an auxiliary assumption about the treatment, not the conclusion. The closest thing to a concern is epistemic rather than circular: semantic equivalence for short paraphrases, tag changes, delimiters, and casing is asserted via manual inspection and a cosine-similarity threshold, with no inter-rater reliability and with mutated templates not released, so an external reader cannot independently check for subtle semantic drift. If some mutations changed meaning, the observed variation would reflect semantic shift rather than prompt sensitivity. That is a validity threat, not a reduction of the result to its inputs. The only self-citation, reference [8], appears as background related work and is not load-bearing for the central claim. The paper is therefore not circular; any weakness lies in the defensibility of the semantic-equivalence assumption, which is a correctness risk rather than a circularity risk.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The study measures external benchmark outcomes, so the ledger is light. The main invented construct is the mutation framework itself, but it is a measurement instrument, not a fitted entity. The cosine threshold is a hand-set parameter governing what counts as semantically similar, so it is the most load-bearing free parameter.

free parameters (1)
  • Cosine similarity threshold for semantic preservation = 0.85
    Hand-set validation threshold (Section 3.3.3, C2) used to reject paraphrases that shift meaning. It directly controls which mutations count as semantically similar, so it shapes the entire measurement. It is justified only by a general reference [41].
assumptions (4)
  • domain assumption The syntax-tree decomposition of a prompt template captures the structural components that matter for prompt sensitivity (text, format, tags, delimiters).
    Section 3.2.1 defines four node types and asserts they represent prompt structure. If the decomposition misses relevant components, mutations may not explore the real prompt space.
  • ad hoc to paper Atomic operations that only modify existing syntax-tree nodes preserve the original prompt structure and semantics (R1, R2).
    Sections 3.2.2 and 3.3 define mutations as node edits without adding or deleting nodes, and assume sequences of such edits remain semantically similar after refinement and validation.
  • domain assumption The adapted postprocessing and evaluation code produces correct pass and fail judgments across all mutated templates and models.
    Section 6.3.1 describes manual sampling of postprocessed generations and fixing unhandled edge cases, but the authors admit it may not work for new models or templates; rare failures are attributed to models and not manually fixed.
  • standard math Standard statistical measures (Z-score, Kendall's W, Pearson's r, two-way ANOVA) apply appropriately to the collected benchmark scores.
    Sections 4.2.4, 4.2.5 and 6.1 use these statistics in their standard forms; no unusual distributional claims are made beyond the p-values reported for ANOVA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Re-Evaluating Code LLM Benchmarks Under Semantic Mutation." pith.science (2026). https://pith.science/paper/MD3HXVF4

@misc{pith2026250617369,
  author       = {Pith},
  title        = {Pith review of: Re-Evaluating Code LLM Benchmarks Under Semantic Mutation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MD3HXVF4}},
  note         = {Machine review of arXiv:2506.17369}
}
read the original abstract

In the era of large language models (LLMs), code benchmarks have become an important research area in software engineering and are widely used by practitioners. These benchmarks evaluate the performance of LLMs on specific code-related tasks, such as code understanding and generation. A critical step in constructing code benchmarks is the design of prompts. However, as existing code benchmarks typically rely on a single prompt template per task, they are prone to the issue of prompt sensitivity, where minor prompt variations could result in substantial performance variations, leading to unreliable evaluations of model capabilities. While previous studies have explored prompt sensitivity, their experimental designs and findings are limited to traditional natural language processing (NLP) tasks. In this paper, we present an empirical study to investigate prompt sensitivity in code benchmarks. We first propose a general framework that modifies prompt templates in a manner that preserves both their semantics and their structure as much as possible. Based on the framework, we conduct extensive experiments across eight code benchmark tasks on 10 representative open-source LLMs, with each task featuring 100 semantically similar prompt templates. We then analyze the evaluation results using various statistical metrics, focusing on both absolute and relative model performance. Our findings suggest that even slight prompt variations can lead to significant shifts in performance. Additionally, we observe that such variations can introduce inconsistencies in the performance rankings across different models. These insights highlight the need for considering prompt sensitivity when designing future code benchmarks, to ensure more reliable and accurate evaluation of LLM capabilities.

Figures

Figures reproduced from arXiv: 2506.17369 by the authors.

Figure 1
Figure 1. Distribution of evaluation results for CRUXEval [12] with 100 semantically similar prompt templates on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our study design. ❶ and ❷ serve as the data preparation phase. ➀ to ➄ represent the execution steps. For the sake of clarity, we first provide definitions of key terms used in our methodology: • Prompt Template: A template string for generating prompts. In our study, a benchmark task corresponds to a prompt template. A prompt is instantiated by filling the prompt template with benchmark data. • Syntax Tr… view at source ↗
Figure 3
Figure 3. An example of a prompt template (partial; simplified for demonstration) in CRUXEval benchmark, and its [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Examples of atomic operations and the resulting prompt template (corresponding to Figure 3) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Prompt templates for generating atomic operations, used in the mutation process [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: An example of semantic correlations between syntax tree nodes and the refinement after mutation [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Quantitative Results of RQ1. Darker cells indicate a larger magnitude of prompt sensitivity. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 36 canonical work pages

  1. [1]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024

    2024

  2. [2]

    The llama 3 herd of models, 2024

    Aaron Grattafiori et al. The llama 3 herd of models, 2024

    2024

  3. [3]

    Qwen technical report, 2023

    Jinze Bai et al. Qwen technical report, 2023

    work page 2023

  4. [4]

    Code llama: Open foundation models for code, 2024

    Baptiste Rozi `ere et al. Code llama: Open foundation models for code, 2024

    work page 2024

  5. [5]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  6. [6]

    Unit test case generation with transformers and focal context

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617, 2020

    arXiv 2009

  7. [7]

    Codexglue: A machine learning benchmark dataset for code understanding and generation, 2021

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding a...

    work page 2021

  8. [8]

    Reasoning Runtime Behavior of a Pro- gram with LLM: How Far Are We?

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. Reasoning Runtime Behavior of a Pro- gram with LLM: How Far Are We? . In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 140–152, Los Alamitos, CA, USA, May 2025. IEEE Computer Society

    work page 2025

Show all 51 references
  1. [9]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  2. [10]

    Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation, 2023

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation, 2023

    2023

  3. [11]

    Testeval: Benchmarking large language models for test case generation, 2025

    Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. Testeval: Benchmarking large language models for test case generation, 2025

    2025

  4. [12]

    CRUXEval: A benchmark for code reasoning, understanding and execution

    Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, ...

    2024

  5. [13]

    Lyu, and Shing-Chi Cheung

    Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, Zibin Zheng, Michael R. Lyu, and Shing-Chi Cheung. How should we build a benchmark? revisiting 274 code-related benchmarks for ...

    2025

  6. [14]

    Introduction to prompting, 2025

    Google Cloud. Introduction to prompting, 2025. Last Accessed: 2025-03-04

    2025

  7. [15]

    Prompt design and engineering: Introduction and advanced methods, 2024

    Xavier Amatriain. Prompt design and engineering: Introduction and advanced methods, 2024. 16

    2024

  8. [16]

    Benchmarking knowledge boundary for large language models: A different perspective on model evaluation

    Xunjian Yin, Xu Zhang, Jie Ruan, and Xiaojun Wan. Benchmarking knowledge boundary for large language models: A different perspective on model evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Co...

    2024

  9. [17]

    Quantifying language models’ sensitivity to spu- rious features in prompt design or: How i learned to start worrying about prompt formatting

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spu- rious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024

    2024

  10. [18]

    State of what art? a call for multi-prompt LLM evaluation

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics , 12:933–949, 2024

    2024

  11. [19]

    On the worst prompt performance of large language models

    Bowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou, and Wai Lam. On the worst prompt performance of large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, ed- itors, Advances in Neural Information Processing Systems, volume 37...

    2024

  12. [20]

    LMentry: A language model benchmark of elementary language tasks

    Avia Efrat, Or Honovich, and Omer Levy. LMentry: A language model benchmark of elementary language tasks. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computa- tional Linguistics: ACL 2023, pages 10476–10501, Toronto, Canada,...

    2023

  13. [21]

    Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks

    Yizhong Wang et al. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, A...

    2022

  14. [22]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res., 2023, 2023

    2023

  15. [23]

    Challenging BIG-bench tasks and whether chain- of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch ¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain- of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Nao...

    2023

  16. [24]

    Codegemma: Open code models based on gemma, 2024

    CodeGemma Team. Codegemma: Open code models based on gemma, 2024

    2024

  17. [25]

    Qwen2.5-coder technical report, 2024

    Binyuan Hui et al. Qwen2.5-coder technical report, 2024

    2024

  18. [26]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y . Wu, Y . K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024

    2024

  19. [27]

    Automated program repair in the era of large pre- trained language models

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre- trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 1482–1494. IEEE, 2023

    2023

  20. [28]

    Repobench: Benchmarking repository-level code auto- completion systems

    Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto- completion systems. arXiv preprint arXiv:2306.03091, 2023

    2023 arXiv

  21. [29]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software ...

    2024

  22. [30]

    Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024

    2024

  23. [31]

    Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks, 2025

    Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, and David Lo. Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks, 2025

    2025

  24. [32]

    Coderujb: An executable and unified java benchmark for practical programming scenarios

    Zhengran Zeng, Yidong Wang, Rui Xie, Wei Ye, and Shikun Zhang. Coderujb: An executable and unified java benchmark for practical programming scenarios. In Proceedings of the 33rd ACM SIGSOFT International Sym- posium on Software Testing and Analysis, ISSTA 2024, page 124–136, N...

    2024

  25. [33]

    Promptbench: A unified library for evaluation of large language models

    Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. Promptbench: A unified library for evaluation of large language models. Journal of Machine Learning Research, 25(254):1–22, 2024

    2024

  26. [34]

    The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance

    Abel Salinas and Fred Morstatter. The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024 ,...

    2024

  27. [35]

    ProSA: Assessing and understanding the prompt sensitivity of LLMs

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. ProSA: Assessing and understanding the prompt sensitivity of LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP ...

    2024

  28. [36]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models

    Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analy...

    2023

  29. [37]

    Fuzz4all: Universal fuzzing with large language models

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Fuzz4all: Universal fuzzing with large language models. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New York, NY , USA, 2024. Association f...

    2024

  30. [38]

    Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries

    Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries. In Proceedings of the IEEE/ACM 46th International Conference on...

    2024

  31. [39]

    Large language model guided protocol fuzzing

    Ruijie Meng, Martin Mirchev, Marcel B¨ohme, and Abhik Roychoudhury. Large language model guided protocol fuzzing. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS), 2024

    2024

  32. [40]

    From one thousand pages of specification to unveiling hidden bugs: Large language model assisted fuzzing of matter IoT devices

    Xiaoyue Ma, Lannan Luo, and Qiang Zeng. From one thousand pages of specification to unveiling hidden bugs: Large language model assisted fuzzing of matter IoT devices. In 33rd USENIX Security Symposium (USENIX Security 24), pages 4783–4800, Philadelphia, PA, August 2024. USENI...

    2024

  33. [41]

    Similarity thresholds in retrieval-augmented generation

    Irina Radeva, Ivan Popchev, and Miroslava Dimitrova. Similarity thresholds in retrieval-augmented generation. In 2024 IEEE 12th International Conference on Intelligent Systems (IS), pages 1–7, 2024

    2024

  34. [42]

    Jiang et al

    Albert Q. Jiang et al. Mistral 7b, 2023

    2023

  35. [43]

    Ren ´e Just, Darioush Jalali, and Michael D. Ernst. Defects4j: a database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, page 437–440, New York, NY , US...

    2014

  36. [44]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  37. [45]

    M. G. KENDALL. A new measure of rank correlation. Biometrika, 30(1-2):81–93, 06 1938

    1938

  38. [46]

    Statistics (international student edition)

    David Freedman, Robert Pisani, and Roger Purves. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York, 2007

    2007

  39. [47]

    ANOVA: Repeated measures

    Ellen R Girden. ANOVA: Repeated measures. Number 84. sage, 1992

    1992

  40. [48]

    Openai o3-mini, 2025

    OpenAI. Openai o3-mini, 2025. Last Accessed: 2025-03-11

    2025

  41. [49]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  42. [50]

    deepseek-ai/deepseek-r1 - hugging face, 2025

    DeepSeek AI. deepseek-ai/deepseek-r1 - hugging face, 2025. Last Accessed: 2025-03-04. 18

    2025

  43. [2024]

    Association for Computational Linguistics

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.