Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLOVER, a new benchmark for unit-test generation, shows that even with the most informative source files selected by coverage, no evaluated language model writes tests that meet coverage requirements on more than a third of problems.

desk verdict A useful, well-scoped test-generation benchmark whose headline result depends on an unvalidated 'oracle' retriever; send it to review, but condition acceptance on an ablation and artifact release. read the letter →

arxiv 2502.08806 v1 pith:XLT4XFP2 submitted 2025-02-12 cs.SE cs.AIcs.LG

Jiacheng Xu , Bo Pang , Jin Qu , Hiroaki Hayashi , Caiming Xiong , Yingbo Zhou This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords testcasegenerationcodecoveragelong-contextlanguagemodelsbenchmarkoracleretrievalPythonunittestingpytestcontextutilization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLOVER is a benchmark for evaluating how well large language models generate and complete unit tests in realistic, long-context settings. Built from 845 problems extracted from 12 Python repositories, it spans three tasks: filling masked assertions, writing targeted tests for a given function or class, and producing tests that cover specified code blocks across multiple files. The paper constructs an oracle retrieval context from pytest coverage data to give models the most informative source files within a token budget, then measures success by executing the generated tests and checking line coverage. The central finding is that even with this oracle context, no model reaches 35% success on the hardest coverage task, with GPT-4o the best at 32.7% at 64k tokens. If correct, the benchmark exposes a real gap between current LLMs and the routine engineering task of writing coverage-complete tests.

What carries the argument

The central object is the coverage-calibrated retrieval context. For each test case, the authors run pytest-cov twice, once with the real test and once with an empty test, to obtain a coverage tensor, then classify source files into three tiers: Qrepo (coverage unchanged, deemed low-information), Qpeer (lines covered by exactly one test, deemed most informative), and Q' (everything else). Files are inserted into the prompt in the order Qpeer, Q', Qrepo within a token budget, creating context lengths from 4k to 128k tokens. The three tasks, assertion cloze, targeted test implementation, and coverage-oriented test implementation, are executed in a Docker sandbox, and performance is scored by whether generated code runs and whether it meets the coverage or target-use requirement.

What would settle it

A direct test would be to run the benchmark with the oracle context replaced by randomly sampled files; if random file selection matches or exceeds the reported success rates, the claim that the coverage-ranked oracle context represents a near-upper bound would be falsified, along with the interpretation that models fail even with ideal context.

Watch

Extended reading notes

Core claim

The paper claims to establish that CLOVER provides a valid and scalable measure of test-case generation ability under long-context constraints, and that current LLMs fail at the coverage-oriented task even when given near-ideal context. The benchmark's oracle retrieval calibrates coverage information by comparing each test's coverage against an empty test to identify uninformative files (Qrepo) and files uniquely covered by the target test (Qpeer), then fills the context budget in that priority order. Across 10 open-source and 4 proprietary models, performance is comparable on short contexts, but a sharp degradation appears at 16k tokens for most open-source models, and on Task III all models score below 35% even with the oracle context. The authors interpret this as evidence that long-context instruction following and context utilization are significant bottlenecks for current models, and that the benchmark can support both model evaluation and the development of code agents.

Load-bearing premise

The oracle retrieval assumes that files whose coverage is unchanged between the real test and an empty test are uninformative, and that files covered by only one test are the most informative, but this ordering is not validated against random or length-based selection of files.

Editorial extensions

If this is right

  • If CLOVER's results hold, claims that an LLM handles 128k-token context should be tested on coverage-complete test generation, not just retrieval or summarization.
  • The coverage-based file ranking offers a cheap signal for building code agents that decide which source files to read when writing tests.
  • The sharp open-source performance drop at 16k tokens pinpoints a specific failure mode: models can technically accept long inputs but lose the ability to follow detailed instructions or generate non-gibberish code.
  • The three-task design gives a graded difficulty ladder, from assertion completion to multi-file coverage, which can isolate where a model's test-writing ability breaks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The oracle retrieval's file ranking could be compared against simpler retrievers such as lexical or embedding-based similarity to test whether coverage-derived relevance is truly near-optimal; the paper does not report such an ablation.
  • The context-utilization metric, defined as the best and worst performance change from the context-free baseline, could generalize beyond testing as a way to measure how much any long-context benchmark rewards adding context.
  • The sharp degradation at 16k for open-source models may reflect a length-generalization or attention-scattering effect rather than a coding-specific deficit, and could be investigated by controlled experiments that add irrelevant files while keeping oracle files fixed.
  • Task III's coverage-as-success criterion could be extended to mutation testing or branch coverage, making the benchmark sensitive to the quality of assertions, not just whether target lines are executed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces CLOVER, a benchmark for evaluating LLMs on Python unit-test generation and completion. It is built from 12 GitHub repositories, yielding 845 problems that are expanded into 5312 instances across three tasks: assertion cloze filling (Task I), targeted test implementation (Task II), and coverage-oriented test implementation (Task III). The benchmark supplies context windows from 4k to 128k tokens, constructed by a coverage-based ranking that classifies source files as Qpeer, Q', and Qrepo, with Qpeer prioritized. The authors evaluate 10 open-source and 4 proprietary models using execution-based metrics (pytest with coverage), and report that all models score below 35% on Task III even with the 'oracle context' (GPT-4o at 32.7% at 64k), while many open-source models degrade sharply at 16k context. They also propose a context utilization metric delta. The abstract and Section 1 state that code, data, and a Dockerized environment will be released.

Significance. If the benchmark construction is valid and the artifacts are released, CLOVER would fill a gap in the test-generation benchmark landscape: existing benchmarks such as TestBench, SWT-Bench, and TestGenEval mostly cap context at 32-50k tokens and do not combine long-context evaluation with coverage-verified test generation. The execution-based evaluation, using pytest-cov to verify generated tests, is a genuine strength, as is the coverage-driven retrieval idea. The headline claim that frontier models remain below 35% on multi-file coverage-complete test generation is falsifiable and would motivate research on long-context code generation and retrieval. However, the significance is contingent on two things: demonstrating that the coverage-ranked context is truly a near-upper-bound retrieval (not an arbitrary heuristic), and making the instances and evaluation pipeline inspectable by the community. The paper's limitation section acknowledges the Python/pytest scope and the lack of agentic evaluation, but it does not mention these two gaps.

major comments (3)
  1. [Section 3.1, Abstract] The central claim that 'all models score below 35% on the complex Task III, even with the oracle context provided' rests on the assertion that the coverage-ranked context is a near-upper-bound, sufficient retrieval. The paper provides no ablation of the Qpeer > Q' > Qrepo ordering against random file selection, length-based selection, or an embedding-based retriever, and it does not check whether the gold test's required symbols are actually present in the supplied context. Because Section 3.1 selects files randomly within each category, the context is nondeterministic, and the Qrepo heuristic can actively drop necessary files: an import-time constant covered by the empty test is assigned to Qrepo, and a class exercised by several peer tests can fall outside Qpeer. The sub-35% Task III ceiling may therefore reflect retrieval insufficiency rather than task difficulty. Please add (i) a retrieval ablation (random, length-based, embedding-based, and gold-file-only contexts), (ii) a per-instance sufficiency check that the gold test can be reconstructed from the supplied context, and (iii) at least a small human baseline or a known-answer baseline using the same context.
  2. [Abstract and Section 1] The manuscript describes a benchmark but the data, code, and Docker image are only promised ('we will release', Abstract and Section 1; 'We plan to release' in Section 1). Without the 5312 instances, the random within-category selection described in Section 3.1 cannot be reproduced, and the numbers in Tables 3-8 cannot be independently checked. For a benchmark submission, the central artifact should be available at review time, e.g., an anonymous repository containing instance IDs, the exact prompts, model generations, and the evaluation scripts. If the artifact cannot be released at this stage, the paper should clearly state which parts of the benchmark are already available and which are planned, and the evaluation results should be treated as preliminary.
  3. [Section 2.3] All numbers in Tables 3-8 come from a single sampling run at temperature 0.2 and top-p 1.0. Many headline comparisons are small: for example, Table 4 reports delta_max values of 0.6-3.5%, and Table 6 shows differences of 0.9-1.8% between the 64k and 128k settings. Success rates near 0% in Task II/III are sensitive to one or two instances. The paper should report multiple seeds or bootstrap confidence intervals, at least for the headline Task III results and for the context-utilization metric, so the reader can judge whether the 'all models below 35%' claim and the open-source versus proprietary gaps are statistically robust. Without this, the precise ordering of models and the significance of small context-length effects are not reliable.
minor comments (6)
  1. [Section 4.2] The text states that Claude 3.5-S achieves 'a 3.0% gain from 72.6% to 75.6%', but Table 4 reports 72.4% and 75.4%; please align the numbers.
  2. [Section 5] The statement 'In the context free setting of this task, it provides no context to the model, not even the file name required to complete the task' appears to contradict Figure 2, where the Task II checklist includes the target file name in the instruction. Please clarify what information is withheld in the Problem-Only setting.
  3. [Section 3.1] The set-builder notation for Qrepo and Qpeer is not well-formed: 'argf [1(...)]' is unclear, and the Peer Baseline condition sums over t' without explicitly quantifying over l. Please rewrite with explicit predicates, e.g., Qpeer_t = {x_f : exists l, C_base_{t,f,l}=1 and sum_{t'} C_base_{t',f,l}=1}.
  4. [Section 3.1] The sentence 'This implies that the files in Qrepo_t offer minimal information gain in terms of entropy' uses 'entropy' informally. Either provide a formal definition or soften the claim (e.g., 'are assumed to offer minimal information gain').
  5. [Section 2.3] The paper states that 'The tokenizer from Mistral-7B was used in this process' for context-length determination. Please state explicitly that all context lengths in the benchmark are measured with the Mistral tokenizer and discuss the implications for models with different tokenizers, which may have different actual token counts for the same byte length.
  6. [Table 2] The layout of Table 2 is hard to read: the columns labeled 'PO 4k 8k 16k 32k 64k 128k' mix a setting name with context lengths, and dashes are ambiguous. Please restructure the table to clearly separate settings and token budgets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; CLOVER is an empirical benchmark whose central model scores are externally measured by executing generated tests.

full rationale

The paper does not derive a predicted quantity from a fitted parameter, nor does it invoke a load-bearing self-citation or an imported uniqueness theorem. The central claims are empirical measurements: model-generated test cases are executed, and success is defined by execution plus coverage or target-use checks. The oracle retrieval in Section 3.1 is constructed from coverage information, and Task III's coverage requirements are also selected using the same Qpeer machinery; this creates a self-referential instrumentation design, but it is not a circular derivation because the reported model scores are not implied by the construction. A sub-35% result could have been different under a different retrieval scheme, and the absence of an ablation against random or embedding-based file selection is a validation concern, not a reduction of the conclusion to its inputs. No equation in the paper defines the benchmark outcome in terms of the oracle context itself, and no fitted value is renamed as a prediction. The paper is self-contained against external model evaluations, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on domain assumptions about coverage as a proxy, sandbox fidelity, and the unablated oracle retrieval ordering. No numerical parameters are fitted to data; the hand-chosen thresholds shape example selection but not the central conclusion that models perform poorly on Task III.

free parameters (4)
  • assertion frequency filter threshold = 0.01
    Task I selection drops reference answers appearing in more than 1% of a repository's problems (Section 4, p(x_i)). This hand-chosen threshold shapes which assertions appear and affects Task I difficulty.
  • per-repository example caps = 50 (Task I), 25 (Tasks II and III)
    Section 2.2 caps examples per repository to limit evaluation to 2 hours per model. This shapes the benchmark composition and the number of instances (5312 total).
  • coverage block constraints = up to 10 files; at least 5 lines per block
    Section 6 filters code blocks used in Task III coverage requirements. These thresholds determine which coverage targets are feasible and affect Task III difficulty.
  • context budget settings = 4k, 8k, 16k, 32k, 64k, 128k
    Evaluation settings for context windows; they are hand-chosen rather than fitted, but they define the long-context claims in Tables 4-6.
assumptions (4)
  • domain assumption Line coverage from pytest-cov is a valid and sufficient proxy for test case quality and task success.
    All three tasks define correctness via execution and line coverage (Sections 4-6); if coverage can be achieved with trivial or lower-quality assertions, the benchmark overstates model capability.
  • domain assumption The auto-configured Docker sandbox faithfully reproduces each repository's intended runtime behavior.
    Section 2.2 configures environments automatically; misconfiguration would make generated tests fail for environmental reasons, understating model performance.
  • domain assumption The coverage-calibrated oracle retrieval (Qrepo, Qpeer, Q') correctly ranks source files by informativeness.
    Section 3.1 defines the ordering used to build oracle contexts; no ablation validates this ordering, so the "near-upper-bound" interpretation depends on it.
  • domain assumption Extracted test cases are self-contained and the target code blocks coverable by a generated test.
    Section 2.1 heuristically isolates setup code; if extraction drops needed imports or fixtures, tasks may be unsolvable or too easy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification." pith.science (2026). https://pith.science/paper/XLT4XFP2

@misc{pith2026250208806,
  author       = {Pith},
  title        = {Pith review of: CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLT4XFP2}},
  note         = {Machine review of arXiv:2502.08806}
}
read the original abstract

Software testing is a critical aspect of software development, yet generating test cases remains a routine task for engineers. This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases under specific conditions. Spanning from simple assertion completions to writing test cases that cover specific code blocks across multiple files, these tasks are based on 12 python repositories, analyzing 845 problems with context lengths ranging from 4k to 128k tokens. Utilizing code testing frameworks, we propose a method to construct retrieval contexts using coverage information. While models exhibit comparable performance with short contexts, notable differences emerge with 16k contexts. Notably, models like GPT-4o and Claude 3.5 can effectively leverage relevant snippets; however, all models score below 35\% on the complex Task III, even with the oracle context provided, underscoring the benchmark's significance and the potential for model improvement. The benchmark is containerized for code execution across tasks, and we will release the code, data, and construction methodologies.

Figures

Figures reproduced from arXiv: 2502.08806 by the authors.

Figure 1
Figure 1. Pipeline overview. In this example, we focus on a test function test iter, which covers the use of Token and TokenStream classes from the source code. There are four major steps: (1) we extract the problem from a test file test lexnparse.py.; (2) verify of the extracted case(s) by running pytest; (3) assemble task prompts with pre-constructed or￾acle dependent files; (4) obtain model response and verify the executio… view at source ↗
Figure 2
Figure 2. Task specific prompt template for Task I, II and III. For the complete prompts, check Sec C. Objective The objective of constructing the oracle re￾trieval is to provide the most relevant or informative content within a constrained context budget. For the rest of this sec￾tion, we will describe how we prioritize salient information with coverage information. 3.1. Calibration of Coverage Within a file, we represent th… view at source ↗
Figure 3
Figure 3. PO prompt of Task I. Success Rate Execution Rate Model 64k 128k 64k 128k YI-CODER-9B 10.3% 10.3% 22.4% 17.8% LLAMA 3.1-8B 0.9% 0.9% 4.7% 1.9% LLAMA 3.1-70B 11.2% 4.7% 14.0% 7.5% GPT-4O-MINI 21.5% 20.6% 29.9% 28.0% GEMINI 1.5-F 26.2% 25.2% 36.4% 34.6% CLAUDE 3.5-S 32.7% 29.9% 42.1% 37.4% GPT-4O 28.0% 29.9% 35.5% 41.1% [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Contextual prompt of Task I. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Contextual promptfull of Task II. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Contextual promptfull for Task III. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Reference graph

Works this paper leans on

45 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Meet yi-coder: A small but mighty llm for code, September 2024

    01.AI. Meet yi-coder: A small but mighty llm for code, September 2024. URL https://01-ai.github.io/blog.html?post=en/2024-09-05-A-Small-but-Mighty-LLM-for-Code.md

    work page 2024

  3. [3]

    Automated unit test improvement using large language models at meta

    Alshahwan, N., Chheda, J., Finogenova, A., Gokkaya, B., Harman, M., Harper, I., Marginean, A., Sengupta, S., and Wang, E. Automated unit test improvement using large language models at meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pp.\ 185--196, 2024

    work page 2024

  4. [5]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021 b

    arXiv 2021

  5. [6]

    Super: Evaluating agents on setting up and executing tasks from research repositories, 2024

    Bogin, B., Yang, K., Gupta, S., Richardson, K., Bransom, E., Clark, P., Sabharwal, A., and Khot, T. Super: Evaluating agents on setting up and executing tasks from research repositories, 2024. URL https://arxiv.org/abs/2409.07440

    arXiv 2024

  6. [7]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  7. [8]

    Chatunitest: A framework for llm-based test generation

    Chen, Y., Hu, Z., Zhi, C., Han, J., Deng, S., and Yin, J. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pp.\ 572--576, 2024

    work page 2024

  8. [9]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 45 references
  1. [10]

    J., Solar-Lezama, A., Synnaeve, G., and Wang, S

    Gu, A., Roziere, B., Leather, H. J., Solar-Lezama, A., Synnaeve, G., and Wang, S. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning, 2024

    2024

  2. [11]

    Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    2024 arXiv

  3. [12]

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  4. [13]

    Testgeneval: A real world unit test generation and test completion benchmark, 2024 a

    Jain, K., Synnaeve, G., and Rozière, B. Testgeneval: A real world unit test generation and test completion benchmark, 2024 a . URL https://arxiv.org/abs/2410.00752

    2024 arXiv

  5. [14]

    R2e: Turning any github repository into a programming agent environment

    Jain, N., Shetty, M., Zhang, T., Han, K., Sen, K., and Stoica, I. R2e: Turning any github repository into a programming agent environment. In Forty-first International Conference on Machine Learning, 2024 b

    2024

  6. [15]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  7. [16]

    E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

    Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66

    2024

  8. [17]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  9. [18]

    Ds-1000: A natural and reliable benchmark for data science code generation

    Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-T., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501, 2022

    2022 arXiv

  10. [19]

    Devbench: A comprehensive benchmark for software development, 2024

    Li, B., Wu, W., Tang, Z., Shi, L., Yang, J., Li, J., Yao, S., Qian, C., Hui, B., Zhang, Q., Yu, Z., Du, H., Yang, P., Lin, D., Peng, C., and Chen, K. Devbench: A comprehensive benchmark for software development, 2024. URL https://arxiv.org/abs/2403.08604

    2024 arXiv

  11. [20]

    M., Han, Y., Ma, Y., Li, G., and Huang, G

    Liu, K., Liu, Y., Chen, Z., Zhang, J. M., Han, Y., Ma, Y., Li, G., and Huang, G. Llm-powered test case generation for detecting tricky bugs. arXiv preprint arXiv:2404.10304, 2024 a

    2024 arXiv

  12. [21]

    Repobench: Benchmarking repository-level code auto-completion systems

    Liu, T., Xu, C., and McAuley, J. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Representations, 2024 b

    2024

  13. [22]

    Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., Liu, T., Tian, M., Kocetkov, D., Zucker, A., Belkada, Y., Wang, Z., Liu, Q., Abulkhanov, D., Paul, I., Li, Z., Li, W.-D., Risdal, M., Li, J., Zhu, J., Zhuo, T. ...

    2024 arXiv

  14. [23]

    Repoagent: An llm-powered open-source framework for repository-level code documentation generation, 2024

    Luo, Q., Ye, Y., Liang, S., Zhang, Z., Qin, Y., Lu, Y., Wu, Y., Cong, X., Lin, Y., Zhang, Y., Che, X., Liu, Z., and Sun, M. Repoagent: An llm-powered open-source framework for repository-level code documentation generation, 2024

    2024

  15. [24]

    Mathews, N. S. and Nagappan, M. Test-driven development for code generation, 2024. URL https://arxiv.org/abs/2402.13521

    2024 arXiv

  16. [25]

    N., He, J., and Vechev, M

    Mündler, N., Müller, M. N., He, J., and Vechev, M. Swt-bench: Testing and validating real-world bug-fixes with code agents, 2024. URL https://arxiv.org/abs/2406.12952

    2024 arXiv

  17. [26]

    J., Mooney, R

    Nie, P., Banerjee, R., Li, J. J., Mooney, R. J., and Gligoric, M. Learning deep semantics for test completion. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp.\ 2111--2123. IEEE, 2023

    2023

  18. [27]

    Codegen: An open large language model for code with multi-turn program synthesis

    Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023

    2023

  19. [28]

    E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  20. [29]

    K., and Ray, B

    Ryan, G., Jain, S., Shang, M., Wang, S., Ma, X., Ramanathan, M. K., and Ray, B. Code-aware prompting: A study of coverage-guided test generation in regression setting using llm. Proceedings of the ACM on Software Engineering, 1 0 (FSE): 0 951--971, 2024

    2024

  21. [30]

    Chatgpt vs sbst: A comparative assessment of unit test suite generation

    Tang, Y., Liu, Z., Zhou, Z., and Luo, X. Chatgpt vs sbst: A comparative assessment of unit test suite generation. IEEE Transactions on Software Engineering, 2024

    2024

  22. [31]

    A., Shen, J., Kelley, J., et al

    Team, C., Zhao, H., Hui, J., Howland, J., Nguyen, N., Zuo, S., Hu, A., Choquette-Choo, C. A., Shen, J., Kelley, J., et al. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409, 2024

    2024 arXiv

  23. [32]

    K., and Sundaresan, N

    Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., and Sundaresan, N. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617, 2020

    2009 arXiv

  24. [33]

    Software testing with large language models: Survey, landscape, and vision

    Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., and Wang, Q. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 2024 a

    2024

  25. [34]

    R., and Ma, L

    Wang, W., Yang, C., Wang, Z., Huang, Y., Chu, Z., Song, D., Zhang, L., Chen, A. R., and Ma, L. Testeval: Benchmarking large language models for test case generation, 2024 b . URL https://arxiv.org/abs/2406.04531

    2024 arXiv

  26. [35]

    F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H

    Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H. H., Li, F., Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y., Hui, B., Lin, J., Brennan, R., Peng, H., Ji, H., and Neubig, G. OpenHands: An Open Platform...

    2024 arXiv

  27. [36]

    Magicoder: Empowering code generation with OSS -instruct

    Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magicoder: Empowering code generation with OSS -instruct. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.\ 52632--52657. PMLR, 21--27 Jul 20...

    2024

  28. [37]

    Qwen2.5 technical report

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, ...

    2024 arXiv

  29. [38]

    and Liang, P

    Yasunaga, M. and Liang, P. Break-it-fix-it: Unsupervised learning for program repair. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 11941--11952. PMLR, 18--2...

    2021

  30. [39]

    and Harman, M

    Yoo, S. and Harman, M. Regression testing minimization, selection and prioritization: a survey. Software testing, verification and reliability, 22 0 (2): 0 67--120, 2012

    2012

  31. [40]

    Evaluating and improving chatgpt for unit test generation

    Yuan, Z., Liu, M., Ding, S., Wang, K., Chen, Y., Peng, X., and Lou, Y. Evaluating and improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering, 1 0 (FSE): 0 1703--1726, 2024

    2024

  32. [41]

    Testbench: Evaluating class-level test case generation capability of large language models, 2024 a

    Zhang, Q., Shang, Y., Fang, C., Gu, S., Zhou, J., and Chen, Z. Testbench: Evaluating class-level test case generation capability of large language models, 2024 a . URL https://arxiv.org/abs/2409.17561

    2024 arXiv

  33. [42]

    B ench: Extending long context evaluation beyond 100 K tokens

    Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., and Sun, M. B ench: Extending long context evaluation beyond 100 K tokens. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Associatio...

    2024

  34. [43]

    Y., Vu, M

    Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. CoRR, 2024

    2024

  35. [44]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  36. [45]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  37. [46]

    a") == "a

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.