Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

Enhancing LLM's Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RATester reduces LLM hallucinations in Go unit tests by fetching definitions from gopls as identifiers appear, lifting average compile rate to 61.84% and line coverage to 26.25%.

desk verdict Solid engineering contribution with a genuine new mechanism, but the causal claim about gopls is under-supported and the empirical evidence lacks variance and artifacts. read the letter →

arxiv 2501.07425 v1 pith:V3UK5AH4 submitted 2025-01-13 cs.SE

Xin Yin , Chao Ni , Xinrui Li , Liushan Chen , Guojun Ma , Xiaohu Yang This is my paper · ORCID
classification cs.SE
keywords unittestgenerationlargelanguagemodelsgoplsrepository-awarecontextGoprogramminghallucinationreductionmutationtestinglinecoverage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RATester is a framework for generating Go unit tests with LLMs that stops the model from guessing at the meanings of identifiers. Instead of relying on a fixed block of context, the generation loop pauses whenever the LLM writes an identifier the prompt does not already contain, asks the Go language server gopls for that identifier's definition and documentation comment, and inserts the answer before generating the next tokens. The paper argues that this gives the model the same just-in-time repository awareness a human tester gets from an IDE, and that it reduces hallucinations such as calls to nonexistent methods or wrong argument types. On a new dataset of eight large Go projects, RATester reaches an average compile rate of 61.84%, against 50.31% for the best baseline, and average line coverage of 26.25%, against 22.57%; it also kills more mutants in mutation testing. The same loop also improves three different open LLMs, which the authors take as evidence that the benefit comes from the retrieval mechanism rather than from any single backbone model.

What carries the argument

The load-bearing mechanism is the identifier-triggered lookup loop of Algorithm 1. The Fetcher first resolves the receiver, parameter types, and return type of the method under test through gopls, the Go language server; then, while the Generator emits tokens, the Formulator accumulates characters that can be part of a Go identifier, and the moment an identifier is completed and is not already in the code context, the loop pauses, calls gopls for the definition and documentation comment, appends it to the prompt, and resumes generation. This makes the context grow along the path the model is actually taking, rather than fixing the context window before generation starts.

What would settle it

Take the same eight Go projects and generate one test per focal method with RATester, but replace the gopls lookup with a random or empty context block whenever an unfamiliar identifier appears; if compile rate and line coverage stay at the same level, the claimed mechanism is not the cause of the improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM hallucinations in unit test generation are largely a context problem, and can be reduced by making context dynamic and precise rather than by collecting more training data or using a fixed extraction pattern. The discovery is that a generation loop which uses a language server to resolve each unfamiliar identifier as it appears, rather than pre-selecting all context, produces tests that compile and cover more code. Concretely, on the eight-project Go dataset the compiled fraction of generated tests rises from 16.67%–63.56% across baselines to 45.58%–69.49% for RATester, and line coverage rises from 7.49%–53.92% to 12.92%–58.09%; the generated tests also kill 25 to 147 more mutants than the compared approaches. The authors present this as evidence that giving the model the same lookups an IDE would give a human tester suppresses the errors that come from inventing APIs.

Load-bearing premise

The method assumes that gopls can resolve the identifier the LLM has just produced at the moment of lookup, even though the test file is being written token by token, does not yet exist as a saved parseable file, and may name something the repository does not define; if gopls returns nothing or resolves a different meaning, the prompt receives no grounding and the observed gains would weaken.

Editorial extensions

If this is right

  • If the measured gains hold, LLM-based Go test generation becomes usable without a dedicated repair phase: most generated tests compile on the first pass, so effort shifts from fixing syntax to designing assertions.
  • The mechanism transfers across backbone models, so the benefit lies in the retrieval loop rather than in any single pretrained model, making it a model-agnostic improvement strategy.
  • Because one RATester candidate per focal method beats ten candidates from the base LLM on line coverage, generating one high-quality test per function becomes the cost-effective operating point.
  • Generated tests complement existing human-written suites: adding them raises average repository line coverage from 56.49% to 60.98%, with the largest gains appearing in projects that previously had very weak coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because gopls is Go-specific, a natural extension is to test the same pause-and-resolve loop in languages with capable language servers such as Java, Python, and TypeScript; resolver quality and lookup latency may then dominate the gains more than the choice of LLM.
  • The design points to a general recipe for reducing hallucination in code generation: instead of packing maximal context up front, let the generated tokens drive context acquisition, which could apply to API usage, configuration generation, and test-double construction.
  • A testable extension is to measure which failure class the lookups remove; if the compile-rate gain comes mostly from correct method signatures, then a cheaper post-generation static check of identifiers might capture a large share of the benefit without per-token lookups.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents RATester, a framework for generating Go unit tests using LLMs. RATester builds a prompt containing the focal method and a partially generated test snippet, and during token-by-token generation it uses the gopls language server to fetch definitions and documentation for identifiers not already present in the accumulated prompt context. These fetched snippets are appended to the context, mimicking a developer's use of IDE tooling. The framework is evaluated on eight popular Go projects with three backbone LLMs (CodeLlama, DeepSeek-Coder, Magicoder) against baselines including NxtUnit, UniTester, ChatUniTest, and the bare LLMs, using compile rate, line coverage, and mutation testing. The paper reports consistent improvements for RATester over all baselines on all metrics.

Significance. If the reported results are reliable, the paper contributes a practical and cheap mechanism for improving LLM-based unit test generation without task-specific training: dynamic retrieval of project context via a language server. The evaluation covers substantial real-world projects and includes mutation testing, which is a stronger quality signal than compile rate alone. However, the attribution of the gains to the gopls-based 'precise context injection' is not isolated, and the absence of any variance analysis or statistical tests leaves the strength of the claims unclear.

major comments (4)
  1. [Section 5.1, Table 2] All reported values in Tables 2–6 are single-run point estimates with no variance, confidence intervals, or statistical significance tests. For claims of 'significant' improvement (e.g., the RQ-1 answer on compile rate and line coverage), the paper should provide at least bootstrap confidence intervals over the eight projects or a paired test (e.g., Wilcoxon signed-rank) across project-level metrics, or a per-test statistical comparison given the large number of generated tests (11,195). Without this, the reader cannot assess whether the differences are stable or due to a single favorable run.
  2. [Algorithm 1, Section 3.2] The central attribution to 'precise contextual information injection' is not isolated: RATester is never run without the Fetcher, so the reported gains over basic LLMs mix at least three changes: the gopls-based lookup, the token-by-token Formulator/Generator loop with a continuously growing prompt, and the prompt template itself. An ablation with the Fetcher disabled (or with non-gopls context, e.g., randomly selected definitions) is necessary to support the claim that gopls specifically drives the improvements. RQ-2 shows improvements over basic LLMs, but those are improvements of the whole RATester pipeline, not of the injection mechanism.
  3. [Algorithm 1, lines 10–12; Section 2.1] For hallucinated identifiers such as calls to non-existent methods, gopls by construction has no definition to return, so the Fetcher injects nothing for exactly the failure mode emphasized in the abstract and introduction. The paper reports no statistics on gopls lookup success/failure, no breakdown of compile errors by whether the offending identifier was resolvable, and no example where an invented identifier was successfully grounded. The motivating example in Section 2.1 concerns an existing method ('String'), not a non-existent one. This is a correctness-risk concern that requires additional data (e.g., lookup success rates and error analysis) or a more narrowly scoped claim.
  4. [Section 5.3, Section 4.4] The default configuration (one candidate test per focal method and a 512-token generation cap) is justified post hoc by the candidate-number experiment in Section 5.3. This post-hoc selection risks overfitting to the evaluation set. The paper should either pre-register the configuration or show that the main conclusions are robust to reasonable variations (e.g., candidate counts 1, 2, and 5) across compile rate, line coverage, and mutation testing, rather than line coverage alone.
minor comments (6)
  1. [Section 5.1, Table 3] In the paragraph following Table 3, the sentence 'the average compile rate also shows a notable improvement, rising from 56.49% to 60.98%' mentions compile rate, but Table 3 reports line coverage; the metric should be corrected to line coverage.
  2. [Section 4.2] The paper says 'we consider six baselines' but then splits the comparison across RQ-1 (NxtUnit, UniTester, ChatUniTest) and RQ-2 (CodeLlama, DeepSeek-Coder, Magicoder); a brief paragraph clarifying how these two sets relate would help the reader.
  3. [Figures 4 and 6] Figures 4 and 6 contain what appears to be corrupted or placeholder text (long runs of '/uni0000...'), which obscures the presented data and should be replaced with the intended rendered figures.
  4. [Section 2.2] There is a typo in the phrase 'an global-aware tester'; it should be 'a global-aware tester'.
  5. [Table 1] Table 1 shows that frp and nps have near-zero original line coverage; a one-sentence note explaining why (e.g., few existing unit tests) would help interpret the large relative jumps in Table 3.
  6. [Section 5.1, first paragraph of Results] The phrase 'improving from 16.67%–63.56% to 45.58%–69.49%' combines numbers from two different baselines into a single 'from' range; reporting each pair (UniTester vs. RATester and ChatUniTest vs. RATester) separately would avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RATester's reported gains are measured against external baselines, and no fitted parameter or self-cited premise is renamed as a result.

full rationale

RATester's central claims are empirical: compile rate, line coverage, and mutation scores are evaluated against external baselines (NxtUnit, UniTester, ChatUniTest, and basic LLMs) on eight real-world projects. No parameter is fitted to the evaluation data and then reported as a predicted quantity; the 512-token cap and the choice of one candidate per focal method are stated as cost/performance tradeoffs, not as results derived from the approach's own definitions. The gopls-based Fetcher is a mechanism whose causal contribution is not isolated by an ablation, and gopls cannot resolve hallucinated non-existent identifiers, but this is an attribution/validity weakness rather than a circular step: the reported improvements are not equivalent to the mechanism's inputs by construction. Self-citations [24, 49-52] appear only as related work and do not supply a load-bearing uniqueness theorem or a premise on which the evaluation depends. Accordingly, no circular step can be exhibited from the paper's own equations or citation chain.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

RATester introduces no fitted mathematical parameters and no invented entities. Its burden sits in engineering assumptions: gopls must correctly resolve identifiers, line coverage and mutation kill counts must proxy test quality, the eight chosen projects must be representative, and identifier tokenization must be clean enough for the online lookup trigger. Two hyperparameters, the 512-token generation cap and the single test candidate per method, are hand-chosen to balance cost and performance.

free parameters (2)
  • max_generation_tokens = 512
    Algorithm 1 caps generation at 512 tokens; a hand-chosen cost control, not derived from theory.
  • candidate tests per focal method = 1
    RQ-3 explores 1 to 10 candidates and then selects 1 as the default for all reported results based on a performance and cost tradeoff.
assumptions (4)
  • domain assumption gopls resolves any identifier the LLM generates in the project's package context.
    Section 3.2 and Algorithm 1 depend on gopls returning the real definition for unseen identifiers; unresolved identifiers would leave the model without the promised grounding.
  • domain assumption Line coverage and mutation score, measured by go test and Gremlins, are valid proxies for unit test quality.
    Section 4.3 defines the evaluation metrics; no human study or developer preference assessment is provided.
  • domain assumption The eight highly starred Go projects form a representative benchmark.
    Section 4.1 selects beego, echo, fiber, frp, gin, hugo, nps, and traefik; Section 6 concedes limited external validity to other languages and projects.
  • domain assumption Generated identifiers can be segmented by simple Golang identifier rules.
    Algorithm 1 relies on is_golang_identifier_part to decide when an identifier is complete; unusual tokenization could delay or mis-trigger fetches.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing LLM's Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection." pith.science (2026). https://pith.science/paper/V3UK5AH4

@misc{pith2026250107425,
  author       = {Pith},
  title        = {Pith review of: Enhancing LLM's Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3UK5AH4}},
  note         = {Machine review of arXiv:2501.07425}
}
read the original abstract

Though many learning-based approaches have been proposed for unit test generation and achieved remarkable performance, they still have limitations in relying on task-specific datasets. Recently, Large Language Models (LLMs) guided by prompt engineering have gained attention for their ability to handle a broad range of tasks, including unit test generation. Despite their success, LLMs may exhibit hallucinations when generating unit tests for focal methods or functions due to their lack of awareness regarding the project's global context. These hallucinations may manifest as calls to non-existent methods, as well as incorrect parameters or return values, such as mismatched parameter types or numbers. While many studies have explored the role of context, they often extract fixed patterns of context for different models and focal methods, which may not be suitable for all generation processes (e.g., excessive irrelevant context could lead to redundancy, preventing the model from focusing on essential information). To overcome this limitation, we propose RATester, which enhances the LLM's ability to generate more repository-aware unit tests through global contextual information injection. To equip LLMs with global knowledge similar to that of human testers, we integrate the language server gopls, which provides essential features (e.g., definition lookup) to assist the LLM. When RATester encounters an unfamiliar identifier (e.g., an unfamiliar struct name), it first leverages gopls to fetch relevant definitions and documentation comments, and then uses this global knowledge to guide the LLM. By utilizing gopls, RATester enriches the LLM's knowledge of the project's global context, thereby reducing hallucinations during unit test generation.

Figures

Figures reproduced from arXiv: 2501.07425 by the authors.

Figure 1
Figure 1. A focal method along with the unit tests generated by DeepSeek-Coder for a project named gin 2.2 Key Ideas Based on the above observation, we propose RATester for unit test generation that utilizes gopls to fetch definitions and documentation comments for unfamiliar structs, methods, functions, and more. This approach provides LLMs with the precise context needed for generation, effectively reducing hallucinations d… view at source ↗
Figure 2
Figure 2. Overview of RATester 3.2 Fetcher When generating unit tests for focal methods or functions, LLMs often produce hallucinations, which can manifest as calls to non-existent methods, as well as incorrect parameter assignments and return values, such as mismatched parameter types or an improper number of parameters. In contrast, human testers typically possess a strong understanding of the various methods, functions, an… view at source ↗
Figure 3
Figure 3. An example of prompt for the unit test generation 3.3 Formulator In both the initial stage and the continuous phase of unit test generation, the formulator fills the fetched code context and the test snippet into the prompt template, where fill means the procedure of inserting content into corresponding positions of the template. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: RQ-2: RATester vs. Basic LLMs across different projects on compile rate presents the unit test generated by CodeLlama, and the right shows the one generated by RATester. For the test case generated by CodeLlama, it tests the “Render” method of a struct named “String”. …
Figure 5
Figure 5. Figure 5: RQ-2: The unit tests generated by CodeLlama and RATester for the gin project , Vol. 1, No. 1, Article . Publication date: January 2025 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: RQ-3: (Left) Total generation time of RATester and baselines; (Right) Performance of RATester and DeepSeek-Coder across varying unit test candidate number Answer to RQ-3: (1) RATester requires more generation time than the basic LLMs; however, considering the performan…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Uncovering Business Logic Bugs via Semantics-Driven Unit Test Generation

    cs.SE 2026-04 conditional novelty 7.0 of 10

    SeGa extracts business semantics from requirements to generate unit tests that detect 22-25 more real-world business logic bugs than prior LLM-based methods in industrial Go projects.

  2. Knowledge-Guided Synthetic Bug Feedback for LLM-Based Unit Test Generation

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Mechanism-guided synthetic-bug feedback raises real-bug detection of LLM unit tests to 72.67% RBDR on 172 Defects4J tasks, outperforming mutation and coverage baselines by ~20 points.

  3. Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation

    cs.SE 2026-07 conditional novelty 6.0 of 10

    CATGen improves LLM unit-test reliability by combining structured project-context retrieval, deterministic test-class skeletons, and static-analysis repair, beating six baselines on compilation success, coverage, and cost.

  4. YATE: The Role of Test Repair in LLM-Based Unit Test Generation

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A test-repair pipeline, combining static analysis and re-prompting, raises LLM-generated unit test coverage and mutation killing by roughly 20-30 percent over a plain prompt baseline on six Java projects.

  5. Large Language Models for Unit Testing: A Systematic Literature Review

    cs.SE 2025-06 conditional novelty 5.0 of 10

    The paper presents the first systematic literature review of large language model based unit testing, covering 105 papers up to March 2025.

  6. A Survey on LLM-based News Recommender Systems

    cs.IR 2025-02 conditional novelty 5.0 of 10

    A survey that categorizes LLM-based news recommender systems and reports benchmark comparisons on MIND and Adressa.

Reference graph

Works this paper leans on

57 extracted references · 27 canonical work pages · cited by 6 Pith papers

  1. [1]

    2024. gopls. https://github.com/golang/tools/tree/master/gopls

    work page 2024

  2. [2]

    gremlins

    2024. gremlins. https://github.com/go-gremlins/gremlins

    work page 2024

  3. [3]

    Hugging Face

    2024. Hugging Face. https://huggingface.co

    work page 2024

  4. [4]

    Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.06333 (2021)

    arXiv 2021

  5. [5]

    DeepSeek AI. 2023. DeepSeek Coder: Let the Code Write Itself. https://github.com/deepseek-ai/DeepSeek-Coder

    work page 2023

  6. [6]

    Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2023. A3Test: Assertion-Augmented Automated Test Case Generation. arXiv preprint arXiv:2302.10352 (2023)

    arXiv 2023

  7. [7]

    Aldeida Aleti, Irene Moser, and Lars Grunske. 2017. Analysing the fitness landscape of search-based software testing problems. Automated Software Engineering 24 (2017), 603–621

    work page 2017

  8. [8]

    Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering 41, 5 (2014), 507–525

    2014

Show all 57 references
  1. [9]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 572–576

    2024

  2. [10]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017)

    2017

  3. [11]

    Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022. Toga: A neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering . 2130–2141

    2022

  4. [12]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al . 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020), 1536–1547

    2020 arXiv

  5. [13]

    Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419

    2011

  6. [14]

    Michael Fu and Chakkrit Tantithamthavorn. 2022. Linevul: A transformer-based line-level vulnerability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories . 608–620

    2022

  7. [15]

    Shuzheng Gao, Chaozheng Wang, Cuiyun Gao, Xiaoqian Jiao, Chun Yong Chong, Shan Gao, and Michael Lyu. 2025. The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation. arXiv preprint arXiv:2501.01329 (2025)

    2025 arXiv

  8. [16]

    Vahid Garousi and Junji Zhi. 2013. A survey of software testing practices in Canada. Journal of Systems and Software 86, 5 (2013), 1354–1376

    2013

  9. [17]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow.arXiv preprint arXiv:2009.08366 (2020)

    2020 arXiv

  10. [18]

    Yifeng He, Jiabo Huang, Yuyang Rong, Yiwen Guo, Ethan Wang, and Hao Chen. 2024. UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testing. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and A...

    2024

  11. [19]

    David Hin, Andrey Kan, Huaming Chen, and M Ali Babar. 2022. LineVD: Statement-level Vulnerability Detection using Graph Neural Networks. arXiv preprint arXiv:2203.05181 (2022)

    2022 arXiv

  12. [20]

    Jihyun Lee, Sungwon Kang, and Danhyung Lee. 2012. Survey on software testing practices. IET software 6, 3 (2012), 275–282

    2012

  13. [21]

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on , Vol. 1, No. 1, Article . Publication date: Ja...

    2023

  14. [22]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you!arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  15. [23]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1–35

    2023

  16. [24]

    Chao Ni, Xiaoya Wang, Liushan Chen, Dehai Zhao, Zhengong Cai, Shaohua Wang, and Xiaohu Yang. 2024. CasModaTest: A Cascaded and Model-agnostic Self-directed Framework for Unit Test Generation. arXiv preprint arXiv:2406.15743 (2024)

    2024 arXiv

  17. [25]

    Chao Ni, Xin Yin, Kaiwen Yang, Dehai Zhao, Zhenchang Xing, and Xin Xia. 2023. Distinguishing Look-Alike Innocent and Vulnerable Code by Subtle Semantic Representation Learning and Explanation. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Sy...

    2023

  18. [26]

    Carlos Oliveira, Aldeida Aleti, Lars Grunske, and Kate Smith-Miles. 2018. Mapping the effectiveness of automated test suite generation techniques. IEEE Transactions on Reliability 67, 3 (2018), 771–785

    2018

  19. [27]

    OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. (2022). https://openai.com/blog/chatgpt/

    2022

  20. [28]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35...

    2022

  21. [29]

    Carlos Pacheco and Michael D Ernst. 2007. Randoop: feedback-directed random testing for Java. In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion . 815–816

    2007

  22. [30]

    Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2015. Reformulating branch coverage as a many- objective optimization problem. In 2015 IEEE 8th international conference on software testing, verification and validation (ICST). IEEE, 1–10

    2015

  23. [31]

    Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2017. Automated test case generation as a many- objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering 44, 2 (2017), 122–158

    2017

  24. [32]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  25. [33]

    http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high- performance-deep-learning-library.pdf

    Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high- performance-deep-learning-library.pdf

  26. [34]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. (2018)

    2018

  27. [35]

    Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J Hellendoorn. 2023. CAT-LM training language models on aligned code and tests. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 409–420

    2023

  28. [36]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al . 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  29. [37]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray

  30. [38]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering (2023)

    2023

  31. [39]

    Laurence Saes. 2018. Unit test generation using machine learning. Universiteit van Amsterdamg (2018)

    2018

  32. [40]

    Jiho Shin, Sepehr Hashtroudi, Hadi Hemmati, and Song Wang. 2024. Domain Adaptation for Code Model-Based Unit Test Case Generation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1211–1222

    2024

  33. [41]

    Jessica Shieh. 2023. Best practices for prompt engineering with OpenAI API. OpenAI, February https://help.openai. com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api (2023)

    2023

  34. [42]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617 (2020). , Vol. 1, No. 1, Article . Publication date: January 2025. 22 Xin Yin, Chao Ni, Xin...

    2020 arXiv

  35. [43]

    Michele Tufano, Shao Kun Deng, Neel Sundaresan, and Alexey Svyatkovskiy. 2022. Methods2Test: A dataset of focal methods mapped to test cases. In Proceedings of the 19th International Conference on Mining Software Repositories . 299–303

    2022

  36. [44]

    Siwei Wang, Xue Mao, Ziguang Cao, Yujun Gao, Qucheng Shen, and Chao Peng. 2023. NxtUnit: Automated Unit Test Generation for Go. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. 176–179

    2023

  37. [45]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  38. [46]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120 (2023)

    2023 arXiv

  39. [47]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder- decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)

    2021 arXiv

  40. [48]

    Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023)

    2023 arXiv

  41. [49]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery

    2023

  42. [50]

    Xin Yin, Chao Ni, and Shaohua Wang. 2024. Multitask-based evaluation of open-source llm on software vulnerability. IEEE Transactions on Software Engineering (2024)

    2024

  43. [51]

    Xin Yin, Chao Ni, Tien N Nguyen, Shaohua Wang, and Xiaohu Yang. 2024. Rectifier: Code Translation with Corrector via LLMs. arXiv preprint arXiv:2407.07472 (2024)

    2024 arXiv

  44. [52]

    Xin Yin, Chao Ni, Xiaodan Xu, and Xiaohu Yang. 2024. What You See Is What You Get: Attention-based Self-guided Automatic Unit Test Generation. arXiv preprint arXiv:2412.00828 (2024)

    2024 arXiv

  45. [53]

    Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. 2024. Thinkrepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1274–1286

    2024

  46. [54]

    Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023. No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.04207 (2023)

    2023 arXiv

  47. [55]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726

    2024

  48. [57]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019). , Vol. 1, No. 1, Article . Publication date: January 2025

    2019 arXiv

  49. [2024]

    Proceedings of the ACM on Software Engineering 1, FSE (2024), 951–971

    Code-aware prompting: A study of coverage-guided test generation in regression setting using llm. Proceedings of the ACM on Software Engineering 1, FSE (2024), 951–971

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.