Agent-Based Test Assertion Generation via Diverse Perspective Aggregation

Dong Wang; Guangtai Liang; Jianyi Zhou; Junjie Chen; Lin Yang; Qiaoyu Han

REVIEW 3 major objections 5 minor 51 references

Agent-Based Test Assertion Generation via Diverse Perspective Aggregation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AssertMate claims that decomposing assertion generation into actual-value construction and multi-perspective expected-value prediction lets a small open-source LLM beat ChatGPT-based baselines on Defects4J.

desk verdict Solid engineering result with a clean decomposition, but the undisclosed RAG retrieval corpus is a real leakage risk that needs fixing before the main claims are trustworthy. read the letter →

arxiv 2608.05822 v1 pith:25IQSDMB submitted 2026-08-06 cs.SE

Dong Wang , Qiaoyu Han , Lin Yang , Jianyi Zhou , Guangtai Liang , Junjie Chen This is my paper · ORCID

classification cs.SE

keywords testassertiongenerationlargelanguagemodelsmulti-agentcollaborationstaticprogramanalysisretrieval-augmentedchain-of-thoughtLLM-as-a-JudgeDefects4J

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes AssertMate, an agent-based framework that generates unit-test assertions by splitting the task into two halves: first deciding what to test (actual values, via static analysis and type-aware rules) and then deciding what to expect (expected values, via three complementary LLM agents using code generation, retrieval augmentation, and chain-of-thought reasoning). A fourth LLM judge selects the best candidate assertion. On the Defects4J benchmark, AssertMate reports a compilation success rate of 76.78%, a pass rate of 60.09%, and a bug detection rate of 46.25%, outperforming the strongest baseline, ChatAssert*, by large relative margins while using a 7B open-source model. The authors argue that structured task decomposition reduces hallucination and oversampling, and that aggregating diverse reasoning perspectives is more reliable than one-shot prompting.

What carries the argument

The central mechanism is a fine-grained decomposition of an assertion into three components: assertion function, expected value, and actual value. The Actual Value Constructor (ActVCon) uses static program analysis, type-aware iterative traversal, and LLM assistance for external types to produce concrete assertion targets and candidate assertion functions. The Multi-Perspective Expected Value Generator (EVGen) then runs three agents (CGA, RAGA, and CoTA), oversamples five responses per agent, reranks by average token probability within the assertion portion, and passes the top candidate from each agent to a Judge agent that uses chain-of-thought reasoning to select the final assertion.

What would settle it

Inspect the retrieval corpus used by the RAG agent in the released replication package to check whether it contains any of the 112 Defects4J bugs or the 667 FM-TP pairs; then remove the RAGA agent and measure the resulting change in CSR, PR, and BDR. If the corpus overlaps the evaluation set, or if removing RAGA does not degrade performance as expected, the central claim about diverse perspectives is not supported.

Watch

Extended reading notes

Core claim

The central claim is that monolithic, one-shot assertion generation is the bottleneck, and that separating the 'what to test' decision from the 'what to expect' decision, then aggregating several independent reasoning strategies with an LLM-as-a-Judge, yields assertions that are far more likely to compile, pass, and detect real bugs. In the paper's experiments, the full AssertMate pipeline achieves mean CSR 76.78%, PR 60.09%, and BDR 46.25% on human-written Defects4J FM-TP pairs, outperforming ChatAssert* by 52.92% relative in CSR, 50.94% relative in PR, and by a statistically significant margin in BDR (p = 0.0489, delta = 0.152), while using a smaller open-source backbone model and comparable wall-clock time.

Load-bearing premise

The RAG agent retrieves the test case with the highest Jaccard similarity to the input test prefix from an unspecified corpus, so the premise that this retrieved example provides independent, non-leaky guidance is load-bearing for the claimed benefit of diverse perspective aggregation.

Editorial extensions

If this is right

Assertion generation can be substantially improved without execution-based repair loops, by combining program analysis with structured LLM reasoning.
The actual-value construction component is transferable: adding ActVCon to ChatAssert* improved its PR and BDR, suggesting other oracle generators can benefit from explicit target identification.
A small open-source model (DeepSeek-Coder-7B) can outperform a proprietary API-based model (ChatGPT) on this task when scaffolded with structured agents.
When integrated with EvoSuite, AssertMate-generated assertions cover more mutants and kill more mutants than the baselines, indicating the approach works as an oracle-enhancement layer for search-based test generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The three-perspective set (code generation, RAG, CoT) is presented as a representative, not exhaustive, configuration; the framework suggests that any diverse, non-redundant set of reasoning strategies could be plugged in, and the judge mechanism preserves rare correct candidates that voting or debate lose.
If the RAG agent's retrieval corpus overlaps the Defects4J evaluation set—the paper does not specify the corpus—the reported contribution of RAGA and hence of diverse perspective aggregation would be inflated; the released replication package should make the corpus identifiable.
The 46.25% BDR also implies that for over half of the evaluated bugs, no generated assertion catches the bug; combining the decomposition with automated discovery of public utility methods, as the getRMS case suggests, is a concrete next step.
The pass-rate and mutation-kill metrics are assessed at the assertion level under an equal-budget protocol; extending the evaluation to multi-assertion test cases, where 'where to assert' matters, remains an open problem that the paper explicitly flags.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: AssertMate does something real. It factors assertion generation into target selection, expected value prediction, and selection, and shows the decomposition pays off. The evaluation is extensive and the ablation logic is sound. But the paper has one load-bearing hole: the RAG agent's retrieval corpus is never described, and if that corpus contains Defects4J tests, the one-shot example leaks the expected assertion. That needs to be addressed before I'd accept the headline numbers.

What's genuinely good: the actual value constructor (ActVCon) is a sensible use of static analysis to bound the LLM's search space; it transfers to ChatAssert* with a 31% relative PR gain, which argues the component is not just a trick. The Judge-vs-Debate/Voting comparison is interesting, with McNemar tests showing the override behavior is net positive. The authors also do the right thing by diagnosing TOGLL's training-set overlap and retraining it; that kind of self-check is rare. The replication package is a plus.

Soft spots, in order of severity. The RAG corpus is the big one. Section III-B describes retrieving the test case with highest Jaccard similarity, but never says from what pool. If the pool is Defects4J's own test methods, then for a given input prefix the most similar case can be the same test minus the removed assertion, or a sibling in the same class, with the expected value right there. That would inflate RAGA's standalone metrics and any selection where the Judge picks the RAG candidate. The paper explicitly flags leakage in TOGLL yet does not disclose or decontaminate its own retrieval pool. This is not a stylistic quibble; it's the load-bearing premise for 'diverse perspective aggregation.'

Two smaller issues. First, the BDR definition doesn't state that generated assertions are run on the fixed version to filter out false positives; a wrong assertion could fail on both buggy and fixed versions and count as a detected bug. The threats section mentions using human-written tests, but that's not the same as filtering generated assertions. Second, RQ2's mutation results are reported as single numbers with no variance or runs; a few repeated runs or at least a statement that it's a single run would help. Also, the paper criticizes ChatAssert for oversampling and then uses 15 candidates per input; the difference is reranking, which is fine, but the abstract oversells it a bit.

Recommendation: this deserves a serious referee. The architecture and engineering look credible, the gains are large even before accounting for the RAG issue (CSR 76.78 vs 50.21 for ChatAssert*), and ActVCon's transferability is a nice result. But the revision must specify the retrieval corpus, run the BDR evaluation with a fixed-version pass check, and add variance or a clear statement about RQ2. If the corpus is indeed overlapping, the RAG ablation numbers change and the paper needs to be substantially revised. I'd invite them to resubmit after fixing these.

Referee Report

3 major / 5 minor

Summary. The paper proposes AssertMate, an agent-based framework for test assertion generation that decomposes the task into an actual-value constructor (ActVCon), which identifies assertion targets via static analysis and type-aware heuristics, and a multi-perspective expected-value generator (EVGen) that combines code-generation, retrieval-augmented generation, and chain-of-thought agents, followed by an LLM-as-a-Judge collaboration stage. The framework is evaluated on Defects4J human-written and EvoSuite-generated test prefixes, reporting large gains over baselines such as ChatAssert*, EditAS, TOGLL, and RetriGen on compilation success rate, pass rate, and bug detection rate, plus integration results with EvoSuite and ablation studies. The paper claims that each component contributes complementary gains and that the framework works with a smaller open-source model.

Significance. If the reported results hold, AssertMate would be a substantial advance in LLM-based assertion generation, improving both syntactic validity and fault detection while using a relatively small open-source model. The paper's strengths include an extensive Defects4J evaluation, repeated runs with statistical testing (Wilcoxon signed-rank, McNemar), a detailed ablation study, and a released replication package. The decomposition of assertion generation into actual-value construction and multi-perspective expected-value prediction is a thoughtful and potentially reusable design. However, three load-bearing concerns—the undisclosed RAG retrieval corpus, a contradiction in the ablation table regarding the contribution of EVGen, and the lack of an explicit false-positive filter in the bug-detection metric—prevent the results from being accepted at face value.

major comments (3)

[Section III-B] The RAG agent is described as retrieving 'the test case with the highest Jaccard similarity to the input test prefix' and using it as a one-shot example, but the paper never specifies the retrieval corpus. This is a load-bearing omission because the evaluation inputs are 667 FM-TP pairs constructed by removing the target assertion from Defects4J human-written tests (Section IV-A); if the retrieval corpus contains those test methods or near-duplicates, the retrieved example can contain the expected assertion, leaking ground truth into the prompt. This would inflate the standalone RAGA metrics in Table III and any full-system result where the Judge selects the RAG candidate. Section VI.E discusses contamination of the backbone LLM but does not address the retrieval pool. Please specify the corpus, confirm that it does not overlap Defects4J, and describe any decontamination steps.
[Table III and Section V-C] In Table III, the variant ChatAssert*+ (ChatAssert with ActVCon integrated) achieves a BDR of 48.21%, which is higher than the full AssertMate's 46.25%. The text in Section V-C claims that EVGen's collaborative design leads to relative improvements in CSR, PR, and BDR over the best standalone agent, but it does not acknowledge that the full system is worse than ChatAssert*+ on BDR. This directly contradicts the paper's claim that each of the three components makes a significant and complementary contribution. Please explain this discrepancy or re-analyze the contribution of EVGen and the Judge relative to the ActVCon-only variant.
[Section IV-C and Section V-A] The Bug Detection Rate (BDR) is defined as the ratio of detected bugs to total applicable bugs, but the paper does not specify whether a generated assertion must pass on the fixed version to count as a true detection. A generated assertion that fails on both the buggy and fixed versions would be a false positive, yet the metric as stated could count it as a successful detection. Section VI.E says the human-written bug-triggering tests are expected to pass on the fixed version and fail on the buggy version, but generated assertions are not guaranteed to satisfy this property. Please explicitly state the false-positive filtering procedure for BDR, or report BDR with and without such filtering.

minor comments (5)

[Section III-B] There is a typo in the sentence 'EVGen first generates the matching expected values ... though multi-perspective inference'; 'though' should be 'through'.
[Table I] The Token Overhead and Average Time entries for EditAS, TOGLL, and RetriGen are marked with '–'; please clarify whether these were not measured or are not applicable, and if not measured, why not.
[Section IV-A] The paper states that 112 eligible bugs were identified, but it does not report how many of those remain after focal-method matching to form the 667 FM-TP pairs; reporting this dropout would help readers assess dataset composition.
[Section V-B] The phrase 'Data new dataset' should be 'new dataset'.
[Table II] The row label 'TOGLL_DN*' is not explained in the table caption or the surrounding text; please introduce the notation clearly.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity: AssertMate's headline results are benchmark evaluations, not quantities derived from their inputs.

full rationale

We found no circular step in the claimed derivation chain. AssertMate's central results (Section V-A, Table I) are external-benchmark comparisons on Defects4J human-written FM-TP pairs, and the framework does not fit any parameter to those pairs; the components (ActVCon, CGA, RAGA, CoTA, and the Judge agent) are ablated and measured rather than defined in terms of the reported CSR, PR, or BDR metrics. The RAG agent in Section III-B retrieves a one-shot example using Jaccard similarity, but the paper does not identify the retrieval corpus; this is a potential data-leakage threat to validity, not a circularity, because the paper nowhere states that the corpus contains the evaluation pairs, and no reported quantity is algebraically forced by the retrieval input. The only self-citations ([5] and [27]) support design choices such as prompt prefill and backbone-model selection; these choices are independently validated by the ablation and comparison experiments and do not carry the central claim. No equation in the paper reduces a prediction to an input by construction, and no uniqueness theorem or fitted-parameter renaming is used. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on heuristic design choices and domain assumptions rather than mathematical axioms. No free parameter is fitted to the Defects4J test set, which keeps the ledger relatively clean, but the unspecified RAG retrieval pool leaves an open question about whether one perspective is anchored to the evaluation data.

free parameters (6)

num_samples_per_agent = 5
Oversampling count per perspective; chosen to match prior work (ChatAssert, ATLAS) where top-5 gains plateau, not fitted to Defects4J.
temperature = 1.0
Sampling temperature to encourage diversity across candidates; fixed design choice.
top_p = 1.0
No probability truncation; fixed design choice.
max_tokens = 4000
Response length limit; fixed design choice.
ActVCon target rules = return value + public getter chains
Heuristic scope of what to test; covers 86.5% of targets in developer-written tests in the study, but excludes parameterized invocation targets.
perspective set = CGA, RAGA, CoTA
The three prompt strategies were chosen based on a survey of prior work; their combination is evaluated but not optimized on the test set.

assumptions (5)

domain assumption Average token probability in the assertion portion is a reliable confidence signal for ranking sampled responses.
Invoked in Section III-B 'Probability-Based Response Re-Rank' to select one candidate per agent without execution feedback.
domain assumption Human-written bug-triggering tests from Defects4J, when run on the fixed version, define the correct intended behavior for evaluating pass rate and bug detection.
Used in Section IV-C metric definitions; the paper acknowledges uncertainty about focal-method correctness and relies on this ground truth.
domain assumption The most Jaccard-similar test prefix retrieved from the unspecified corpus is a useful one-shot example for predicting the expected value.
RAGA's design in Section III-B builds on EditAS's retrieval algorithm, but the retrieval pool is not validated or described.
domain assumption Five assertion function types (assertEquals, assertTrue, assertFalse, assertNull, assertNotNull) cover a sufficient share of real assertions.
Section III-A relies on Yu et al.'s empirical finding of 86% coverage; the framework cannot express other assertion styles.
domain assumption The focal method and test prefix contain enough information for the LLM agents to infer correct expected values without seeing the fixed version.
Foundation of the whole approach; the paper notes this is challenging and that pass rate remains 60%, indicating the assumption holds only partially.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Agent-Based Test Assertion Generation via Diverse Perspective Aggregation." pith.science (2026). https://pith.science/paper/25IQSDMB

@misc{pith2026260805822,
  author       = {Pith},
  title        = {Pith review of: Agent-Based Test Assertion Generation via Diverse Perspective Aggregation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25IQSDMB}},
  note         = {Machine review of arXiv:2608.05822}
}

read the original abstract

Test assertions are critical elements of unit tests, serving as checkpoints to validate expected behavior and ensure software correctness. Numerous techniques have been proposed to automate assertion generation, with recent progress notably driven by large language models (LLMs). Despite the promise, existing approaches such as ChatAssert suffer from modest accuracy, heavy reliance on oversampling, and vulnerability to model randomness due to one-shot prompting. To address these limitations, we propose AssertMate, a novel agent-based assertion generation framework that enhances the quality and reliability of LLM-generated assertions through three key components: (1) actual value construction that identifies assertion targets via static analysis and type-aware heuristics; (2) multi-perspective expected value prediction using code generation, retrieval-augmented generation (RAG), and chain-of-thought (CoT) reasoning agents; and (3) an LLM-as-a-Judge collaboration mechanism to select the most appropriate assertion. Evaluation on the Defects4J benchmark demonstrates that AssertMate significantly outperforms state-of-the-art techniques in compilation success and pass rates, along with substantially higher bug detection capabilities. Integration with EvoSuite further validates AssertMate's practicality, yielding superior mutation coverage and kill counts. Ablation studies reveal that each of the three components makes a significant and complementary contribution to the overall performance. This work affirms the great potential of aggregating diverse perspectives to enhance the effectiveness of LLM-based assertion generation.

Figures

Figures reproduced from arXiv: 2608.05822 by the authors.

**Figure 3.** Overview of the agent-based test assertion generation [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Example of LLMs’ response tokens and probabilities [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: A representative case study of bound parameter esti [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 46 canonical work pages

[1]

Chatassert: Llm- based test oracle generation with external tools assis- tance,

I. Hayet, A. Scott, and M. d’Amorim, “Chatassert: Llm- based test oracle generation with external tools assis- tance,”IEEE Trans. Software Eng., vol. 51, no. 1, pp. 305–319, 2025

work page 2025
[2]

Evaluating and improving chatgpt for unit test generation,

Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving chatgpt for unit test generation,”Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 1703–1726, 2024

2024
[3]

Chatunitest: A framework for llm-based test genera- tion,

Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test genera- tion,” inSIGSOFT FSE Companion, 2024, pp. 572–576

work page 2024
[4]

Revis- iting and improving retrieval-augmented deep assertion generation,

W. Sun, H. Li, M. Yan, Y . Lei, and H. Zhang, “Revis- iting and improving retrieval-augmented deep assertion generation,” inASE, 2023, pp. 1123–1135

work page 2023
[5]

On the evaluation of large language models in unit test generation,

L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wang, and J. Chen, “On the evaluation of large language models in unit test generation,” inASE, 2024, pp. 1607–1619

work page 2024
[6]

On learning meaningful assert state- ments for unit test cases,

C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On learning meaningful assert state- ments for unit test cases,” inICSE, 2020, pp. 1398–1409

work page 2020
[7]

Automated assertion generation via information retrieval and its integration with deep learning,

H. Yu, Y . Lou, K. Sun, D. Ran, T. Xie, D. Hao, Y . Li, G. Li, and Q. Wang, “Automated assertion generation via information retrieval and its integration with deep learning,” inICSE, 2022, pp. 163–174

work page 2022
[8]

TOGLL: correct and strong test oracle generation with llms,

S. B. Hossain and M. B. Dwyer, “TOGLL: correct and strong test oracle generation with llms,”CoRR, vol. abs/2405.03786, 2024

arXiv 2024

Show all 51 references

[9]

Java enterprise edition support in search-based junit test generation,

A. Arcuri and G. Fraser, “Java enterprise edition support in search-based junit test generation,” inSSBSE, vol. 9962, 2016, pp. 3–17

2016
[10]

Randoop: feedback- directed random testing for java,

C. Pacheco and M. D. Ernst, “Randoop: feedback- directed random testing for java,” inOOPSLA Compan- ion, 2007, pp. 815–816. IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING 15

2007
[11]

SELECT - a formal system for testing and debugging programs by symbolic execution,

R. S. Boyer, B. Elspas, and K. N. Levitt, “SELECT - a formal system for testing and debugging programs by symbolic execution,” inReliable Software. ACM, 1975

1975
[12]

Symbolic testing and the DISSECT symbolic evaluation system,

W. E. Howden, “Symbolic testing and the DISSECT symbolic evaluation system,”IEEE Trans. Software Eng., vol. 3, no. 4, pp. 266–278, 1977

1977
[13]

A new approach to program testing,

J. C. King, “A new approach to program testing,” inPro- gramming Methodology, ser. Lecture Notes in Computer Science, vol. 23. Springer, 1974, pp. 278–290

1974
[14]

General- ized symbolic execution for model checking and testing,

S. Khurshid, C. S. Pasareanu, and W. Visser, “General- ized symbolic execution for model checking and testing,” inTACAS, ser. Lecture Notes in Computer Science, vol
[15]

KLEE: unas- sisted and automatic generation of high-coverage tests for complex systems programs,

C. Cadar, D. Dunbar, and D. R. Engler, “KLEE: unas- sisted and automatic generation of high-coverage tests for complex systems programs,” inOSDI. USENIX Association, 2008, pp. 209–224

2008
[16]

Symbolic pathfinder: symbolic execution of java bytecode,

C. S. Pasareanu and N. Rungta, “Symbolic pathfinder: symbolic execution of java bytecode,” inASE. ACM, 2010, pp. 179–180

2010
[17]

Mining test oracles of web search engines,

W. Zheng, H. Ma, M. R. Lyu, T. Xie, and I. King, “Mining test oracles of web search engines,” inASE, 2011, pp. 408–411

2011
[18]

@tcomment: Testing javadoc comments to detect comment-code inconsistencies,

S. H. Tan, D. Marinov, L. Tan, and G. T. Leav- ens, “@tcomment: Testing javadoc comments to detect comment-code inconsistencies,” inICST. IEEE Com- puter Society, 2012, pp. 260–269

2012
[19]

Auto- matic generation of oracles for exceptional behaviors,

A. Goffi, A. Gorla, M. D. Ernst, and M. Pezz `e, “Auto- matic generation of oracles for exceptional behaviors,” inISSTA. ACM, 2016, pp. 213–224

2016
[20]

A survey of symbolic execution techniques,

R. Baldoni, E. Coppa, D. C. D’Elia, C. Demetrescu, and I. Finocchi, “A survey of symbolic execution techniques,” ACM Comput. Surv., vol. 51, no. 3, pp. 50:1–50:39, 2018

2018
[21]

TOGA: A neural method for test oracle generation,

E. Dinella, G. Ryan, T. Mytkowicz, and S. K. Lahiri, “TOGA: A neural method for test oracle generation,” in ICSE. ACM, 2022, pp. 2130–2141

2022
[22]

Self-contrast: Better reflection through in- consistent solving perspectives,

W. Zhang, Y . Shen, L. Wu, Q. Peng, J. Wang, Y . Zhuang, and W. Lu, “Self-contrast: Better reflection through in- consistent solving perspectives,” inACL (1). Association for Computational Linguistics, 2024, pp. 3602–3622

2024
[23]

Judging llm-as-a-judge with mt-bench and chatbot arena,

L. Zheng, W. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” inNeurIPS, 2023

2023
[24]

Defects4j: a database of existing faults to enable controlled testing studies for java programs,

R. Just, D. Jalali, and M. D. Ernst, “Defects4j: a database of existing faults to enable controlled testing studies for java programs,” inISSTA. ACM, 2014, pp. 437–440

2014
[25]

Homepage,

“Homepage,” Accessed: 2025, https://github.com/ LeonYang95/AssertMate

2025
[26]

Large language models for equivalent mutant detection: How far are we?

Z. Tian, H. Shu, D. Wang, X. Cao, Y . Kamei, and J. Chen, “Large language models for equivalent mutant detection: How far are we?” inISSTA, 2024, pp. 1733–1745

2024
[27]

Clarifying semantics of in-context examples for unit test generation,

C. Yang, L. Yang, Z. Wang, D. Wang, J. Zhou, and J. Chen, “Clarifying semantics of in-context examples for unit test generation,”ASE, 2025

2025
[28]

Improving deep assertion generation via fine-tuning retrieval-augmented pre-trained language models,

Q. Zhang, C. Fang, Y . Zheng, Y . Zhang, Y . Zhao, R. Huang, J. Zhou, Y . Yang, T. Zheng, and Z. Chen, “Improving deep assertion generation via fine-tuning retrieval-augmented pre-trained language models,”ACM Trans. Softw. Eng. Methodol., 2025

2025
[29]

Assert5: Test assertion generation using a fine-tuned code language model,

S. Primbs, B. Fein, and G. Fraser, “Assert5: Test assertion generation using a fine-tuned code language model,” arXiv preprint arXiv:2502.02708, 2025

2025 arXiv
[30]

Learning deep semantics for test completion,

P. Nie, R. Banerjee, J. J. Li, R. J. Mooney, and M. Glig- oric, “Learning deep semantics for test completion,” in ICSE. IEEE, 2023, pp. 2111–2123

2023
[31]

Unit test case generation with transformers,

M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers,”CoRR, vol. abs/2009.05617, 2020

2009 arXiv
[32]

Hybrid automated program repair by combining large language models and program analysis,

F. Li, J. Jiang, J. Sun, and H. Zhang, “Hybrid automated program repair by combining large language models and program analysis,”ACM TOSEM, 2024

2024
[33]

Variable- based fault localization via enhanced decision tree,

J. Jiang, Y . Wang, J. Chen, D. Lv, and M. Liu, “Variable- based fault localization via enhanced decision tree,”ACM Trans. Softw. Eng. Methodol., pp. 1–32, 2023

2023
[34]

Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t),

S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t),” inASE, 2015, pp. 201–211

2015
[35]

Long short-term memory,

S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural Comput., pp. 1735–1780, 1997

1997
[36]

Codegen: An open large language model for code with multi-turn program synthesis,

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” inICLR. OpenReview.net, 2023

2023
[37]

Unixcoder: Unified cross-modal pre-training for code representation,

D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” inACL (1), 2022, pp. 7212–7225

2022
[38]

Fixing large language models’ specification misunderstanding for better code generation,

Z. Tian, J. Chen, and X. Zhang, “Fixing large language models’ specification misunderstanding for better code generation,” inICSE. IEEE, 2025, pp. 1514–1526

2025
[39]

Deepseek-coder: When the large language model meets programming - the rise of code intelli- gence,

D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . K. Li, F. Luo, Y . Xiong, and W. Liang, “Deepseek-coder: When the large language model meets programming - the rise of code intelli- gence,”CoRR, vol. abs/2401.14196, 2024

2024 arXiv
[40]

Chatassert homepage,

“Chatassert homepage,” Accessed: 2025, https://github. com/ncsu-swat/chatassert

2025
[41]

Tree sitter: An incremental parsing system for program- ming tools,

“Tree sitter: An incremental parsing system for program- ming tools,” Accessed: 2025, https://tree-sitter.github.io/ tree-sitter/

2025
[42]

Pytorch,

“Pytorch,” Accessed: 2025, https://pytorch.org

2025
[43]

Huggingface transformers library,

“Huggingface transformers library,” Accessed: 2025, https://huggingface.co/docs/transformers/index

2025
[44]

vllm library,

“vllm library,” Accessed: 2025, https://github.com/ vllm-project/vllm

2025
[45]

Wilcoxon signed-rank test,

R. F. Woolson, “Wilcoxon signed-rank test,” inWiley Encyclopedia of Clinical Trials. John Wiley & Sons, 2007, pp. 1–3

2007
[46]

Dominance statistics: Ordinal analyses to an- swer ordinal questions,

N. Cliff, “Dominance statistics: Ordinal analyses to an- swer ordinal questions,”Psychological Bulletin, vol. 114, no. 3, pp. 494–509, 1993

1993
[47]

A large-scale evaluation of automated unit test generation using evosuite,

G. Fraser and A. Arcuri, “A large-scale evaluation of automated unit test generation using evosuite,”ACM Trans. Softw. Eng. Methodol., pp. 8:1–8:42, 2014

2014
[48]

Ex- IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING 16 amining inter-consistency of large language models col- laboration: An in-depth analysis via debate,

K. Xiong, X. Ding, Y . Cao, T. Liu, and B. Qin, “Ex- IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING 16 amining inter-consistency of large language models col- laboration: An in-depth analysis via debate,” inEMNLP (Findings), 2023, pp. 7572–7590

2023
[49]

Large language model based multi-agents: A survey of progress and challenges,

T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges,” inIJCAI. ijcai.org, 2024, pp. 8048–8057

2024
[50]

Which is the correct statistical test to use?

E. McCrum-Gardner, “Which is the correct statistical test to use?”British Journal of Oral and Maxillofacial Surgery, vol. 46, no. 1, pp. 38–41, 2008

2008
[2619]

Springer, 2003, pp. 553–568

2003

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Chatassert: Llm- based test oracle generation with external tools assis- tance,

I. Hayet, A. Scott, and M. d’Amorim, “Chatassert: Llm- based test oracle generation with external tools assis- tance,”IEEE Trans. Software Eng., vol. 51, no. 1, pp. 305–319, 2025

work page 2025

[2] [2]

Evaluating and improving chatgpt for unit test generation,

Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving chatgpt for unit test generation,”Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 1703–1726, 2024

2024

[3] [3]

Chatunitest: A framework for llm-based test genera- tion,

Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test genera- tion,” inSIGSOFT FSE Companion, 2024, pp. 572–576

work page 2024

[4] [4]

Revis- iting and improving retrieval-augmented deep assertion generation,

W. Sun, H. Li, M. Yan, Y . Lei, and H. Zhang, “Revis- iting and improving retrieval-augmented deep assertion generation,” inASE, 2023, pp. 1123–1135

work page 2023

[5] [5]

On the evaluation of large language models in unit test generation,

L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wang, and J. Chen, “On the evaluation of large language models in unit test generation,” inASE, 2024, pp. 1607–1619

work page 2024

[6] [6]

On learning meaningful assert state- ments for unit test cases,

C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On learning meaningful assert state- ments for unit test cases,” inICSE, 2020, pp. 1398–1409

work page 2020

[7] [7]

Automated assertion generation via information retrieval and its integration with deep learning,

H. Yu, Y . Lou, K. Sun, D. Ran, T. Xie, D. Hao, Y . Li, G. Li, and Q. Wang, “Automated assertion generation via information retrieval and its integration with deep learning,” inICSE, 2022, pp. 163–174

work page 2022

[8] [8]

TOGLL: correct and strong test oracle generation with llms,

S. B. Hossain and M. B. Dwyer, “TOGLL: correct and strong test oracle generation with llms,”CoRR, vol. abs/2405.03786, 2024

arXiv 2024