{"id":"4affab46-f1be-4fc3-8260-019989b427ea","arxiv_id":"2607.09101","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A three-agent LLM workflow plus a test-specialized knowledge graph generates unit tests that beat prior LLM and search-based tools on coverage and mutation score.","lead":"TestAgent is a multi-agent LLM system that writes unit tests by mimicking how developers plan requirements, generate tests, and review them, backed by a project-wide code knowledge graph. It reports higher coverage and mutation scores than prior LLM and search-based tools on Java and Python codebases.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Single-run metrics without variance leave the superiority claims (esp. mutation and coverage rankings) exposed to LLM non-determinism.","rationale":"The reader correctly isolates the single most load-bearing soft spot for the strongest empirical claim. The rest of the paper (ablation showing KG contribution, industrial/Python/user-study extensions, cost-utility plot) is solid and internally consistent; the single-run limitation is already acknowledged by the authors and is the natural reason for a CONDITIONAL rather than ACCEPT verdict. No stronger internal inconsistency or hidden assumption was found that would move the verdict further. Multi-seed replication (or public artifacts enabling it) would resolve the residual uncertainty without altering the methodological contribution.","tokens_in":22208,"tokens_out":485,"duration_ms":23392,"concrete_test":"Re-execute the full TestAgent pipeline (GPT-4o backbone, identical tool/KG settings) and the two LLM baselines on the two hardest projects (Gson + Ruler) for three independent seeds; recompute the six metrics and their std. dev. If any baseline mean exceeds TestAgent’s on mutation score or branch coverage within one standard deviation, the superiority claim requires qualification or multi-seed reporting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on aggregate point estimates (97.46% exe, 92.34% line, 90.24% branch, 83.69% mut) from one run per configuration (Table III, RQ1; also RQ3–RQ6). Section V-C (Threats) explicitly notes that monetary/time cost forced single runs and that LLM randomness can affect individual metrics; the three-LLM voting step for defect confirmation (§III-D.2, Discussion B) is likewise unrepeated. Ablation (Table IV) and cross-LLM results (Table V) show large absolute gaps, yet without seed-level std. dev. or multi-run ranking stability it is unknown whether the ordering vs. ChatUniTest/HITS (same GPT-4o) or the 83.69% vs. 43.59% gap vs. EvoSuite would reverse on re-execution. Consistent trends across projects/languages mitigate but do not quantify the risk.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes TestAgent, a multi-agent LLM framework for unit test generation that emulates a human testing workflow with three specialized agents (requirement planner, test generator, test reviewer), a test-specialized repository knowledge graph built by static analysis, and on-demand tool APIs. On six Java projects it reports 97.46% execution rate, 92.34% line coverage, 90.24% branch coverage, and 83.69% mutation score, outperforming ChatUniTest and HITS under the same GPT-4o backbone and substantially exceeding EvoSuite on mutation score (83.69% vs 43.59%). Supporting evidence includes ablations (knowledge graph removal is the largest drop), three LLM backends, a private industrial set, a Python port vs CodaMosa/CoverUp, a small user study, and 154/167 confirmed bug-revealing non-regression tests (92.22% precision).","tokens_in":22594,"tokens_out":1005,"duration_ms":11192,"significance":"If the results hold under re-execution, the work is a solid empirical contribution to LLM-based software testing: it couples a human-inspired multi-agent loop with a test-specific knowledge graph that stores both static dependencies and dynamic testing artifacts, and it shows large gains on mutation score and cross-language transfer. Strengths include a clear ablation isolating the knowledge graph, multi-backend and industrial evaluation, a Python reimplementation, and a careful manual analysis of real-bug detection. These elements go beyond typical single-prompt LLM test generators and are of practical interest to the SE community.","major_comments":[{"comment":"Table III (RQ1) and the abstract state superiority on coverage and mutation from single-run aggregates. Section V-C (Threats) explicitly notes that cost forced single runs and that LLM non-determinism can affect individual metrics; no seed-level variance, multi-run ranking stability, or error bars are reported for the same GPT-4o backbone vs ChatUniTest/HITS or for the 83.69% vs 43.59% gap vs EvoSuite. Given that the central claim is comparative superiority, at least a multi-seed re-run on a subset of projects (or bootstrap intervals) is needed to show that the ordering is stable; large absolute gaps mitigate but do not quantify the risk.","section":null},{"comment":"Section III-D.2 and Discussion B rest the non-regression bug claim on a three-LLM voting confirmation (o3-mini, DeepSeek-R1, Qwen3-235B) that a failure is a real defect rather than a bad test. Pairwise Cohen's κ is only fair-to-moderate (0.39–0.55) and individual positive rates differ substantially (21–46%). The 154/167 (92.22%) figure is useful but depends on this unvalidated axiom and a single manual review pass; the paper should either validate the vote against a larger human-labeled set or present the 154 as human-confirmed candidates with the vote as a filter, not as an automatic ground truth.","section":null}],"minor_comments":[{"comment":"Table II: Lang and Chart are subsampled to 200 methods each without a stated sampling procedure or seed; a short note on stratification (or lack thereof) would improve reproducibility.","section":null},{"comment":"Section IV-A.5: free parameters (max recursion depth 50, top-5 requirements, Jaccard threshold k, elastic 30/40-turn context control) are listed but not sensitivity-analyzed; a brief note on robustness would help.","section":null},{"comment":"Figure 3 (cost-effectiveness) uses bubble size for time but the legend is only in the caption; an explicit legend would improve readability.","section":null},{"comment":"Throughout: the system is alternately written TESTAGENT / TestAgent / TESTAGENT; pick one spelling for the camera-ready version.","section":null},{"comment":"Related work (II-B) correctly distinguishes the test-specialized KG from RepoGraph/AutoCodeRover; a short explicit comparison table of node/edge types would make the novelty sharper.","section":null}],"recommendation":"major_revision","confidential_remarks":"The empirical package is stronger than many concurrent LLM-for-testing papers (ablations, industrial set, Python port, real-bug analysis). The single-run and voting issues are fixable within revision and do not look like fabrication risk; I would not reject on novelty grounds. Fit for a solid SE journal is good if the variance and bug-confirmation points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a competent systems paper in LLM-based unit testing. What is actually new is not multi-agent scaffolding or a repo graph in the abstract—those already exist—but a test-specialized knowledge graph (test nodes/links plus persistent artifacts such as summaries, reports, and root-cause notes) wired into a three-agent planner/generator/reviewer loop that can call tools on demand and vote on whether a failure is a real defect. That combination is concrete and useful.\n\nThey do the evaluation work. Six Java projects, head-to-head against EvoSuite, ChatUniTest, and HITS under the same GPT-4o backbone; ablations that show the KG is the largest lever (+22% line, +27% mutation when present); three backends including an open model that still beats ChatUniTest+GPT-4o; a private industrial set; a Python port that beats CodaMosa and CoverUp; a small user study on readability; and 154/167 confirmed bug-revealing tests at 92% precision. Mutation score is the standout number (83.69% vs EvoSuite’s 43.59%). Metrics come from external tools (JaCoCo, PITest), so the circularity burden is low. Citation pattern is normal for the subfield.\n\nSoft spots are real but proportionate. Everything is single-run; the threats section admits cost forced that choice and that LLM non-determinism can move individual metrics. The three-model defect vote is likewise unreplicated. Free parameters (recursion depth, top-k requirements, Jaccard threshold, force-turn cutoffs) are hand-set. No public code or data is mentioned in the text, and the industrial subjects are anonymized. Those gaps do not erase the absolute gaps in the tables, but they do mean the ranking claims are not yet locked down.\n\nThis is for people building or evaluating LLM test generators and for industrial tooling groups that care about mutation and readability, not just coverage. It is serious empirical SE work, not a conceptual sketch. I would send it to peer review; referees should demand multi-seed numbers and artifact release, but the design and breadth already justify the time.","headline":"Solid multi-agent test-gen system with a test-specialized KG; big mutation gains over EvoSuite and LLM baselines, but single-run numbers and no public artifacts keep the ranking claims provisional.","tokens_in":23188,"tokens_out":542,"would_cite":true,"duration_ms":6434,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Three collaborating LLM agents that plan, write, and review unit tests beat both other LLM tools and search-based generators on coverage and mutation score.","keywords":["unit test generation","large language models","multi-agent systems","knowledge graph","code coverage","mutation testing","software testing"],"falsifier":"Re-run the entire Java suite under identical GPT-4o settings for several independent seeds and check whether TestAgent still ranks first on line coverage, branch coverage, and especially mutation score against ChatUniTest, HITS, and EvoSuite; a reversal on any of those three metrics would falsify the central performance claim.","tokens_in":23120,"feed_emoji":"🧪","tokens_out":696,"duration_ms":7793,"temperature":0.7,"pith_summary":"Writing good unit tests is slow and hard; prior LLM tools either follow a fixed generate-then-fix loop or pull coarse context that misses the real dependencies of the method under test. This paper claims that the gap can be closed by copying how human developers actually work: first plan what should be tested, then generate and repair tests, then review adequacy and feed suggestions back. TestAgent implements that loop with three specialized agents that can call tools on demand and that read and write a repository-level knowledge graph built for testing (entities, call and inheritance links, plus stored summaries, reports, and failure analyses). On six Java projects the system reaches 97.46% execution rate, 92.34% line coverage, 90.24% branch coverage, and 83.69% mutation score, beating same-backbone LLM baselines on every metric and roughly doubling the mutation score of a strong search-based tool. The same design ports to Python and to industrial codebases, and non-regression runs surface 154 confirmed real bugs at 92% precision. A sympathetic reader cares because the result is immediately usable tests that are both more thorough and more readable than what either pure search or single-shot LLM pipelines currently produce.","feed_headline":"Three LLM agents write unit tests that kill more mutants","feed_subtitle":"Human-style plan-generate-review plus a test knowledge graph beats both LLM and search baselines","key_machinery":"The multi-agent human-testing-inspired workflow plus a test-specialized knowledge graph: three agents that plan requirements, generate/repair tests, and review them, each able to call graph-manipulation, retrieval, and runtime tools on demand, while the graph stores both static code relations and the testing artifacts produced during generation.","core_discovery":"Emulating the human unit-testing workflow with three collaborating LLM agents (requirement planner, test generator, test reviewer) that dynamically invoke tools and reason over a test-specialized repository knowledge graph produces unit tests whose correctness, coverage, and fault-detection power exceed both procedural LLM baselines and search-based generators under comparable conditions.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Three LLM agents plan, write and review unit tests that kill more mutants","Multi-agent LLMs mimic human testing to raise coverage and mutation scores","Planner-generator-reviewer agents plus knowledge graph beat LLM baselines","Human-style LLM trio generates unit tests with higher fault detection","TestAgent agents adaptively tool-use for stronger repository unit tests"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the single-run aggregate numbers (and the three-model vote that labels a failing test as a real defect) are stable enough to ground the superiority claims, even though language-model randomness is acknowledged and multi-seed variance is not reported.","fun_headline_variants_meta":{"raw":{"variants":["Three LLM agents plan, write and review unit tests that kill more mutants","Multi-agent LLMs mimic human testing to raise coverage and mutation scores","Planner-generator-reviewer agents plus knowledge graph beat LLM baselines","Human-style LLM trio generates unit tests with higher fault detection","TestAgent agents adaptively tool-use for stronger repository unit tests"]},"model":"grok-4.5","effort":"low","cost_usd":0.003972,"raw_usage":{"total_tokens":1298,"prompt_tokens":858,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":39720000,"prompt_tokens_details":{"text_tokens":858,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":365,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":858,"tokens_out":75,"duration_ms":5096,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T05:23:43.941479+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the entire Java suite under identical GPT-4o settings for several independent seeds and check whether TestAgent still ranks first on line coverage, branch coverage, and especially mutation score against ChatUniTest, HITS, and EvoSuite; a reversal on any of those three metrics would falsify the central performance claim.","supporting_citations":[],"review_version":1}