{"id":"7304ebdc-1a9c-4757-98cb-05458644ae3c","arxiv_id":"2607.03194","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Tracking individual static and feedback testing objectives across a coverage-then-mutation two-stage LLM workflow raises line/branch coverage and mutation score on complex Java methods versus KTester and PANTA.","lead":"TATG makes LLM unit-test generators track each branch, setup, and assertion goal so later rounds stop redoing finished work and harden weak oracles. On 141 hard Java methods it beats recent open baselines on coverage and mutation score and matches an industrial tool on a subset.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The headline gains are not isolated to tracking; the paper never ablates status/de-duplication/acceptance against an otherwise matched two-stage feedback loop.","rationale":"The reader correctly flags the missing isolation of tracking. The manuscript’s strongest numerical claim is well-supported as an end-to-end comparison (Table III, Wilcoxon tests, two LLMs, industrial subset), and the stage-order ablation is informative. The soft spot is causal attribution of those gains to the objective-state machinery rather than confounds that co-vary with it. That is precisely the condition that must hold for the paper’s framing (“tracking-aware testing objective”) to be the explanation rather than a convenient packaging of richer analysis plus staged prompting. A single matched unstructured-feedback control would settle it; until then CONDITIONAL remains the right verdict, with no need to move to REJECT or UNVERDICTED. Code release would further reduce residual risk but is secondary to the isolation gap.","tokens_in":17302,"tokens_out":506,"duration_ms":4817,"concrete_test":"Implement a matched control: same static synthesizers, same two-stage schedule (5+5), same B_min/B_max/λ, same models, but render every active signal as free-form prompt text without status/key/α updates or de-duplication retirement. Re-run the 141 methods; if line/branch/mutation deltas vs PANTA shrink by more than ~half of the reported margins, the tracking claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim attributes the large average gains over PANTA (line +22.15, branch +20.14, mutation +37.66 pp) to the tracking-aware objective representation (status, de-duplication keys, acceptance predicates, objective cards). RQ2 only removes the structural stage and shows that hardening alone collapses; it does not hold static analysis content, two-stage budget, prompt structure, and iteration count fixed while replacing tracked objectives with unstructured injection of the same signals. Table III and Section V-B therefore leave open that richer static projection (Table I), staged focus, or longer structured prompts—not the tracking machinery itself—drive most of the lift. Without that isolation, the load-bearing causal story for “tracking-aware” remains under-supported even though the end-to-end empirical advance is real.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes TATG, an LLM-based unit test generation framework whose central idea is a unified, tracked objective representation G = ⟨id, src, cat, E, S, A, Q, α, status, key⟩ that encodes branch-level testing requirements from static analysis and dynamic feedback (compilation/execution, coverage, oracle quality, mutation). Objectives are rendered as structured “objective cards” in the prompt and updated across iterations. Generation is organized as a two-stage process: structural rounds first improve reachability and coverage; hardening rounds then use mutation feedback to strengthen oracles. On 141 complex Java methods (110 KTester subjects plus 31 additional methods), TATG is reported to improve line coverage, branch coverage, and mutation score by 22.15, 20.14, and 37.66 percentage points on average over PANTA across two open-source LLMs, with paired Wilcoxon significance, stage-level results, an ablation that removes the structural stage, and a subset comparison against a proprietary industrial tool.","tokens_in":17632,"tokens_out":1002,"duration_ms":17266,"significance":"If the results hold under fair isolation of the claimed mechanism, this is a solid empirical systems contribution to LLM-based unit test generation. The paper addresses a real pain point—complex, state-dependent Java methods where coverage and fault detection require branch-specific setup, entry routes, and strong oracles—and unifies signals that prior tools (KTester, PANTA, MUTGEN) handle separately. Strengths include a non-trivial benchmark (mean CC 18.5), multiple baseline categories, two model scales, method-level JaCoCo/PITest metrics, Wilcoxon tests, stage-level reporting in Table III, and an industrial-tool comparison on a 50-method subset. The motivating example and the structural-before-hardening design rationale are clear and practically useful. The main open question is causal attribution of the large gains specifically to tracking (status, de-duplication keys, acceptance predicates) rather than to the broader system of richer static projection, staged budgets, and structured prompts.","major_comments":[{"comment":"The paper’s title, abstract, and contribution list attribute the gains primarily to the tracking-aware objective representation (status, de-duplication keys, acceptance predicates, objective cards). RQ2 (Section V-C) only ablates removal of the structural stage and shows that hardening alone collapses (e.g., ~26.9% line / 23.45% mutation on Qwen). It does not hold fixed static-analysis content (Table I), two-stage budget allocation, iteration count, and prompt structure while replacing tracked objectives with unstructured injection of the same signals. Without that isolation, Table III and Section V-B leave open that richer static projection, staged focus, or longer structured prompts—not the tracking machinery itself—drive most of the lift. Please add a matched ablation (tracked cards vs. unstructured feedback of the same evidence) or reframe the central claim around the full system rat","section":null},{"comment":"Section V-B states that structural-stage gains over PANTA come from objective tracking, yet the structural stage uses five iterations while PANTA is run for ten (Section IV-D), and TATG additionally injects fine-grained static objectives (StateSetup, DispatchArm, ExceptionPath, EntryRoute, etc.) that PANTA does not receive in the same form. The iteration asymmetry is noted but not controlled; the static-content difference is not ablated. These confounds weaken the stage-level causal claim even if end-to-end superiority remains credible. A same-iteration, same-static-content comparison (or an explicit decomposition of contribution sources) is needed for the load-bearing attribution in RQ1.","section":null},{"comment":"Table III reports two hardening variants (+MUTGEN and +Default). Final headline numbers use +Default, which is TATG’s own mutation-objective path, while MUTGEN is also a baseline for the hardening stage. The text (RQ1) claims higher mutation score than MUTGEN by ~2.9 pp, but the table also shows that hardening further improves line/branch coverage substantially. Please clarify precisely what “Default” hardening implements relative to MUTGEN (same mutant set, same kill criterion, same prompt budget?), and report MUTGEN’s standalone end-to-end numbers on the full 141-method suite under the same total iteration budget so the hardening comparison is fully external rather than partly internal.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: TATG is a competent systems paper that turns static analysis plus coverage/mutation feedback into a single tracked objective schema and a coverage-then-hardening loop, and on 141 hard Java methods it beats KTester and PANTA by large margins on line, branch, and mutation score.\n\nWhat is actually new is not “use feedback” or “use mutation”—those already exist—but the explicit objective tuple (evidence, setup, action, oracle shape, acceptance predicate, status, de-duplication key) rendered as objective cards, plus the deliberate two-stage handoff so mutation feedback only runs once reachability is real. The motivating token-stream example makes the problem concrete. The evaluation is better than average for this area: two open models, traditional/hybrid/LLM baselines, Wilcoxon tests, stage-level numbers, a structural-stage ablation that collapses when you skip coverage first, and a small industrial-tool comparison on a 50-method subset. Metrics are external (JaCoCo, PITest), not self-defined. That package is real work.\n\nThe soft spot the stress note flags is fair and load-bearing for the branding, not for the existence of the gains. RQ2 only removes the structural stage; it does not hold static content, stage budget, and prompt structure fixed while replacing tracked status/keys with unstructured injection of the same signals. So “tracking-aware” is not cleanly isolated from richer static projection and staged prompting. Extra subjects are purposively sampled; code is promised but not yet out. Those are real caveats, not fatal ones. The end-to-end advance still stands as stated against the named baselines.\n\nThis is for people building multi-round LLM test agents or industrial unit-test tooling, not for theory. I would bring it to reading group, cite the empirical comparison if I work in this lane, and send it to peer review. Ask for a tracking-vs-unstructured ablation and artifacts; do not desk-reject.","headline":"Solid end-to-end LLM test-gen system with real gains; the tracking story is under-isolated but the empirical package still deserves referees.","tokens_in":18215,"tokens_out":505,"would_cite":true,"duration_ms":4977,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Tracking each testing goal as a live objective lets LLM unit-test generators cover hard branches and kill more mutants.","keywords":["unit test generation","large language models","objective tracking","coverage","mutation testing","Java","program analysis"],"falsifier":"Run a matched control that keeps the same two-stage schedule, static analysis, and feedback signals but feeds them only as unstructured prompt text without per-objective status, de-duplication keys, or acceptance predicates; if coverage and mutation score then match TATG, the tracking claim fails.","tokens_in":18176,"feed_emoji":"🧪","tokens_out":565,"duration_ms":5205,"temperature":0.7,"pith_summary":"Complex Java methods are hard to test automatically because the real work is not the method signature; it is the branch-specific setups, exception paths, and strong assertions that only become visible once you try. Prior LLM tools inject project context, coverage, or mutation feedback into prompts, but they do not keep a lasting list of which individual requirements are done and which are still open. TATG turns every such requirement into a tracked objective with evidence, required setup, target action, oracle shape, and a measurable acceptance check. A structural stage first raises reachability and coverage; a hardening stage then uses mutation feedback to strengthen assertions. On 141 complex methods the complete system raises line coverage, branch coverage, and mutation score by large margins over strong baselines and stays competitive with an industrial proprietary tool.","feed_headline":"Tracked testing goals lift LLM unit-test scores by 20+ points","feed_subtitle":"A two-stage Java generator keeps each branch and mutant open until it is satisfied.","key_machinery":"The tracking-aware objective: a typed tuple that records identifier, source, category, evidence, required setup, target action, oracle shape, acceptance predicate, lifecycle status, and de-duplication key, so each testing requirement can be opened, attempted, satisfied, or retired across iterations.","core_discovery":"The paper claims that representing static facts and dynamic feedback as a single, status-tracked objective state, then guiding generation in two stages (coverage first, mutation hardening second), produces substantially higher line coverage, branch coverage, and mutation score than existing LLM, hybrid, and traditional generators on complex Java methods.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Tracking test goals lifts LLM Java coverage over 20 points","TATG tracks branches and mutants to raise unit-test scores","Two-stage objective tracking improves LLM test coverages","Status-tracked requirements boost branch and mutation scores","Fine-grained goal tracking strengthens LLM unit-test generation"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The measured gains are caused by the explicit tracking of objectives themselves, not merely by richer static analysis, the two-stage budget split, or longer structured prompts that inject the same signals without status tracking.","fun_headline_variants_meta":{"raw":{"variants":["Tracking test goals lifts LLM Java coverage over 20 points","TATG tracks branches and mutants to raise unit-test scores","Two-stage objective tracking improves LLM test coverages","Status-tracked requirements boost branch and mutation scores","Fine-grained goal tracking strengthens LLM unit-test generation"]},"model":"grok-4.5","effort":"low","cost_usd":0.002962,"raw_usage":{"total_tokens":1089,"prompt_tokens":797,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":29620000,"prompt_tokens_details":{"text_tokens":797,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":230,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":797,"tokens_out":62,"duration_ms":2501,"temperature":1.0,"reasoning_tokens":230,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T04:13:10.558193+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run a matched control that keeps the same two-stage schedule, static analysis, and feedback signals but feeds them only as unstructured prompt text without per-objective status, de-duplication keys, or acceptance predicates; if coverage and mutation score then match TATG, the tracking claim fails.","supporting_citations":[],"review_version":1}