{"id":"3a542c80-9417-4681-9157-2f8016c6fc5d","arxiv_id":"2509.24148","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An agentic framework that selects diverse test cases, retrieves repository context, and reflects on test failures improves repository-level code generation Pass@1 by 9.49 and 2.17 percentage points over prior baselines on RepoCod and RepoEval.","lead":"TENET is an AI coding agent that picks a small set of existing test cases, searches the repository, and iteratively fixes code until the tests pass, achieving higher Pass@1 on two repository-level code-generation benchmarks. It matters because it shows test cases can serve not just for validation but as executable specifications that guide LLM agents in real-world codebases.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"THM's caller-cluster selection signal is unvalidated; degenerate tracebacks could undermine the main ablation result.","rationale":"The reader's weakest assumption identifies the THM selection signal as not directly validated. I agree this is the most load-bearing concern because THM is the paper's core novelty and its ablation produces the largest drop. The paper's strengths—public benchmarks, reproducible link, complete appendix, honest failure case, and systematic RQ4 comparison—support the overall framework, but the causal role of caller diversity remains under-supported. The concern does not invalidate the paper; it motivates a concrete diagnostic that could be added to the appendix. Since the reader's verdict is already CONDITIONAL and this concern falls under that condition, no verdict change is needed.","tokens_in":64,"tokens_out":8548,"duration_ms":160607,"concrete_test":"Instrument the THM pipeline on all 980 RepoCod tasks: record for each task (i) the number of failing tests whose traceback contains the target frame, (ii) the number of distinct immediate-caller clusters, and (iii) the cluster-size distribution. Then re-run the RQ4 comparison (THM vs. IPS, Table 4) restricted to tasks with at least 2 clusters and at least T target-reaching tests. If THM no longer outperforms IPS on this subset, the caller-diversity signal is not the causal mechanism; if the advantage persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"THM (Sec. 3.1) assumes that executing the full test suite against a target body replaced with `raise NotImplementedError`, then clustering failing tests by the immediate caller in the traceback, yields a stable and representative partition of usage scenarios. This requires that (a) every failing test actually reaches the target, (b) the immediate-caller partition is meaningful, and (c) most tasks have non-degenerate clusters. The paper never reports how many tasks have zero failing tests, one dominant cluster, or failing tests whose tracebacks never contain the target (e.g., import/collection errors). If many tasks lack the target frame in their tracebacks, those tests are either silently dropped or clustered by an unrelated caller, corrupting the diversity signal. The ablation in Table 2 attributes a 17.24pp drop to removing THM, but this could partly stem from full-suite context overflow rather than from caller diversity. Similarly, the RQ4 comparison (THM vs. IPS, Table 4) could be biased if THM's advantage comes from filtering out non-reaching tests rather than from the clustering itself. Because the THM is the paper's primary novelty and its removal causes the largest ablation drop, this unvalidated selection signal is load-bearing for the central causal claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TENET, an LLM-agent framework for repository-level code generation under the test-driven development (TDD) paradigm. TENET has three components: a test harness mechanism (THM) that dynamically selects a small, caller-diverse subset of the target function's tests by executing the suite against an unimplemented target and clustering failing tracebacks by immediate caller; a tailored toolset adding four APIs for import inspection, similar-method retrieval, usage-example retrieval, and interactive debugging; and a reflection-based refinement workflow (RRW) that iteratively localizes faults, reviews context, gathers additional evidence, and refines code. On RepoCod and RepoEval, TENET reports Pass@1 of 69.08% and 81.77% with Claude Sonnet 4, outperforming RepoCoder, SpecRover, SWE-Agent, and OpenHands. Ablations on RepoCod with DeepSeek-V3 attribute gains to all three components, and RQ3-RQ5 study test-suite size, selection strategies, and the stage at which tests are injected into the workflow.","tokens_in":29591,"tokens_out":7733,"duration_ms":58131,"significance":"The topic is timely and the work is useful if the central claims hold. This is the first systematic repository-level study of how test-suite characteristics affect LLM code-generation agents under TDD, and it reports token costs alongside accuracy, which is valuable for practical deployment. The paper also ships detailed prompts, per-project tables, and a candid failure case in Appendix A.2.4, showing that the agent can hallucinate and persist in unproductive refinements when context is weak. These strengths make the contribution potentially significant for both the agent-coding and test-driven-development communities. The main risks are methodological: the THM selection signal is not directly validated, the principal ablation confounds selection with test-suite size, and the headline comparisons lack statistical error analysis. These are fixable in revision.","major_comments":[{"comment":"The THM selection signal is asserted but not validated. The paper never reports how many REPOCOD tasks have zero failing tests after replacing the target body with raise NotImplementedError, how many failing tracebacks do not contain the target frame (e.g., import or collection errors), or how many tasks yield a single dominant caller cluster. If the target frame is often absent, or if most tasks have one cluster, the \"caller diversity\" objective is vacuous and THM degenerates to shortest-chain selection. This is load-bearing because Table 2 attributes the largest ablation drop (17.24pp) to removing THM, and Section 5.4 bases its headline comparison on THM vs. IPS. I ask for a diagnostic table reporting cluster counts, non-reaching-test rates, and zero-failure-task rates, and an additional matched-size comparison (THM vs. IPS at T=3) on the same model and benchmark to show that clusterin","section":"Section 3.1, Table 2"},{"comment":"The TENET-THM ablation is not matched in test quantity. Removing THM substitutes the full target test suite (68 tests on average per task, as stated in Section 5.2) for the three selected tests. The observed 17.24pp Pass@1 drop and 40.69% input-token increase may therefore reflect long-context overflow or noise rather than the caller-diversity property of THM. The paper's own explanation in Section 5.2 emphasizes \"redundancy and noise\" from the full suite, which supports this alternative reading. Please add an ablation that keeps the number of tests fixed (e.g., THM-selected vs. randomly selected vs. IPS-selected, all at T=3) using the same model and evaluation setup as Table 2, and present that as the direct evidence for the THM mechanism.","section":"Table 2, Section 5.2"},{"comment":"The Pass@1 computation is ambiguous. The text says the RRW validates code against the \"selected test cases,\" and RQ1 states that all baselines receive \"three randomly selected test cases,\" but it does not state whether final Pass@1 is computed against the full target test suite or against the selected subset. If correctness is evaluated only on the selected tests, then the method is circular: the agent is scored on tests that THM chose and that the agent saw. If, as is standard for RepoCod and RepoEval, the official full test suite is used for final evaluation, please state this explicitly and describe the evaluation command. This clarification is necessary to interpret all comparisons, including Table 1, Table 2, and Table 4.","section":"Section 4, evaluation protocol"},{"comment":"No error bars, confidence intervals, or significance tests are reported. The headline RepoEval gain over the strongest baseline is 2.17pp (Table 1), which may be within sampling noise on 373 tasks; the THM-vs-IPS gap of 7.65pp in Table 4 could also vary across runs due to random selection and LLM nondeterminism even at temperature 0. Please report multiple independent runs with means and standard deviations, or bootstrap confidence intervals / a paired significance test, at least for the main comparisons. This is especially important because the paper draws strong component-wise causal conclusions from single-run ablation percentages.","section":"Tables 1, 2, 4, and 5"}],"minor_comments":[{"comment":"API names are inconsistent between the main text and the appendix. Main text and Figure 2 use search_import_in_file, search_similar_method, search_target_usage, and run_debugger_cmd; Appendix A.1.1 and case studies use search_import_statement, search_relevant_method, search_target_usage_example, and run_pdb_cmd. Please unify the names and tool descriptions.","section":"Section 3.2, Figure 2, Appendix A.1.1"},{"comment":"The failure case (Fig. 9) candidly shows that RRW can overfit to spurious cues, hallucinate dependencies, and persist in unproductive refinement loops. This is a real limitation and should be acknowledged in the main text, not only in the appendix, with some indication of its frequency if available.","section":"Appendix A.2.4"},{"comment":"There are numerous typographical and spacing errors, e.g., \"T ENETfeatures,\" \"mechnism,\" \"acutally,\" \"conatining,\" \"comsumptions,\" and \"fauilure.\" A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The random selection of three test cases for the baselines is not specified: no seed, no sampling protocol, and no indication of variance across draws. This matters because the comparison in Table 1 could depend on the particular random tests drawn. Please provide the protocol or run multiple random draws.","section":"Section 4, RQ1"},{"comment":"The definition of \"average test coverage\" in Table 4 is not given. State the coverage metric (line, branch, function?) and how it is computed, since it is used as supporting evidence for the selection-strategy comparison.","section":"Figure 11, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's core comparison is strong and the artifact appears complete, but the THM validation gap is the main risk: the largest ablation attribute and the key selection-strategy comparison rest on an unvalidated traceback-clustering signal. The clearest path to acceptance is to add the diagnostic distribution of THM clusters, replace or complement the full-suite ablation with a matched-size ablation, and clarify the final Pass@1 evaluation protocol. The statistical-rigor request is secondary but should be addressed. I would be willing to accept a revised version that supplies these items."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful empirical paper on using tests to guide repository-level code generation. The new piece is the test harness mechanism (THM) that selects a small, caller-diverse subset of tests via dynamic analysis, plus the four new agent APIs and the reflection loop. The systematic study of test quantity, selection strategy, and usage stage is a real contribution; the finding that three to five tests beat the full suite is concrete and actionable. The ablations are honest and the appendix documents a hallucination failure mode, which is more than most papers do.\n\nThe soft spots are real but not disqualifying. No error bars or significance tests anywhere; the 2.17pp gain on RepoEval is within plausible noise, and the 9.49pp gain on RepoCod is the more convincing number. T=3 is tuned on sphinx (33 tasks) and then reported as the default on the full benchmark; other budget parameters are fixed without sensitivity analysis. All ablations use DeepSeek-V3 on RepoCod only, so the component contributions may not transfer to Claude. The biggest gap is the THM itself: the paper never reports how many tasks have zero failing tests, degenerate clusters, or tracebacks that never reach the target. If a lot of tasks rely on one caller cluster, the ablation drop attributed to caller diversity could be mostly a context-length effect. The RQ4 comparison (THM vs. IPS) does control for using the same dynamic traces, which mitigates that worry, but doesn't settle it.\n\nI also note the appendix's failure case is genuinely honest: the agent hallucinates dependencies and persists in unproductive refinements. That's a plausible limitation, not a fatal one.\n\nBottom line: this deserves a serious referee. The central claim of component-wise contributions is supported by the ablations, the empirical map of test-suite factors is new, and the results are reproducible in principle (code link, temperature 0, full prompts). A revision should add variance or multiple runs, a THM sanity check (distribution of clusters and reaching tests), and ideally an ablation on the stronger backbone. I'd take this to a reading group and I'd cite the test-quantity and selection findings.","headline":"Useful empirical paper on test-driven repo-level code generation; the THM selection signal is under-validated and there are no error bars, but the test-quantity findings and ablations warrant a serious referee.","tokens_in":30129,"tokens_out":3851,"would_cite":true,"duration_ms":30686,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TENET turns repository-level code generation into a test-driven process, showing that a few diverse test cases can outperform full suites and significantly lift Pass@1 on two benchmarks.","keywords":["Test-Driven Development","LLM agents","repository-level code generation","test selection","dynamic analysis","reflection-based refinement","code retrieval","debugging tools"],"falsifier":"Run TENET on tasks chosen so that replacing the target with a not-implemented error causes collection or import failures before any target logic is exercised; if the caller-clustered selection underperforms random selection on those tasks, the core selection signal is unreliable.","tokens_in":29150,"feed_emoji":"🧪","tokens_out":5154,"duration_ms":49001,"temperature":0.7,"pith_summary":"This paper argues that tests can be more than a final acceptance check: they can act as executable specifications that guide an LLM agent to implement functions inside real repositories. The proposed TENET framework selects a small diverse subset of tests by running the suite against an unimplemented target and clustering failures by their calling context, gives the agent structured retrieval and debugging tools, and then loops through a reflection-based refinement workflow until the chosen tests pass. On two repository-level benchmarks, TENET reaches 69.08% and 81.77% Pass@1 (the fraction of tasks solved on the first generated candidate), improving over the strongest baselines by 9.49 and 2.17 percentage points. The same experiments show that three to five well-chosen tests outperform full suites and that using tests at every stage beats using them only for validation.","feed_headline":"Test-first AI agent lifts repo-code pass rate to 69 percent","feed_subtitle":"Test selection, not test volume, drives the gains: 3-5 diverse tests beat full suites.","key_machinery":"The load-bearing mechanism is the test harness mechanism (THM): dynamic analysis that replaces the target function body with a not-implemented error, runs the repository's test suite, captures the failing tests, and clusters them by the caller function that directly invokes the target in the traceback. From these clusters, TENET selects at most three tests, one per cluster when possible, preferring shortest call chains. This balances diversity (different callers imply different usage patterns and code paths) against clarity (short chains produce focused signals). The other two components—a toolset adding semantic similarity search, usage-example lookup, import inspection, and a debugger comm","core_discovery":"TENET's central discovery is that a handful of test cases, selected for diversity of usage rather than volume, can supply the semantic information that docstrings and code context omit. The test harness mechanism executes the full suite against a target whose body is replaced with a not-implemented error, then groups failing tests by the immediate caller in the traceback; selecting one test per caller cluster with shortest call chains gives the LLM complementary views of the target's behaviour, including edge cases like singleton probability vectors. Paired with a toolset that supports semantic search, usage-example lookup, import analysis, and interactive debugging, plus a reflection loop t","pith_inferences":["The THM selection signal presupposes that the synthetic failure (NotImplementedError) propagates to tests in a way that reflects real usage; on repositories where import or collection errors dominate, clusters could be degenerate, so a guard or fallback to random selection may be needed.","Because THM depends on existing tests, its benefit is bounded by test-suite quality; coupling it with automatic test generation would be a natural extension to handle functions with sparse or missing suites.","The paper's T=3 tuning is global; the cluster structure per task is diagnostic, so an adaptive budget (e.g., more tests when clusters are numerous) might squeeze further gains.","A testable cross-check: if the caller-diversity heuristic is what matters, then suites where every test reaches the target through the same caller should show little benefit from larger T, a prediction the current data could verify."],"forward_implications":["If the central claim holds, test selection is a first-order performance lever: three to five caller-diverse tests beat both random triples and the full suite.","Distinct callers in the traceback act as a cheap, execution-based proxy for behavioural coverage, so maintainers can prune large suites without waiting for generation.","Reflection with debugger access rescues a substantial share of initially failing tasks—roughly 39% of solved tasks in the reported runs—so refinement is not a marginal add-on.","Injecting tests during retrieval and refinement, not only as a final check, improves correctness at an increased token cost; the trade-off is explicit and tunable.","Structured retrieval APIs keep trajectories shorter and token use lower than terminal-command agents, making the accuracy gain affordable."],"fun_headline_variants":["Diverse tests, not many, boost AI repo codegen","Pick 3–5 diverse tests to supercharge AI codegen","TENET: test-driven agent lifts repo code pass@1","Test diversity beats suite size for AI codegen","Agentic TDD: select tests by calls, not count"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework leans on the assumption that grouping failing tests by the immediate caller in the traceback yields stable, representative clusters of the target's usage diversity; if tests fail for reasons unrelated to the target, the selection can mislead the agent.","fun_headline_variants_meta":{"raw":{"variants":["Diverse tests, not many, boost AI repo codegen","Pick 3–5 diverse tests to supercharge AI codegen","TENET: test-driven agent lifts repo code pass@1","Test diversity beats suite size for AI codegen","Agentic TDD: select tests by calls, not count"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1109,"prompt_tokens":775,"completion_tokens":334,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":249}},"tokens_in":519,"tokens_out":334,"duration_ms":13952,"temperature":1.0,"reasoning_tokens":249,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T14:37:12.630830+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TENET on tasks chosen so that replacing the target with a not-implemented error causes collection or import failures before any target logic is exercised; if the caller-clustered selection underperforms random selection on those tasks, the core selection signal is unreliable.","supporting_citations":[],"review_version":1}