Pith. sign in

REVIEW 4 major objections 5 minor 42 references

TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TENET turns repository-level code generation into a test-driven process, showing that a few diverse test cases can outperform full suites and significantly lift Pass@1 on two benchmarks.

desk verdict Useful empirical paper on test-driven repo-level code generation; the THM selection signal is under-validated and there are no error bars, but the test-quantity findings and ablations warrant a serious referee. read the letter →

arxiv 2509.24148 v3 pith:26HWDJYC submitted 2025-09-29 cs.SE cs.AI

Yiran Hu , Nan Jiang , Shanchao Liang , Yi Wu , Lin Tan This is my paper
classification cs.SEcs.AI
keywords Test-DrivenDevelopmentLLMagentsrepository-levelcodegenerationtestselectiondynamicanalysisreflection-basedrefinementretrievaldebuggingtools
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that tests can be more than a final acceptance check: they can act as executable specifications that guide an LLM agent to implement functions inside real repositories. The proposed TENET framework selects a small diverse subset of tests by running the suite against an unimplemented target and clustering failures by their calling context, gives the agent structured retrieval and debugging tools, and then loops through a reflection-based refinement workflow until the chosen tests pass. On two repository-level benchmarks, TENET reaches 69.08% and 81.77% Pass@1 (the fraction of tasks solved on the first generated candidate), improving over the strongest baselines by 9.49 and 2.17 percentage points. The same experiments show that three to five well-chosen tests outperform full suites and that using tests at every stage beats using them only for validation.

What carries the argument

The load-bearing mechanism is the test harness mechanism (THM): dynamic analysis that replaces the target function body with a not-implemented error, runs the repository's test suite, captures the failing tests, and clusters them by the caller function that directly invokes the target in the traceback. From these clusters, TENET selects at most three tests, one per cluster when possible, preferring shortest call chains. This balances diversity (different callers imply different usage patterns and code paths) against clarity (short chains produce focused signals). The other two components—a toolset adding semantic similarity search, usage-example lookup, import inspection, and a debugger comm

What would settle it

Run TENET on tasks chosen so that replacing the target with a not-implemented error causes collection or import failures before any target logic is exercised; if the caller-clustered selection underperforms random selection on those tasks, the core selection signal is unreliable.

Watch

Extended reading notes

Core claim

TENET's central discovery is that a handful of test cases, selected for diversity of usage rather than volume, can supply the semantic information that docstrings and code context omit. The test harness mechanism executes the full suite against a target whose body is replaced with a not-implemented error, then groups failing tests by the immediate caller in the traceback; selecting one test per caller cluster with shortest call chains gives the LLM complementary views of the target's behaviour, including edge cases like singleton probability vectors. Paired with a toolset that supports semantic search, usage-example lookup, import analysis, and interactive debugging, plus a reflection loop t

Load-bearing premise

The framework leans on the assumption that grouping failing tests by the immediate caller in the traceback yields stable, representative clusters of the target's usage diversity; if tests fail for reasons unrelated to the target, the selection can mislead the agent.

Editorial extensions

If this is right

  • If the central claim holds, test selection is a first-order performance lever: three to five caller-diverse tests beat both random triples and the full suite.
  • Distinct callers in the traceback act as a cheap, execution-based proxy for behavioural coverage, so maintainers can prune large suites without waiting for generation.
  • Reflection with debugger access rescues a substantial share of initially failing tasks—roughly 39% of solved tasks in the reported runs—so refinement is not a marginal add-on.
  • Injecting tests during retrieval and refinement, not only as a final check, improves correctness at an increased token cost; the trade-off is explicit and tunable.
  • Structured retrieval APIs keep trajectories shorter and token use lower than terminal-command agents, making the accuracy gain affordable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The THM selection signal presupposes that the synthetic failure (NotImplementedError) propagates to tests in a way that reflects real usage; on repositories where import or collection errors dominate, clusters could be degenerate, so a guard or fallback to random selection may be needed.
  • Because THM depends on existing tests, its benefit is bounded by test-suite quality; coupling it with automatic test generation would be a natural extension to handle functions with sparse or missing suites.
  • The paper's T=3 tuning is global; the cluster structure per task is diagnostic, so an adaptive budget (e.g., more tests when clusters are numerous) might squeeze further gains.
  • A testable cross-check: if the caller-diversity heuristic is what matters, then suites where every test reaches the target through the same caller should show little benefit from larger T, a prediction the current data could verify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TENET, an LLM-agent framework for repository-level code generation under the test-driven development (TDD) paradigm. TENET has three components: a test harness mechanism (THM) that dynamically selects a small, caller-diverse subset of the target function's tests by executing the suite against an unimplemented target and clustering failing tracebacks by immediate caller; a tailored toolset adding four APIs for import inspection, similar-method retrieval, usage-example retrieval, and interactive debugging; and a reflection-based refinement workflow (RRW) that iteratively localizes faults, reviews context, gathers additional evidence, and refines code. On RepoCod and RepoEval, TENET reports Pass@1 of 69.08% and 81.77% with Claude Sonnet 4, outperforming RepoCoder, SpecRover, SWE-Agent, and OpenHands. Ablations on RepoCod with DeepSeek-V3 attribute gains to all three components, and RQ3-RQ5 study test-suite size, selection strategies, and the stage at which tests are injected into the workflow.

Significance. The topic is timely and the work is useful if the central claims hold. This is the first systematic repository-level study of how test-suite characteristics affect LLM code-generation agents under TDD, and it reports token costs alongside accuracy, which is valuable for practical deployment. The paper also ships detailed prompts, per-project tables, and a candid failure case in Appendix A.2.4, showing that the agent can hallucinate and persist in unproductive refinements when context is weak. These strengths make the contribution potentially significant for both the agent-coding and test-driven-development communities. The main risks are methodological: the THM selection signal is not directly validated, the principal ablation confounds selection with test-suite size, and the headline comparisons lack statistical error analysis. These are fixable in revision.

major comments (4)
  1. [Section 3.1, Table 2] The THM selection signal is asserted but not validated. The paper never reports how many REPOCOD tasks have zero failing tests after replacing the target body with raise NotImplementedError, how many failing tracebacks do not contain the target frame (e.g., import or collection errors), or how many tasks yield a single dominant caller cluster. If the target frame is often absent, or if most tasks have one cluster, the "caller diversity" objective is vacuous and THM degenerates to shortest-chain selection. This is load-bearing because Table 2 attributes the largest ablation drop (17.24pp) to removing THM, and Section 5.4 bases its headline comparison on THM vs. IPS. I ask for a diagnostic table reporting cluster counts, non-reaching-test rates, and zero-failure-task rates, and an additional matched-size comparison (THM vs. IPS at T=3) on the same model and benchmark to show that clusterin
  2. [Table 2, Section 5.2] The TENET-THM ablation is not matched in test quantity. Removing THM substitutes the full target test suite (68 tests on average per task, as stated in Section 5.2) for the three selected tests. The observed 17.24pp Pass@1 drop and 40.69% input-token increase may therefore reflect long-context overflow or noise rather than the caller-diversity property of THM. The paper's own explanation in Section 5.2 emphasizes "redundancy and noise" from the full suite, which supports this alternative reading. Please add an ablation that keeps the number of tests fixed (e.g., THM-selected vs. randomly selected vs. IPS-selected, all at T=3) using the same model and evaluation setup as Table 2, and present that as the direct evidence for the THM mechanism.
  3. [Section 4, evaluation protocol] The Pass@1 computation is ambiguous. The text says the RRW validates code against the "selected test cases," and RQ1 states that all baselines receive "three randomly selected test cases," but it does not state whether final Pass@1 is computed against the full target test suite or against the selected subset. If correctness is evaluated only on the selected tests, then the method is circular: the agent is scored on tests that THM chose and that the agent saw. If, as is standard for RepoCod and RepoEval, the official full test suite is used for final evaluation, please state this explicitly and describe the evaluation command. This clarification is necessary to interpret all comparisons, including Table 1, Table 2, and Table 4.
  4. [Tables 1, 2, 4, and 5] No error bars, confidence intervals, or significance tests are reported. The headline RepoEval gain over the strongest baseline is 2.17pp (Table 1), which may be within sampling noise on 373 tasks; the THM-vs-IPS gap of 7.65pp in Table 4 could also vary across runs due to random selection and LLM nondeterminism even at temperature 0. Please report multiple independent runs with means and standard deviations, or bootstrap confidence intervals / a paired significance test, at least for the main comparisons. This is especially important because the paper draws strong component-wise causal conclusions from single-run ablation percentages.
minor comments (5)
  1. [Section 3.2, Figure 2, Appendix A.1.1] API names are inconsistent between the main text and the appendix. Main text and Figure 2 use search_import_in_file, search_similar_method, search_target_usage, and run_debugger_cmd; Appendix A.1.1 and case studies use search_import_statement, search_relevant_method, search_target_usage_example, and run_pdb_cmd. Please unify the names and tool descriptions.
  2. [Appendix A.2.4] The failure case (Fig. 9) candidly shows that RRW can overfit to spurious cues, hallucinate dependencies, and persist in unproductive refinement loops. This is a real limitation and should be acknowledged in the main text, not only in the appendix, with some indication of its frequency if available.
  3. [Throughout] There are numerous typographical and spacing errors, e.g., "T ENETfeatures," "mechnism," "acutally," "conatining," "comsumptions," and "fauilure." A careful proofreading pass is needed.
  4. [Section 4, RQ1] The random selection of three test cases for the baselines is not specified: no seed, no sampling protocol, and no indication of variance across draws. This matters because the comparison in Table 1 could depend on the particular random tests drawn. Please provide the protocol or run multiple random draws.
  5. [Figure 11, Table 4] The definition of "average test coverage" in Table 4 is not given. State the coverage metric (line, branch, function?) and how it is computed, since it is used as supporting evidence for the selection-strategy comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: benchmark evaluation is independent of THM's test-selection input.

full rationale

Pass@1 in Tables 1-5 is the standard benchmark correctness metric over the full per-task test suites of REPOCOD and RepoEval; a task counts as solved only if the generated code passes the entire suite, not merely the 2-3 tests THM selects. Evidence for this is structural: the 'All' row of Table 3 and the TENET-THM ablation (full test suite as input) are scored on the same full suite as the T=3 conditions, otherwise the rows would be measuring different eval sets and the paper's interpretation (context overload, not test-set size) would be incoherent. THM's dynamic-analysis selection therefore constructs an input prompt; the subsequent pass/fail decision is independent of the selection mechanism. The IPS comparison (A.3.4) uses the same stub-execution signal and isolates the caller-clustering contribution, so THM's advantage is an empirical result rather than a definitional one. The only self-citation of note is the REPOCOD benchmark (Liang et al. 2024b), which shares authors; however, REPOCOD is a fixed, public, externally reproducible benchmark that predates TENET, so the citation supplies independent evidence. The T=3 choice on the 33-task sphinx subset is a transparent hyperparameter selection; the RQ3 finding on the full benchmark is a distinct empirical measurement, with no fitted quantity renamed as a prediction. The failure-case study (A.2.4) and the conclusion's 'limitation of THM's reliance on existing tests' are acknowledged validity caveats, not circular reductions. No equation or metric in the paper is defined in terms of the result it is used to support.

Assumptions & free parameters 4 free parameters · 3 assumptions · 3 invented entities

TENET borrows the agent backbone (SpecRover's AST toolset), benchmark (RepoCod/RepoEval), and backbone LLM (Claude Sonnet 4, DeepSeek-V3) from prior work, then adds four tunable budget parameters (T, retrieval rounds, refinement attempts, API rounds) and three system components. The empirical claims rest on the benchmarks and the hand-configurable budgets, not on a mathematical derivation.

free parameters (4)
  • T (number of selected test cases) = 3
    Set based on preliminary experiments on the sphinx project from RepoCod, then validated across the full RepoCod benchmark (Section 3.1, 5.3).
  • Max retrieval rounds before initial generation = 15
    Default configuration for TENET (Section 4). No sensitivity analysis reported.
  • Max code refinement attempts in RRW = 5
    Default configuration for TENET (Section 4). No sensitivity analysis reported.
  • Max API call rounds in RRW = 15
    Default configuration for TENET (Section 4). No sensitivity analysis reported.
assumptions (3)
  • domain assumption Tests selected by dynamic analysis of failing executions against an unimplemented target are representative of the target's behavior.
    Central premise of THM, stated in Section 3.1: the selected subset guides generation. The paper does not report cases where the failure trace is misleading, and the appendix failure case shows the signal can be weak.
  • domain assumption The RepoCod and RepoEval benchmark tasks and their ground-truth tests are correct and well-formed.
    All evaluation depends on the assumption that these public benchmarks are reliable measures of repository-level code generation (Section 4).
  • domain assumption RepoCoder, SpecRover, SWE-Agent, and OpenHands are strong and fairly tuned baselines.
    The comparison in Table 1 depends on this. The paper does not report hyperparameter searches for baselines beyond following original configurations.
invented entities (3)
  • Test Harness Mechanism (THM) independent evidence
    purpose: Selects a small, diverse subset of tests to guide generation.
    It is a system component, not a natural phenomenon. Its effects are evidenced by the ablation in Table 2 and the success of the overall system, though the ablations are on a single backbone.
  • Reflection-based Refinement Workflow (RRW) independent evidence
    purpose: Iteratively analyzes failures and refines code.
    System component; evidenced by ablation (Table 2) and the finding that RRW recovers 186/482 tasks (38.59%) that fail on the first attempt.
  • Four new agent APIs (search_import_in_file, search_similar_method, search_target_usage, run_debugger_cmd) independent evidence
    purpose: Provide efficient retrieval and interactive debugging.
    System components; evidenced by usage-frequency analysis (Figure 3) and the ablation (Table 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation." pith.science (2026). https://pith.science/paper/26HWDJYC

@misc{pith2026250924148,
  author       = {Pith},
  title        = {Pith review of: TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26HWDJYC}},
  note         = {Machine review of arXiv:2509.24148}
}
read the original abstract

Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation. With recent advances in Large Language Models (LLMs), developers can shift from manually writing the code to defining tests as executable specifications and delegating code synthesis to AI agents. However, enabling repository-level TDD under developer-written tests is challenging, requiring: (1) specification enhancement: identifying a concise yet representative test subset from large suites with rich task semantics; (2) retrieval augmentation: using tests to guide reasoning and context retrieval; and (3) test-driven refinement: interpreting test feedback for iterative improvement. We propose TENET, an agentic framework for repository-level code generation under the TDD paradigm. TENET includes: (1) a test harness mechanism that selects a concise test suite to maximize diversity of the target usage scenarios; (2) a tailored agent toolset for efficient retrieval and debugging; and (3) a reflection-based refinement workflow that iteratively analyzes failures and updates implementations. TENET consistently outperforms the strongest baselines across backbones, achieving 69.08% and 81.77% Pass@1 on RepoCod and RepoEval with Claude Sonnet 4, improving by 9.49 and 2.17 percentage points, respectively. Additionally, we present the first systematic study of how test suite characteristics influence LLM agent performance in TDD settings.

Figures

Figures reproduced from arXiv: 2509.24148 by the authors.

Figure 1
Figure 1. Examples of repository-level code generation under standard and test-driven setups. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. TENET workflow. Non-agentic approaches operate without self-planning, but using techniques like Retrieval￾Augmented Generation (RAG) and feedback-driven refinement. Some methods use static or learned retrieval strategies (Zhang et al., 2023; Wang et al., 2025; Wu et al., 2024; Shrivastava et al., 2023; Liao et al., 2024), while others base on semantic or structural graphs (Cheng et al., 2024; Liu et al., 2024b; Li e… view at source ↗
Figure 4
Figure 4. Number of solved tasks per round (TENET, DeepSeek-V3). Test Num. sphinx seaborn flask xarray sympy more-itertools datasets scikit-learn astropy pylint plotly.py Total 1 39.39 47.44 67.44 25.30 26.80 56.98 47.46 22.61 43.52 23.08 43.42 35.71 3 54.55 55.13 72.09 42.17 34.02 70.93 62.71 46.18 47.06 30.77 40.79 49.18 5 42.42 55.13 79.07 37.35 28.87 80.23 59.32 43.95 50.59 34.62 42.11 48.57 10 36.36 57.69 74.42 34.94 26.… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Overlap of solved tests across four settings. 5.3 RQ3: THE IMPACT OF TEST SUITE SIZE As a tunable parameter, we set T = 3 based on preliminary results on the sphinx project (33 tasks) in REPOCOD, where T = 3 achieved the best performance. To address RQ3, [PITH_FULL_IM…
Figure 6
Figure 6. Figure 6: A case study on task seaborn 34 from REPOCOD. This example explains how the test cases selected by the THM guides the TENET toward correct code generation. A.2 CASE STUDIES In this part, we provide case studies about how the test harness mechanism (THM), tailored agent…
Figure 7
Figure 7. Figure 7: A case study on task scikit 47 from REPOCOD. This example explains how the tailored agent toolset contributes to the efficiency and effectiveness of the TENET. The error persists because the agent repeatedly assumes that the target function fit should use the Parallel …
Figure 8
Figure 8. Figure 8: A case study on task scikit 49 from REPOCOD. This example explains how the RRW contributes to the efficiency and effectiveness of the code refinement in TENET. A.2.3 REFLECTION-BASED REFINEMENT WORKFLOW [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: A failure case study on task more itertools-66 from REPOCOD. A.3.1 RANDOM SELECTION Random Selection (RS) acts baseline that uniformly samples a fixed number T of test cases from the full test suite. It helps benchmark the impact of smarter selection strategies. A.3.2 …
Figure 11
Figure 11. Figure 11: Test distributions on RE￾POCOD based on FRS strategy. FR: tests containing failure revealing struc￾ture; NFR: tests that have no failure re￾vealing structure. or multiple assertions, making their failures harder to interpret and potentially misleading for the agent. T…
Figure 12
Figure 12. Figure 12: Test distributions on REPOCOD based on the invocation depth from the test function to the target function. Stage seaborn flask xarray sphinx sympy more-itertools datasets scikit-learn astropy pylint plotly.py NoTest 41.03 53.49 18.07 30.30 24.74 51.16 38.98 20.06 37.6…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references

  1. [1]

    The information of the target function

  2. [3]

    Tests for the target function (if available) Your task is to generate the function body of the target. ## Target Code Information: **Target Function Name: **: ‘log_loss‘; **File Location: **: ‘sklearn/neural_network/_base.py‘; **Line Location: **: from line 175 to line 191; **Source Code: **: ‘‘‘ def log_loss(y_true, y_prob): """Compute Logistic loss for ...

  3. [4]

    IMPLEMENTATION: Make focused, minimal changes to address the problem

  4. [5]

    If the environment is not set up to run tests, consult with the user first before investing time to run tests

    VERIFICATION: If the environment is set up to run tests, test your implementation thoroughly, including edge cases. If the environment is not set up to run tests, consult with the user first before investing time to run tests. </PROBLEM_SOLVING_WORKFLOW> 17 This paper is a preprint currently under review for publication. <SECURITY> * Only use GITHUB_TOKEN...

  5. [8]

    why is X happening

    Do not generate additional code or patches other than the above target function. </issue> The prompt of out tailored agent toolset: Based on the the task, you can use the following search APIs to get more context: - search_test_cases(): Search for test cases of the target function. Analyzing test cases can help you to refine your solution. These test case...

  6. [9]

    EXPLORATION: Thoroughly explore relevant files and understand the context before proposing solutions

  7. [10]

    ANALYSIS: Consider multiple approaches and select the most promising one

  8. [11]

    TESTING: * For bug fixes: Create tests to verify issues before implementing fixes * For new features: Consider test-driven development when appropriate * If the repository lacks testing infrastructure and implementing tests would require extensive setup, consult with the user before investing time in building testing infrastructure * If the environment is...

Show all 42 references
  1. [14]

    First, look around in the repository for existing dependency files (requirements.txt, pyproject.toml, package.json, Gemfile, etc.)

  2. [15]

    If dependency files exist, use them to install all dependencies at once (e.g., ‘pip install -r requirements.txt‘, ‘npm install‘, etc.)

  3. [16]

    </ENVIRONMENT_SETUP> <TROUBLESHOOTING> * If you’ve made repeated attempts to solve a problem but tests still fail or the user reports it’s still broken:

    Only install individual packages directly if no dependency files are found or if only specific packages are needed * Similarly, if you encounter missing dependencies for essential tools requested by the user, install them when possible. </ENVIRONMENT_SETUP> <TROUBLESHOOTING> *...

  4. [17]

    Step back and reflect on 5-7 different possible sources of the problem

  5. [18]

    Assess the likelihood of each possible cause

  6. [19]

    Methodically address the most likely causes, starting with the highest probability

  7. [20]

    Instead, propose a new plan and confirm with the user before proceeding

    Document your reasoning process * When you run into any major issue while executing a plan from the user, please don’t try to directly work around it. Instead, propose a new plan and confirm with the user before proceeding. </TROUBLESHOOTING> I’ve uploaded a python code reposi...

  8. [21]

    openhands@all-hands.dev

    Run ‘git config --global user.email "openhands@all-hands.dev"‘ and ‘git config --global user.name "OpenHands Bot"‘ to set your identity

  9. [22]

    && git commit -q -m "init"‘ to initialize the folder as a new Git repo

    Run ‘rm -rf .git && git init && git add . && git commit -q -m "init"‘ to initialize the folder as a new Git repo. Here are the steps for you to follow:

  10. [23]

    Explore the repository to familiarize yourself with its structure

  11. [24]

    Check the corresponding code of test cases in ‘relevant_test_cases.txt‘ to understand the expected functionality of the target function

  12. [25]

    Complete the body of the target function

  13. [26]

    Execute the test cases in ‘relevant_test_cases.txt‘ to ensure your completed function passes the test cases

  14. [27]

    Additional notes: - When running Python, make sure to use ‘/opt/miniconda3/envs/testbed/bin/python‘

    Use the ‘git diff‘ command to produce a patch file named ‘patch.diff‘ containing your implementation changes. Additional notes: - When running Python, make sure to use ‘/opt/miniconda3/envs/testbed/bin/python‘. - Do not change or delete any code that already exists in the repo...

  15. [28]

    The information of the target function 18 This paper is a preprint currently under review for publication

  16. [29]

    Access to the entire project for retrieval and analysis

  17. [30]

    Tests for the target function (if available) Your task is to generate the function body of the target. ## Target Code Information: **Target Function Name: **: ‘log_loss‘; **File Location: **: ‘sklearn/neural_network/_base.py‘; **Line Location: **: from line 175 to line 191; **...

  18. [31]

    Avoid importing additional packages or libraries unless they already exist or considered necessary

  19. [32]

    Ensure your generated code has correct indentation and follows the same formatting style as the context

  20. [33]

    Do not generate additional code or patches other than the above target function. </issue> A.1.4 SWE-AGENT SETTING: You are a helpful assistant and a senior developper that can interact with a computer terminal and other provided tools to solve code generation tasks. The specia...

  21. [34]

    && git commit -q -m "init"‘ first to initialize the folder as a new Git repo

    Run ‘rm -rf .git && git init && git add . && git commit -q -m "init"‘ first to initialize the folder as a new Git repo. THIS IS A MUST!

  22. [35]

    Quickly find the file where the target function is located and find the specific line number where the target function is located

  23. [36]

    Explore the repository and collect necessary context to familiarize yourself with the repo and the target

  24. [37]

    DO NOT USE EXTRA TESTS OR WRITE YOUR OWN

    Utilize the test cases to make sure the completed function passes the test cases. DO NOT USE EXTRA TESTS OR WRITE YOUR OWN

  25. [38]

    when you believe you finish the task, use the ‘submit‘ action to submit the task. TIPS:

  26. [39]

    ONLY complete the body of the target function, and DO NOT change or delete any code that already exists in the repo

  27. [40]

    If you open a file and need to get to an area around a specific line, using the goto command, such as ‘goto 583‘, is much quicker

  28. [41]

    Make sure to look at the currently open file and the current working directory (which appears right after the currently open file). The currently open file might be in a different directory than the working directory! Note that some commands, such as ’create ’, open files, so ...

  29. [42]

    Always check the code after you issue an edit to make sure that it reflects what you wanted to accomplish

    When editing files, it is easy to accidentally specify a wrong line number or to write code with incorrect indentation. Always check the code after you issue an edit to make sure that it reflects what you wanted to accomplish. If it didn’t, issue another command to fix it

  30. [43]

    MKAE SURE your ouput in each round only consider ONE discussion and ONE command! Please wait for a response from the shell before continuing with more discussion and commands

  31. [44]

    blue", 50) def test_desaturate(): out5 = utils.desaturate(

    Again, if all three provided tests passes, you no longger need do extra tests. You may consider yourself already finished the task. Now, let’s start solving the task. (Open file: ) (Current directory: /testbed) bash-$ 20 This paper is a preprint currently under review for publ...

    1918

  32. [151]

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin

    URLhttps://aclanthology.org/2023.emnlp-main.151/. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.),...

    2023

  33. [896]

    more_itertools/more.py

    The agent first invokes three APIs in [1] to search for the similar method, usage examples of the target, and the top-level import statements at filemore itertools/more.py. After analyzing the context, TENETreasons that the target functionwindowedshould generate sliding window...

    1978

  34. [2025]

    doi: 10.1109/ICSE55347.2025.00108

    IEEE Computer Society. doi: 10.1109/ICSE55347.2025.00108. URLhttps://doi. ieeecomputersociety.org/10.1109/ICSE55347.2025.00108. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions elicit better llm agents. InProceedin...

    2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.