Pith. sign in

REVIEW 5 major objections 4 minor 53 references

A behavior-first handbook, generated from source code, makes agent harnesses navigable by behavior and lifts edit-plan win rates by up to 19 points while using fewer planner tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:36 UTC pith:QB3YXSH6

load-bearing objection A genuinely useful behavior-centric handbook for agent harnesses, but the headline gains rest on LLM-built answer keys drawn from the same source the handbook is built from; worth reviewing, with human-validated ground truth demanded. the 5 major comments →

arxiv 2607.13285 v1 pith:QB3YXSH6 submitted 2026-07-14 cs.AI cs.SE

Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

Ruhan Wang , Yucheng Shi , Zongxia Li , Zhongzhi Li , Yue Yu , Junyao Yang , Kishan Panaganti , Haitao Mi
show 2 more authors
Dongruo Zhou Leoweiliang
This is my paper
classification cs.AI cs.SE
keywords agent harnessbehavior localizationcode modification planningrepository representationLLM agentsprogressive disclosureharness evolutionstate-register view
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Behavior localization — finding every code site that implements a behavior described in a modification request — is the real bottleneck in evolving AI-agent harnesses, the paper argues. To remove it, the authors introduce the Harness Handbook, an automatically synthesized behavior-centric representation that links each runtime behavior to its source, and Behavior-Guided Progressive Disclosure (BGPD), a workflow that navigates from high-level behaviors to verified implementation details. On two open-source agent harnesses, handbook-assisted planning beat direct repository exploration on plan quality and localization accuracy while using fewer planner tokens: overall win rates rose from 28.3% to 38.3% and from 26.7% to 45.6%, and token use fell by 12.7% and 8.6%. The paper concludes that making behavior-to-code mapping explicit, and keeping it synchronized as code changes, is a necessary step not only for editing but also for behavior auditing and regression-impact analysis.

Core claim

The paper's central claim is that organizing a harness repository around runtime behavior, rather than files and functions, makes behavior localization tractable for coding agents. The Harness Handbook does this with a three-level document tree — system overview, stage/component overviews, and source-grounded unit details — plus a state-register view that records every read and write site of cross-stage state. BGPD uses the handbook as a pure location index, progressively disclosing relevant stages and entries, expanding candidates along call relations, and then verifying each candidate locator against the current repository before any edit is planned; a resynchronization step refreshes the

What carries the argument

The load-bearing object is the Harness Handbook, an L1–L3 document tree plus a state-register view, constructed automatically from a repository via deterministic static analysis (call graph, function/state facts) followed by LLM-assisted behavioral structuring under two leaf modes — function-as-leaf for compact harnesses with a trusted seed skeleton, and file-as-leaf for large repositories where the stage skeleton is inferred. BGPD (Behavior-Guided Progressive Disclosure) is the navigation policy that takes a request from coarse behavior descriptions to fine source locators and verifies them against the live repository, ensuring the repository, not the handbook, remains authoritative for the

Load-bearing premise

The evaluation assumes that LLM-generated answer keys and LLM judges, built from the same source and the same model families as the system under test, provide valid ground truth for localization and plan quality.

What would settle it

Have human experts with no stake in the handbook independently annotate the required edit sites for the 60 modification requests and score the baseline versus handbook-assisted plans; if the win-rate gap narrows or reverses, the core claim does not survive.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Handbook-assisted planning improves overall win rates on both harnesses (28.3% to 38.3% and 26.7% to 45.6%) while cutting planner token use by 12.7% and 8.6%.
  • A weaker planner's localized edit sites align far more closely with stronger models' reference plans: all 24 Recall/Precision/F1 comparisons improve, by margins from 5.0 to 18.8 points, and complete localization failures fall by up to 25.9 points.
  • Gains hold across request types (existing-behavior changes, cross-file capabilities, search-hostile sites) and across easy, medium, and hard localization difficulty.
  • Because the handbook resynchronizes after every diff, it can serve as living, code-accurate documentation rather than a one-shot index.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The behavior-first representation is likely to generalize beyond agent harnesses to any codebase where behaviors are distributed across modules and hidden in fallback or cold paths; the construction cost would need to be traded against code volatility.
  • A direct extension would test whether the handbook's benefit grows with repository scale and coupling: the largest gains here appear on the larger, multi-crate harness and on cross-file and search-hostile requests.
  • The paper's evaluation relies on LLM-built answer keys and LLM judges; a human-annotated subset of the 60 requests would provide an independent check on whether the gains are real or reflect shared model biases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper addresses behavior localization in the evolution of agent harnesses: given a natural-language modification request, a developer or coding agent must find every code location implementing the target behavior. The authors propose Harness Handbook, a three-level (L1–L3) behavior-centric representation automatically constructed from a repository via deterministic static analysis plus LLM-assisted organization, together with Behavior-Guided Progressive Disclosure (BGPD), a localization workflow that navigates the handbook coarse-to-fine and verifies candidate locators against the live repository. The handbook is automatically resynchronized after repository diffs. The evaluation compares a Baseline planner (DeepSeek-V4-Pro with read-only repository tools) against a Handbook-Assisted arm on 30 modification requests each from Terminus-2 (Python, function-as-leaf) and Codex (Rust, file-as-leaf). Reported results: overall win rates rise from 28.3% to 38.3% on Codex and 26.7% to 45.6% on Terminus-2; planner tokens fall by 12.7% and 8.6%; all 24 file/symbol Recall–Precision–F1 comparisons against LLM-generated reference plans favor the Handbook-Assisted arm; and win-rate gains are reported across all request types and difficulty levels. The central claim is that making the behavior-to-code mapping explicit improves localization and edit-plan quality at lower cost.

Significance. The problem is well chosen and the proposed representation is concrete and automatable. Strengths include: a clear task definition; a construction pipeline with a deterministic, LLM-free Phase I and explicit validation/freezing rules; detailed algorithms (Algorithms 1 and 2) and full prompt templates; and a two-harness design that instantiates both leaf modes at very different scales. The evaluation is also falsifiable: it reports win rates, F1, and token counts on defined slices. If the results hold, the work would be a useful step for repository understanding for coding agents. However, the empirical support for the central claim currently rests on an evaluation whose ground truth is LLM-generated from the same pristine source used to build the handbook, with no human validation, no external benchmark, and no confidence intervals or significance tests. The reported gains are therefore plausible but not yet established.

major comments (5)
  1. [§4.1, App. D.3.1] The evaluation's ground truth is not established as independent of the intervention. The answer key is generated by an LLM reading the pristine source, and the Handbook is constructed from that same source; no human validation of the keys is reported, and one of the three scoring judges (DeepSeek-V4-Pro) is the same model as the planner. This does not force the reported gains—the Baseline could in principle match the key—but a shared bias is plausible: if the key's notion of 'required_sites' emphasizes the same stage/register structure the Handbook encodes, the Handbook-Assisted arm is scored against a key it is structurally aligned with. The paper reports no measure of key stability. Please add: (i) human-expert validation of a sample of answer keys with agreement statistics; (ii) cross-judge agreement on required_sites for the same requests; (iii) sensitivity of the headline win rates
  2. [§4.2.1, Fig. 3] No uncertainty is reported for any headline number. With n=30 requests per harness, the Codex win-rate gap (10.0 points overall) is within roughly two standard errors of a proportion, and the Figure 5 cells, which split 30 requests into three types or difficulties, have about 10 requests each. Report Wilson intervals, paired tests at the request level, the number of valid judge–request comparisons per slice (the denominator is never stated), and the per-request distribution of judge scores. The claim of 'consistent gains' across all slices cannot be assessed without this, especially the 3.7- to 33.3-point difficulty-stratified gaps.
  3. [Table 1] The Terminus-2 Wrong values are identical at file and symbol granularity (24.1/13.8 for Opus 4.8 and 20.0/6.7 for GPT-5.5, respectively), and several other file/symbol pairs are nearly identical. This suggests the 'symbol' granularity is not actually resolving reference sites to symbols, or the block was copy-pasted. Since the paper uses symbol-level results to argue that gains do not come from returning more candidates, clarify how symbol-level overlap is computed (must a predicted site match a function/region anchor?) and fix the table. Also, some Codex gaps against the GPT-5.5 reference are tiny (+0.4 file Recall) or zero (Wrong 0.0), so 'all 24 comparisons higher' overstates uniformity; report which comparisons survive a confidence interval.
  4. [§4.1, App. C.3] The two arms differ not only in handbook availability but in the full localization instruction policy: the Handbook-Assisted arm receives SKILL.md with explicit directives ('Do NOT prematurely narrow to the one obvious stage,' 'note EVERY write and read site'), while the Baseline receives no corresponding guidance. The paper states that the arms differ only in 'handbook access and navigation,' but navigation instructions are part of the treatment, so the design cannot attribute the gains to the behavior-centric representation rather than to the BGPD policy alone. Add an ablation: the same planner with BGPD-style instructions and a non-behavioral index (e.g., the raw file tree) instead of the Handbook. If that arm also gains, the RQ1 attribution must be restated.
  5. [§4.1, App. D.3.2] The evaluation scores plans, not executed edits: the judge prompt explicitly says 'Do NOT grade the final code diff or execution correctness.' Consequently 'edit-plan quality' is LLM-judged plan quality against an LLM-generated key. The executor and its syntax gate are described in App. B.3, but their outcomes are never reported. Please report executor application success and syntax-gate results on the Handbook-Assisted plans (and ideally on a sample of Baseline plans), so that plan quality is anchored to something beyond the judge's own opinion.
minor comments (4)
  1. [Fig. 3(b)] The per-judge bars are cramped and the extracted numbers show exactly +10.0 for every judge on Codex. Verify the aggregation and state whether the per-judge values are rounded; identical rounded values across three models are suspicious and deserve a comment.
  2. [§4.1, Eq. (1)] Clarify the scale mapping between the 0–5 dimension scores and the 0–100 score S, and how the δ=3 win threshold is applied (in scaled points or raw points). The polarity of scope_bloat vs. Scope Control should be stated in the main text, not only in App. D.3.2.
  3. [App. D.3.1] The Codex answer-key prompt is said to 'mirror' the Terminus-2 prompt with correct_approach in place of correct_localization. Reproduce the Codex prompt in full so the two-key construction is auditable.
  4. [§4.1] Define 'valid judge–request comparisons' precisely: which outputs are excluded as 'missing' or 'planner errors,' and what are the per-slice denominators? Without denominators, the win-rate percentages in Figures 3 and 5 cannot be interpreted.

Circularity Check

0 steps flagged

No significant circularity: the handbook gains are measured by an empirical arm comparison with reference keys generated independently of the handbook.

full rationale

The paper's central claims (higher win rates, better localization F1, lower token cost with Handbook-Assisted planning) are empirical comparisons between two arms that differ only in handbook access; both arms share the same requests, repository snapshot, planner model, tool permissions, and decoding settings (Section 4.1; Appendix C.3). The Harness Handbook is constructed from the repository via Phase I deterministic static analysis plus LLM-assisted behavioral organization (Section 3.2; Appendix A), while the answer keys used for plan scoring are built by judge LLMs from the pristine source before any plan is seen, with judge prompts that never reference the handbook (Appendix D.3.1). There is no equation by which predicted sites are defined in terms of the handbook, no parameter is fitted to the evaluation data and then renamed a prediction, and no load-bearing uniqueness theorem is imported from the authors' prior work. The shared-source provenance of the handbook and the answer keys, and the fact that one judge (DeepSeek-V4-Pro) is the same model family as the planner, are threats to evaluation validity: the LLM-generated reference may share systematic biases with the handbook-assisted planner, which could inflate measured agreement. But this is not circularity by construction, because the baseline could in principle match the reference and the comparisons are not forced by definition. Those concerns belong in a validity or correctness-risk review, not in a circularity finding under the strict reduction-based standard used here.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

The central claim is not a derivation; it is an empirical claim about a tool. It depends on the assumptions above and on an invented representation artifact. No numeric free parameters are fitted.

axioms (3)
  • domain assumption Behavior localization is a central bottleneck in harness evolution.
    Stated in Introduction and used to motivate the whole contribution; not empirically demonstrated against alternative bottlenecks.
  • domain assumption LLM judges and LLM-generated answer keys reliably measure localization and plan quality.
    Used in Section 4.1/D.3; no human annotation or inter-judge agreement analysis.
  • domain assumption The two selected harnesses and 30 hand-authored requests per harness are representative of production harness evolution.
    Section 4.1/C.2; request and difficulty labels are authored by the paper's authors.
invented entities (1)
  • Harness Handbook no independent evidence
    purpose: Behavior-centric representation linking behaviors to source locations to guide edit planning
    The artifact's claimed benefit is supported only by the paper's own evaluation; no independent benchmark or external falsifiable prediction is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 24287 in / 11466 out tokens · 93007 ms · 2026-08-02T05:36:09.469177+00:00 · methodology

0 comments
read the original abstract

The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the target behavior. This is difficult because production harnesses are large, tightly coupled, and behaviorally distributed, while modification requests describe what the system should do and repositories are organized by files and modules. Code search, repository indexing, and long-context processing ease inspection, but still leave this behavior-to-code mapping to be recovered by hand. Behavior localization is therefore a central bottleneck in harness evolution. We introduce the Harness Handbook, a behavior-centric representation synthesized automatically from a harness codebase via static analysis and LLM-assisted structuring, linking each behavior to its corresponding source. We also introduce Behavior-Guided Progressive Disclosure (BGPD), which guides agents from high-level behaviors to relevant implementation details and verifies candidate locations against the current source. On diverse modification requests from two open-source harnesses, Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens, with the largest gains on scattered sites, rarely executed paths, and cross-module interactions. Evolving complex agentic systems thus depends not only on generating edits, but also on determining where those edits should be made.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 19 linked inside Pith

  1. [1]

    A survey on large language model based autonomous agents.Frontiers of Computer Science, 18 (6):186345, 2024

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18 (6):186345, 2024

    2024

  2. [2]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Pith/arXiv arXiv 2022

  3. [3]

    Towards agentic recommender systems in the era of multimodal large language models.arXiv preprint arXiv:2503.16734, 2025

    Chengkai Huang, Junda Wu, Yu Xia, Zixu Yu, Ruhan Wang, Tong Yu, Ruiyi Zhang, Ryan A Rossi, Branislav Kveton, Dongruo Zhou, et al. Towards agentic recommender systems in the era of multimodal large language models.arXiv preprint arXiv:2503.16734, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36:68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in neural information processing systems, 36:68539–68551, 2023

    2023

  5. [5]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606, 2024

    2024

  6. [6]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    2024

  7. [7]

    Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  8. [8]

    Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

    Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, et al. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Harness engineering: Leveraging codex in an agent-first world.https://openai.com/index/ harness-engineering/, February 2026

    Ryan Lopopolo. Harness engineering: Leveraging codex in an agent-first world.https://openai.com/index/ harness-engineering/, February 2026. OpenAI Engineering Blog

    2026

  10. [10]

    Ai harness engineering: A runtime substrate for foundation-model software agents.arXiv preprint arXiv:2605.13357, 2026

    Hailin Zhong and Shengxin Zhu. Ai harness engineering: A runtime substrate for foundation-model software agents.arXiv preprint arXiv:2605.13357, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    Federated in-context learning: Iterative refinement for improved answer quality.arXiv preprint arXiv:2506.07440, 2025

    Ruhan Wang, Zhiyong Wang, Chengkai Huang, Rui Wang, Tong Yu, Lina Yao, John Lui, and Dongruo Zhou. Federated in-context learning: Iterative refinement for improved answer quality.arXiv preprint arXiv:2506.07440, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Fera: Uncertainty-aware federated reasoning for large language models.arXiv preprint arXiv:2605.10082, 2026

    Ruhan Wang, Chengkai Huang, Zhiyong Wang, Junda Wu, Rui Wang, Tong Yu, Julian McAuley, Lina Yao, and Dongruo Zhou. Fera: Uncertainty-aware federated reasoning for large language models.arXiv preprint arXiv:2605.10082, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pages 65882–65919, 2025

    2025

  14. [14]

    Semaclaw: A step towards general-purpose personal ai agents through harness engineering

    Ningyan Zhu, Huacan Wang, Jie Zhou, Feiyu Chen, Shuo Zhang, Ge Chen, Chen Liu, Jiarou Wu, Wangyi Chen, Xiaofeng Mou, et al. Semaclaw: A step towards general-purpose personal ai agents through harness engineering. arXiv preprint arXiv:2604.11548, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Code as agent harness.arXiv preprint arXiv:2605.18747, 2026

    Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, Zhining Liu, Ting-Wei Li, et al. Code as agent harness.arXiv preprint arXiv:2605.18747, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    Swe-explore: Benchmarking how coding agents explore repositories.arXiv preprint arXiv:2606.07297, 2026

    Shaoqiu Zhang, Yuhang Wang, Jialiang Liang, Yuling Shi, Wenhao Zeng, Maoquan Wang, Shilin He, Ningyuan Xu, Siyu Ye, Kai Cai, et al. Swe-explore: Benchmarking how coding agents explore repositories.arXiv preprint arXiv:2606.07297, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    Sherloc: Structured diagnostic localization for code repair agents.arXiv preprint arXiv:2606.24820, 2026

    Hovhannes Tamoyan, Sean Narenthiran, Erik Arakelyan, Mira Mezini, and Boris Ginsburg. Sherloc: Structured diagnostic localization for code repair agents.arXiv preprint arXiv:2606.24820, 2026. 11

    Pith/arXiv arXiv 2026

  18. [18]

    Code isn’t memory: A structural codebase index inside a coding agent.arXiv preprint arXiv:2606.22417, 2026

    Ishaan Bhola, Adithyan Krishnan, Sravanth Kurmala, and Mukunda NS. Code isn’t memory: A structural codebase index inside a coding agent.arXiv preprint arXiv:2606.22417, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Improving code localization with repository memory.arXiv preprint arXiv:2510.01003, 2025

    Boshi Wang, Weijian Xu, Yunsheng Li, Mei Gao, Yujia Xie, Huan Sun, and Dongdong Chen. Improving code localization with repository memory.arXiv preprint arXiv:2510.01003, 2025

    arXiv 2025

  20. [20]

    Coding agents are effective long-context processors

    Weili Cao, Xunjian Yin, Bhuwan Dhingra, and Shuyan Zhou. Coding agents are effective long-context processors. arXiv preprint arXiv:2603.20432, 2026

    arXiv 2026

  21. [21]

    Agent harness for large language model agents: A survey

    Qianyu Meng, Yanan Wang, Liyi Chen, Qimeng Wang, Chengqiang Lu, Wei Wu, Yan Gao, Yi Wu, and Yao Hu. Agent harness for large language model agents: A survey. 2026

    2026

  22. [22]

    Harness engineering for language agents: The harness layer as control, agency, and runtime

    Chaoyue He, Xin Zhou, Di Wang, Hong Xu, Wei Liu, and Chunyan Miao. Harness engineering for language agents: The harness layer as control, agency, and runtime. 2026

    2026

  23. [23]

    Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Harness design for long-running application development.https://www.anthropic.com/ engineering/harness-design-long-running-apps, March 2026

    Prithvi Rajasekaran. Harness design for long-running application development.https://www.anthropic.com/ engineering/harness-design-long-running-apps, March 2026. Anthropic Engineering Blog

    2026

  25. [25]

    Introducing codex

    OpenAI. Introducing codex. https://openai.com/index/introducing-codex/, May 2025. OpenAI Research Preview

    2025

  26. [26]

    Harnessx: A composable, adaptive, and evolvable agent harness foundry.arXiv preprint arXiv:2606.14249, 2026

    Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, et al. Harnessx: A composable, adaptive, and evolvable agent harness foundry.arXiv preprint arXiv:2606.14249, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Harnessforge: Joint harness and policy evolution for adaptive agent systems.arXiv preprint arXiv:2606.01779, 2026

    Mingju Chen, Can Lv, Guibin Zhang, Heng Chang, and Shiji Zhou. Harnessforge: Joint harness and policy evolution for adaptive agent systems.arXiv preprint arXiv:2606.01779, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    From failed trajectories to reliable llm agents: Diagnosing and repairing harness flaws.arXiv preprint arXiv:2606.06324, 2026

    Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, and Qing Wang. From failed trajectories to reliable llm agents: Diagnosing and repairing harness flaws.arXiv preprint arXiv:2606.06324, 2026

    Pith/arXiv arXiv 2026

  29. [29]

    Repository intelligence graph: Deterministic architectural map for llm code assistants.arXiv preprint arXiv:2601.10112, 2026

    Tsvi Cherny-Shahar and Amiram Yehudai. Repository intelligence graph: Deterministic architectural map for llm code assistants.arXiv preprint arXiv:2601.10112, 2026

    arXiv 2026

  30. [30]

    Natural-language agent harnesses.arXiv preprint arXiv:2603.25723, 2026

    Linyue Pan, Lexiao Zou, Shuo Guo, Jingchen Ni, and Hai-Tao Zheng. Natural-language agent harnesses.arXiv preprint arXiv:2603.25723, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    Nex-n1: Agentic models trained via a unified ecosystem for large-scale environment construction.arXiv preprint arXiv:2512.04987, 2025

    Nex-AGI Team, Yuxuan Cai, Lu Chen, Qiaoling Chen, Yuyang Ding, Liwen Fan, Wenjie Fu, Yufei Gao, Honglin Guo, Pinxue Guo, et al. Nex-n1: Agentic models trained via a unified ecosystem for large-scale environment construction.arXiv preprint arXiv:2512.04987, 2025. URLhttps://arxiv.org/abs/2512.04987

    arXiv 2025

  32. [32]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. InThe Fourteenth International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2601.11868

    Pith/arXiv arXiv 2026

  33. [33]

    Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anand- kumar. Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Repository-level code understanding by llms via hierarchical summarization: Improving code search and bug localization

    Amirkia Rafiei Oskooei, Selcan Yukcu, Mehmet Cevheri Bozoglan, and Mehmet S Aktas. Repository-level code understanding by llms via hierarchical summarization: Improving code search and bug localization. In International Conference on Computational Science and Its Applications, pages 88–105. Springer, 2025

    2025

  35. [35]

    Proconsul: Project context for code summarization with llms

    Vadim Lomshakov, Andrey Podivilov, Sergey Savin, Oleg Baryshnikov, Alena Lisevych, and Sergey Nikolenko. Proconsul: Project context for code summarization with llms. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 866–880, 2024

    2024

  36. [36]

    are you sure?

    Siddharth Gandhi, Luyu Gao, and Jamie Callan. Repository-level code search with neural retrieval methods. arXiv preprint arXiv:2502.07067, 2025. 12 Appendix Contents A Harness Handbook Construction Details 14 A.1 Phase I: Shared Static Fact Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 A.2 Phase II: Behavioral Organization . ...

    Pith/arXiv arXiv 2025

  37. [37]

    Read references/overview.md first to understand the whole system

  38. [38]

    Do NOT prematurely narrow to the one obvious stage: a single change often touches sites in several stages

    Read references/index.md to identify the stages, leaves, AND state registers your change involves. Do NOT prematurely narrow to the one obvious stage: a single change often touches sites in several stages

  39. [39]

    This read/write registry surfaces scattered, non-adjacent sites a top-down code read would miss

    For EVERY state register your change touches, read references/registers.md and note EVERY write and read site. This read/write registry surfaces scattered, non-adjacent sites a top-down code read would miss

  40. [40]

    Open the relevant references/stages/<id>.md for detail and coupled assumptions

  41. [41]

    For each site the handbook names, open the real code with read_file / search_file_content and locate the precise lines

  42. [42]

    function

    Your plan must account for every site the handbook surfaced. The handbook tells you WHERE things live and how they connect. You still decide what the change should be, and you verify every location against the real code before planning it. Figure 6The SKILL.md manifest that exposes a handbook to the planner in the Handbook-Assisted arm. The same structure...

  43. [43]

    handles" /

    ACTION: what the function does, concretely (no "handles" / "manages")

  44. [44]

    INPUTS / STATE READ: arguments + self._* attrs that determine behavior

  45. [45]

    OUTPUTS / STATE WRITTEN: return value + self._* attrs mutated

  46. [46]

    WHEN INVOKED: who calls it, under what condition

  47. [47]

    qualname

    NON-OBVIOUS: retry logic, fallback paths, design choices a reader would miss CONSISTENCY WITH CONTEXT Look at the caller/callee context provided -- your classification should be consistent with their stage assignments. A function that’s only called from stage-4.2’s region typically belongs near stage-4.2. OUTPUT Return ONLY a JSON object inside a ‘‘‘json ...

  48. [48]

    Understand the request’s true intent: the behavior delta, and the state/conditions/values it fixes

  49. [49]

    Read its ‘SKILL.md‘, then ‘index.md‘, then only the ‘stages/<id>.md‘ chapters and ‘registers.md‘ entries your intent points to

    Route with the handbook. Read its ‘SKILL.md‘, then ‘index.md‘, then only the ‘stages/<id>.md‘ chapters and ‘registers.md‘ entries your intent points to. Assemble the candidate set: every file + function + anchor the change must touch. Watch for scattered/mirror sites (a parser change usually has a twin in the OTHER parser and in both prompt templates; a s...

  50. [50]

    Confirm the exact body, control flow, conditions, and that the site does what the card implied

    Read the REAL source of every site you intend to edit with ‘read_file‘ on the actual code files. Confirm the exact body, control flow, conditions, and that the site does what the card implied

  51. [51]

    Match whitespace and indentation exactly, and include at least 3 lines of context BEFORE and AFTER the changed lines so the snippet is UNIQUE in the file

    For EACH edit, produce a self-contained EDIT BLOCK (format below) whose ‘old_string‘ is copy-pasted verbatim from the ‘read_file‘ output you just saw -- never retyped from memory, never paraphrased. Match whitespace and indentation exactly, and include at least 3 lines of context BEFORE and AFTER the changed lines so the snippet is UNIQUE in the file

  52. [52]

    A change can also silently break something the request never mentions; if you find such a coupled assumption, add an edit (or note it) accordingly

  53. [53]

    (new file)

    Only include edits you are confident the request requires. ## EDIT BLOCK format (the executor applies these directly) For every edit, output exactly: ### EDIT <n> - file: ‘<path relative to the working dir, e.g. {{PATH_EXAMPLE}}>‘ - where: ‘<{{WHERE_EXAMPLE}}>‘ -- why this change ‘‘‘old <EXACT current text, copied verbatim from read_file -- whitespace-per...