Pith. sign in

REVIEW 4 major objections 5 minor 44 references

HarnessCompass claims that constraining evolution edits to task-agnostic changes, adding trajectory-grounded first-person agent feedback, and optimizing harness components on separate tracks produces an evolved coding-agent harness that jum

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A constrained, feedback-augmented, component-wise harness evolution loop improves coding agent Pass@1 on SWE-bench Verified and transfers to held-out tasks and another base model.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A well-designed harness-evolution framework with credible held-out gains; the main caveat is that its key constraint is only prompt-enforced, not mechanically audited. the 4 major comments →

arxiv 2608.01918 v1 pith:BNUTSSXB submitted 2026-08-03 cs.LG cs.CL

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Luan Zhang , Ruochen Zhou , Dandan Song , Zhengyu Chen , Yuhang Tian , Jun Yang , Huipeng Ma , Chenhao Li
show 4 more authors
Guangyuan Feng Xudong Li Yizhou Jin Yan Xu
This is my paper
classification cs.LG cs.CL
keywords automatic harness evolutionLLM agentsSWE-benchgeneralizationfirst-person feedbackcomponent-wise optimizationgeneralization gatecoding agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that automatic harness evolution—searching over the software layer that connects a language model to its environment—fails when the search is unconstrained, starved of evidence, and edits everything at once. HarnessCompass applies three disciplines: edits must be task-agnostic (a generalization gate), the agent's own first-person reports about harness friction are elicited and then checked against trajectories, and structural and guidance components are evolved on separate tracks before being merged. On SWE-bench Verified with GPT-5.4, the evolved harness raises Pass@1 from 54% to 66% in five evolution iterations, beating the prior method AHE's 63% in twenty, and it transfers to the 450 held-out tasks and to a second base model. A sympathetic reader should care because it reframes harness design as a disciplined optimization problem where generalization is built in rather than hoped for.

Core claim

The central claim: automatic harness evolution overfits, misattributes, and interferes mainly because the search loop is unstructured; three mechanisms fix it. A Generalization Gate in the meta-agent's prompt forbids task-specific content and placement, forcing reusable criteria. The agent's first-person reports—solicited blind and with hindsight—stay only if the trajectory supports them. Two parallel variants evolve structural and guidance components, then R3 merge (Revision, Recombination, Refinement) folds the loser's useful edits into the winner. The paper reports 66.0% on 50 evolution tasks, 60.4% on 450 held-out, 61.0% overall, versus AHE's 63.0/54.7/55.5; the frozen harness also impro

What carries the argument

The load-bearing object is the harness itself, split into seven component types grouped into structural (tool implementations, middleware, sub-agent configurations) and guidance (system prompt, tool descriptions, skills, long-term memory). The argument runs through three mechanisms: a Generalization Gate defined in the meta-agent's prompt that bans task-specific content and placement; a proactive-feedback pipeline in which the same base model produces blind and hindsight reports, reconciles them, and grounds them against trajectory evidence before aggregating them into confidence-scored evidence; and component-wise optimization with two parallel variants plus an R3 merge that revises the los

Load-bearing premise

The load-bearing premise is that the meta-agent actually obeys the Generalization Gate: the gate exists only as instructions in the system prompt plus a self-check, and the paper's qualitative case study inspects memory artifacts without systematically verifying every committed edit across all iterations, so if the meta-agent occasionally slips, the claimed generalization could be inflated.

What would settle it

Inspect the full git history of the evolved harness across all five iterations and search every committed tool, middleware, system-prompt, and memory change for SWE-bench instance ids (e.g., django__django-13158), test names, private symbols, or keyword-matching branches; finding any such edit that survived to the final harness would falsify the generalization-gate claim. A cheaper check: rerun the same evolution with an automated gate verifier that mechanically rejects banned tokens and see whether performance and held-out transfer reproduce.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is right, automatic harness evolution can deliver gains comparable to model upgrades while keeping the base model fixed, and in far fewer evaluation rounds.
  • Evolved harnesses that pass the gate should transfer to held-out tasks, so search and evaluation sets can be disjoint without sacrificing progress.
  • A harness evolved on one base model can improve a different unseen base model, implying reusable engineering principles rather than model-specific quirks.
  • The three mechanisms are complementary: the gate alone improves generalization, feedback alone improves effectiveness but can hurt held-out performance, and R3 restores generalization while improving efficiency.
  • The approach implies that each edit's content and placement matters: guidance belongs in prompts and memory, capability belongs in executable code.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the Generalization Gate is enforced through a prompt rather than an automated verifier, a natural next step is a mechanical checker that scans every committed diff for banned tokens; if such checks become standard, gate violations would be caught before they enter the harness.
  • Component-wise separation suggests a broader recipe for iterative agent-improvement loops: if edits in different surfaces are correlated, optimizing them independently and then merging may avoid interference even beyond harness evolution.
  • First-person feedback may still be vulnerable to post-hoc rationalization despite the blind/hindsight design; a targeted test would compare feedback quality on tasks where the trace clearly shows a harness shortcoming versus tasks where the agent erred on its own.
  • If cross-model transfer is robust, harness evolution could become a form of model-agnostic infrastructure: evolve once with a strong model, deploy the harness with cheaper or newer models, and update only when the model's behavioral profile shifts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HarnessCompass, an iterative automatic harness-evolution framework for LLM coding agents. It augments the standard evaluate–analyze–improve loop with three mechanisms: a Generalization Gate that instructs the meta-agent to make only task-agnostic edits; proactive first-person feedback elicited from the agent before and after the verifier verdict and then grounded against trajectories; and component-wise optimization that evolves structural and guidance components on separate tracks before merging them via an R3 (Revision/Recombination/Refinement) procedure. On SWE-bench Verified with GPT-5.4, the paper reports improving Pass@1 from 54.0% (seed) to 66.0% in 5 evolution iterations, versus AHE's 63.0% in 20 iterations; on 450 held-out tasks the evolved harness reaches 60.4% versus AHE's 54.7%. A frozen-harness cross-model experiment on Claude-Sonnet-4.6 reports a +3.8 point total improvement. A cumulative ablation attributes the gains to the three principles.

Significance. If the results hold, the paper makes a useful contribution: it shows that harness evolution can be made substantially more sample-efficient and can transfer beyond the evolution set, and the three design principles are well motivated. The evaluation has real strengths: 450 held-out tasks disjoint from evolution, a minimal seed harness so that every later component is introduced by the loop, a fixed base model across all roles, and a frozen-harness cross-model transfer experiment. The held-out and cross-model numbers are the most persuasive evidence. The main weakness is that the mechanism claimed to produce generalization — the Generalization Gate — is only a prompt-level constraint and is never mechanically audited, and the final merged harness is accepted without being evaluated in its merged form. These concerns are fixable but currently temper the causal attribution.

major comments (4)
  1. [Constrained Evolution; Algorithm 1; Appendix 'Generalization Gate'] The Generalization Gate is implemented solely as a meta-agent system prompt and a self-check; there is no automated verifier that scans candidate edits or committed diffs for forbidden task ids, test names, private symbols, or keyword-matching branches. Algorithm 1's GatedEvolve is a black box, and nothing checks H_t after R3. Because the 450 held-out tasks are drawn from the same SWE-bench repositories as the 50 evolution tasks, a gate violation that names a repo-level pattern could inflate held-out Pass@1 and still look like transfer. The appendix case study explicitly concedes that 'Memory inspection alone does not prove generalization' and inspects only LongTermMEMORY, not the other six component surfaces. To support the central attribution, the paper should either mechanically enforce the gate (e.g., automated lint/scan of manifests and diffs) or audit all committed edits across all
  2. [Algorithm 1, lines 10-13; Component-wise Optimization] The merged harness H_t is not evaluated before acceptance. The acceptance condition uses Pass@1(T_w), the winner variant's score, not the score of the merged harness produced by R3. The R3 prompt asserts that the result is '>= winner, never worse', but that is a prompt instruction, not evidence. The final evolved harness may therefore never have been rolled out in its final merged configuration. This is load-bearing for the central effectiveness claim. Please evaluate the accepted H_t directly (or roll it out at the start of the next iteration and report the final artifact's own score).
  3. [Tables 1-3; Setup] The results report point estimates from what appears to be a single evolution run, with k=2 rollouts per task. The evolution set has only 50 tasks, so the 66.0% vs 63.0% difference against AHE is a small absolute change on a coarsely quantized sample; the held-out difference is more robust but still comes from one evolution trajectory. Without repeated seeds, confidence intervals, or bootstrap estimates, the 'outperforms AHE' claim is not strongly supported beyond a point estimate. Additionally, the Pass@1 definition is ambiguous: the main text says 'mean binary success rate over k rollouts per task', while Table 4 states per-trial success over 2n trials. Please clarify and report variability.
  4. [Ablation Study, Table 2] The cumulative ablation does not cleanly support the claimed marginal contribution of proactive feedback. Adding proactive feedback on top of the gate lowers held-out Pass@1 from 58.4% (gate-only) to 55.8% and increases turns from 2 to 12; held-out recovers to 60.4% only after R3 is added. As presented, feedback appears to add overfitting that R3 later removes, rather than to 'expand the optimization signal' on its own. To make the attribution load-bearing, the paper should report which feedback-driven edits were grounded and retained, or run an ablation that adds R3 without feedback to separate the two effects.
minor comments (5)
  1. [Evolution Efficiency; Figure 2] The text says HarnessCompass 'reaches its peak around iteration 6', while Figure 2 and Table 1 report the peak at iteration 5. Please align these statements.
  2. [HarnessCompass, first paragraph] 'motivated in Section.' has a missing section number.
  3. [Setup, Metrics] Please define Pass@1 precisely. With k=2 rollouts, is Pass@1 the per-task first-attempt success rate, the per-trial success rate, or the proportion of tasks solved in at least one rollout? The per-repository table implies per-trial success, which differs from the standard Pass@1 definition.
  4. [Table 2] The 'Turns' column would benefit from a definition: is it the first iteration at which the best-so-far harness reaches its final score, or the total number of iterations run before stopping? The ablation rows stop at different turns, and the stopping criterion is not stated.
  5. [Appendix, Case Study] The case study is clearly labeled as not proving generalization, which is commendable; consider moving this caveat into the main text so the mechanistic claim is not overstated there.

Circularity Check

0 steps flagged

No significant circularity: held-out and cross-model evaluations are external; self-citations are not load-bearing.

full rationale

No circular derivation chain is present. The headline results are measured on an external benchmark with an explicit train/eval separation: evolution uses 50 SWE-bench Verified tasks, while the remaining 450 tasks are 'kept unseen and used solely for evaluation' (Experiments/Setup); the cross-model result freezes the GPT-5.4-evolved harness and re-evaluates it on Claude-Sonnet-4.6. These are out-of-sample measurements, not quantities reconstructed from the method's own definitions. No fitted parameter is renamed as a prediction: the meta-agent's edits are accepted by Pass@1 on the evolution set, and the held-out score is then measured, not predicted. The self-citations (PruneTIR in Related Works; ActiShade/hallucination references in Future Work) are contextual and not load-bearing; the framework's component decomposition follows external work (Lin et al. 2026). The mild self-referential design — the same base model generates trajectories, feedback, and edits — is explicitly handled by grounding feedback against trajectories and treating it as 'candidate evidence only,' and the final claims rest on objective Pass@1. Limitations that should be weighed as correctness risks, not circularity, include: the Generalization Gate is enforced via the meta-agent's system prompt and self-check rather than an automated audit (Appendix), and the Case Study concedes 'Memory inspection alone does not prove generalization.' Algorithm 1 also accepts the merged harness H_t based on Pass@1(T_w) rather than directly evaluating H_t, which is an evaluation-validity concern but not a reduction of output to input. Thus the paper is self-contained against external benchmarks and exhibits no construction-level circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No numeric free parameters are fitted: the harness components are edited text/code artifacts, and the main design choices are experimental settings (50-task evolution set, k=2 rollouts, 5-20 turns). The central claims rest on behavioral assumptions about LLM compliance and benchmark representativeness rather than parameter fits. No new physical or architectural entities are postulated.

axioms (5)
  • domain assumption The meta-agent reliably follows the Generalization Gate and track-boundary instructions given in its system prompt.
    The gate is implemented as prompt text (Appendix 'Generalization Gate'), not as a mechanical verifier; all claims of task-agnostic edits depend on LLM instruction-following. Location: Constrained Evolution section and Appendix.
  • domain assumption First-person self-reports, after blind/hindsight reconciliation and trajectory grounding, provide valid evidence about harness friction.
    The feedback pipeline assumes the same base model's introspection is informative once grounded; unsupported claims are dropped, but the value of the survived signal is taken as given. Location: Proactive Feedback section.
  • domain assumption The 50-task evolution sample is representative enough of the remaining 450 SWE-bench Verified tasks for held-out Pass@1 to measure transfer.
    Tasks are sampled from the same 12 repositories, so held-out performance may overstate cross-domain generalization. Location: Experiments Setup.
  • ad hoc to paper The R3 merge does not degrade the winner variant.
    Algorithm 1 accepts the merged harness by comparing the winner variant's Pass@1 to the previous harness's Pass@1, without rolling out the merged harness; the 'never worse' property is asserted in the R3 prompt, not verified. Location: Algorithm 1 lines 10-14 and R3 integrator prompt.
  • domain assumption Pass@1 with k=2 rollouts is a stable estimator for comparing harnesses.
    No variance or significance is reported; small repositories have coarse quantization (e.g., one task, 50-point increments). Location: Experiments Setup and Per-Repository Results.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses." pith.science (2026). https://pith.science/paper/BNUTSSXB

@misc{pith2026260801918,
  author       = {Pith},
  title        = {Pith review of: HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNUTSSXB}},
  note         = {Machine review of arXiv:2608.01918}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54\% to 66\% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.

Figures

Figures reproduced from arXiv: 2608.01918 by Chenhao Li, Dandan Song, Guangyuan Feng, Huipeng Ma, Jun Yang, Luan Zhang, Ruochen Zhou, Xudong Li, Yan Xu, Yizhou Jin, Yuhang Tian, Zhengyu Chen.

Figure 1
Figure 1. Figure 1: Overview of HarnessCompass. they trace back to a single cause. Current methods give the meta agent broad freedom to change the harness while feed￾ing it only external outcomes and letting it revise everything at once. Unrestricted edits invite overfitting, outcome-only evidence invites misattribution, and simultaneous edits in￾vite interference. The core challenge, therefore, is not har￾ness evolution itse… view at source ↗
Figure 2
Figure 2. Figure 2: Pass@1 across evolution iterations on the SWE-Bench Verified Sample. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-repository Pass@1 on the full 500-task SWE-Bench Verified benchmark for GPT-5.4 and Claude-Sonnet-4.6. Each model compares the bash-only seed against its best evolved HarnessCompass harness. Repositories are ordered by task count, from django (231) to pallets (1); the rightmost group shows the task-level average Pass@1 across all 500 tasks. R 3 preserves complementary gains while removing redun￾dant or… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 3 canonical work pages

  1. [1]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

  2. [2]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

  3. [3]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  4. [4]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

  5. [5]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    doi:10.1016/s0020-7373(84)80003-6 1984

  6. [6]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

  7. [7]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

  8. [8]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

  9. [9]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

  10. [10]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    2023

  11. [11]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

  12. [12]

    2023 , eprint =

    Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , eprint =

    2023

  13. [13]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =. 2024 , eprint =

    2024

  14. [14]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle =

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle =. 2024 , eprint =

    2024

  15. [15]

    and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and Tran, Hoang H

    Wang, Xingyao and Li, Boxuan and Song, Yufan and Xu, Frank F. and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and Tran, Hoang H. and Li, Fuqiang and Ma, Ren and Zheng, Mingzhang and Qian, Bill and Shao, Yanjun and Muennighoff, Niklas and Zhang, Yizhe and Hui, Binyuan and Lin, Junyang and Brennan, Robe...

    2025

  16. [16]

    Zhang, Jenny and Hu, Shengran and Lu, Cong and Lange, Robert and Clune, Jeff , journal =. Darwin. 2025 , eprint =

    2025

  17. [17]

    2025 , eprint =

    Xia, Chunqiu Steven and Wang, Zhe and Yang, Yan and Wei, Yuxiang and Zhang, Lingming , journal =. 2025 , eprint =

    2025

  18. [18]

    arXiv preprint arXiv:2507.21046 , year =

    A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence , author =. arXiv preprint arXiv:2507.21046 , year =. 2507.21046 , archivePrefix =

    Pith/arXiv arXiv

  19. [19]

    International Conference on Learning Representations (ICLR) , year =

    Large Language Models as Optimizers , author =. International Conference on Learning Representations (ICLR) , year =. 2309.03409 , archivePrefix =

    Pith/arXiv arXiv

  20. [20]

    2024 , eprint =

    Yuksekgonul, Mert and Bianchi, Federico and Boen, Joseph and Liu, Sheng and Huang, Zhi and Guestrin, Carlos and Zou, James , journal =. 2024 , eprint =

    2024

  21. [21]

    International Conference on Learning Representations (ICLR) , year =

    Automated Design of Agentic Systems , author =. International Conference on Learning Representations (ICLR) , year =. 2408.08435 , archivePrefix =

    Pith/arXiv arXiv

  22. [22]

    2025 , eprint =

    Zhang, Jiayi and Xiang, Jinyu and Yu, Zhaoyang and Teng, Fengwei and Chen, Xiong-Hui and Chen, Jiaqi and Zhuge, Mingchen and Cheng, Xin and Hong, Sirui and Wang, Jinlin and Zheng, Bingnan and Liu, Bang and Luo, Yuyu and Wu, Chenglin , booktitle =. 2025 , eprint =

    2025

  23. [23]

    arXiv preprint arXiv:2502.06855 , year =

    Self-Supervised Prompt Optimization , author =. arXiv preprint arXiv:2502.06855 , year =. 2502.06855 , archivePrefix =

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2502.11705 , year =

    W. arXiv preprint arXiv:2502.11705 , year =. 2502.11705 , archivePrefix =

    Pith/arXiv arXiv

  25. [25]

    2025 , eprint =

    Yue, Murong and Liu, Zhiwei and Yang, Liangwei and Zhang, Jianguo and Chen, Haolin and Yao, Ziyu and Savarese, Silvio and Xiong, Caiming and Heinecke, Shelby and Wang, Huan , journal =. 2025 , eprint =

    2025

  26. [26]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year =

    A Systematic Survey of Automatic Prompt Optimization Techniques , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year =. 2502.16923 , archivePrefix =

    arXiv 2025

  27. [27]

    arXiv preprint arXiv:2604.25850 , year =

    Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses , author =. arXiv preprint arXiv:2604.25850 , year =. 2604.25850 , archivePrefix =

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2607.12227 , year =

    Rethinking the Evaluation of Harness Evolution for Agents , author =. arXiv preprint arXiv:2607.12227 , year =. 2607.12227 , archivePrefix =

    Pith/arXiv arXiv

  29. [29]

    and Shaw, Alexander G

    Merrill, Mike A. and Shaw, Alexander G. and Carlini, Nicholas and Li, Boxuan and others , journal =. 2026 , eprint =

    2026

  30. [30]

    International Conference on Learning Representations (ICLR) , year =

    Mialon, Gr\'. International Conference on Learning Representations (ICLR) , year =. 2311.12983 , archivePrefix =

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2606.22388 , year =

    Liu, Jiayu and Lin, Qihan and Qian, Cheng and Wang, Rui and Acikgoz, Emre Can and Yang, Xiaocheng and Wang, Zhenhailong and Chen, Xiusi and Ji, Heng and Hakkani-T\". arXiv preprint arXiv:2606.22388 , year =. 2606.22388 , archivePrefix =

    Pith/arXiv arXiv

  32. [32]

    2025 , howpublished =

    2025

  33. [33]

    Bui, Nghi D. Q. and others , journal =. Building Effective. 2026 , eprint =

    2026

  34. [34]

    arXiv preprint arXiv:2607.03691 , year =

    Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality , author =. arXiv preprint arXiv:2607.03691 , year =. 2607.03691 , archivePrefix =

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2603.28052 , year =

    Meta-Harness: End-to-End Optimization of Model Harnesses , author =. arXiv preprint arXiv:2603.28052 , year =. 2603.28052 , archivePrefix =

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2606.09498 , year =

    Self-Harness: Harnesses That Improve Themselves , author =. arXiv preprint arXiv:2606.09498 , year =. 2606.09498 , archivePrefix =

    Pith/arXiv arXiv

  37. [37]

    Evolving Excellence: Automated Optimization of

    Evers-Hood, Will and Nair, Hema and Brookes, Paul and Voskanyan, Vardan and Giavrimis, Rafail and Truscott, Matthew and Ilieva, Mina and Pavlou, Chrystalla and Staicu, Alexandru and Adham, Manal and Gong, Jingzhi and Zhang, Kejia and Fedoseev, Matvey and Sharma, Vishal and Bauer, Roman and Wang, Zheng and Jie, Wei and Xu, Tianhua and Constantin, Aurora an...

    2025

  38. [38]

    arXiv preprint arXiv:2605.09931 , year=

    PruneTIR: Inference-Time Tool Call Pruning for Effective yet Efficient Tool-Integrated Reasoning , author=. arXiv preprint arXiv:2605.09931 , year=

    Pith/arXiv arXiv

  39. [39]

    , author=

    Detecting Hallucination in Large Language Models Through Deep Internal Representation Analysis. , author=. IJCAI , pages=

  40. [40]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    ActiShade: Activating Overshadowed Knowledge to Guide Multi-Hop Reasoning in Large Language Models , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  41. [41]

    arXiv preprint arXiv:2509.18970 , year=

    Llm-based agents suffer from hallucinations: A survey of taxonomy, methods, and directions , author=. arXiv preprint arXiv:2509.18970 , year=

    arXiv

  42. [42]

    HEAL : An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models

    Chakraborty, Trishna and Ghosh, Udita and Zhang, Xiaopan and Niloy, Fahim Faisal and Dong, Yue and Li, Jiachen and Roy-Chowdhury, Amit and Song, Chengyu. HEAL : An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-...

    work page doi:10.18653/v1/2025.findings-emnlp.1158 2025

  43. [43]

    I nfi A gent: An Infinite-Horizon Framework for General-Purpose Autonomous Agents

    Yu, Chenglin and Wang, Yuchen and Wang, Songmiao and Yang, Hongxia and Ming, Li. I nfi A gent: An Infinite-Horizon Framework for General-Purpose Autonomous Agents. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findings-acl.1787

    work page doi:10.18653/v1/2026.findings-acl.1787 2026

  44. [44]

    Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning

    Liang, Qiao and Zhu, Yuke and Ge, Chao and Yang, Lei and Shen, Ying and Zheng, Bo and Guo, Sheng. Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Papers). 2026. doi:10.18653/v1/2026.acl-long.504

    work page doi:10.18653/v1/2026.acl-long.504 2026

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.