Artificial Intelligence · arXiv:2609.20804 v1 · submitted 17 Sep 2026 · 41 references
An Empirical Study of Harness Design for Coding Agents
The paper says
“Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear.”
more of the abstract
To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 20 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: minor revision.
Solid, carefully done empirical study of coding harness components; the conditional claims for planning and action space are less fully tested than for context management, but this is transparently acknowledged and the paper is still a useful contribution. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 41 references say.
The paper says, at §3.2, §6 (Limitations), Tables 3–4
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The paper's central claim of conditionality on 'context-window budget' is fully supported only for context management, because planning and action-space ablations are performed exclusively at T4/128k (as acknowledged in §6). The abstract states that 'each component’s effectiveness depends on ... resource budget,' but for planning and action space the budget dependence is not empirically established.
more of objection 1
The authors should either add cross-budget ablations (if feasible) or explicitly scope the budget-conditionality claim to context management in the abstract and conclusions. The current Limitations paragraph is clear but the main text could still mislead a casual reader into over-generalizing.
The paper says, at §3.1, Tables 3–4 (Terminal-Bench)
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
Terminal-Bench contains only 89 tasks, so many contrasts do not reach significance under the paired McNemar test (as noted in §6). The paper's approach of resting conclusions on consistent directions across models and budgets is reasonable and transparent. However, the authors should verify that the trajectory-level analysis (Figures 8–9 and Appendix C.4) is based on all runs, not only on resolved ones, to avoid survivorship bias in the behavioral summaries.
more of objection 2
The description in §4 states these figures use T4/128k; it would be helpful to confirm that the same phase proportions hold when restricting to the subset where planning/action-space comparisons are made.
6 minor objections
The paper says, at Figure 5(b)
Pith says: minor objection 1 of 6
The y-axis uses a logarithmic scale. Please state this explicitly in the caption for clarity.
The paper says, at Tables 3–4
Pith says: minor objection 2 of 6
The definition of 'bold' (highest success rate per model) is clear, but for Nemotron-3 30B on SWE-Bench at 128k, the bold appears on T2 (26.0%). Please double-check that this is intentional and not a typesetting artifact, given that T0 (24.8%) and T4 (25.2%) are close.
show 4 more
The paper says, at §2.2, Table 1
Pith says: minor objection 3 of 6
The bash-only condition also removes web_fetch. The text states 'the bash-only setting removes predefined file, search, and web tools.' It would be helpful to note explicitly that web_fetch is removed because SWE-Bench tasks may leak ground-truth patches via search (stated in §2.2). This is a good design choice, but the reader may wonder why web access is removed in the ablation.
The paper says, at Table 5
Pith says: minor objection 4 of 6
For Nemotron-3 120B on Terminal-Bench, the percentage changes are negative (turns -15.0%, tool calls -23.1%). In Table 11, the median turns for T4 is 34 and for T4 w/o plan is 30, which yields a decrease of about 12%. The discrepancy between -15% and -12% is small and likely due to mean vs. median. Please check whether Table 5 reports mean or median changes and clarify in the table caption or text.
The paper says, at Appendix C.2 (human validation)
Pith says: minor objection 5 of 6
The validation shows high agreement (Cohen's κ ~0.93). This is excellent. Please provide the exact human–judge agreement percentages and κ values for each benchmark and phase type in a supplementary table rather than only in prose, to aid reproducibility.
The paper says, at General
Pith says: minor objection 6 of 6
The paper uses OpenRouter pricing. Because API prices change over time, the absolute cost figures may not be reproducible. It would be helpful to also report cost in terms of input/output token counts or a normalized cost metric (e.g., cost per 1M tokens) so that future readers can apply current pricing.
What would settle it
Run the planning ablation (Nemotron-3 30B and 550B) under T0/32k and T4/32k on SWE-Bench. If the success-rate gap between planning on/off for Nemotron-3 30B shrinks from +11.6pp (T4/128k) to <5pp at 32k, the planning effect is budget-dependent. Run the bash-only ablation for Nemotron-3 550B under T4/32k; if its success advantage over the full tool set (currently +3.6pp at T4/128k) disappears or becomes negative, the action-space crossover is budget-dependent.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 20 Sep 2026
- Reader holds up
- Skeptic partly agrees: says it holds up only with conditions Planning and action-space ablations limited to T4/128k, leaving dependence on context budget untested for those components.
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a methodologically serious paper that fills a real gap. Prior work evaluated coding harnesses as monoliths or ablated only one component. This paper runs controlled component-level comparisons across four models and two benchmarks, with 176 settings, statistical testing with FDR control, and trajectory analysis. The main findings are backed by the evidence: context management is most valuable under tight budgets, the staged elision-then-summarization policy (T4) gives the best cost-accuracy trade-off, recall is nearly never used, planning shifts from accuracy scaffold for weak models to cost saver for strong ones, and the preferred action space depends on both model bash proficiency and task shell-centeredness. The trajectory analysis adds mechanistic insight beyond just success rate.
The soft spot flagged by the stress-test is real and load-bearing. The conditionality on context-window budget—the paper's most emphasized claim—is fully tested only for context management. Planning and action-space ablations are run exclusively at T4/128k. The authors acknowledge this in the limitations, but it means that the reported shifts in planning's role or the bash-only crossover are not demonstrated to hold under tighter budgets. The stress-test gives a plausible counterexample: at 32k, overflow truncation could erase any planning benefit even for weak models, and bash-only's denser outputs could reverse the cost advantage. Without evidence, we cannot know. The paper would have been stronger if it had at least spot-checked one lower budget for planning and action space.
Two minor points. Terminal-Bench has only 89 tasks; many contrasts are underpowered, and the paper leans on consistency of direction rather than per-cell significance. That's acceptable given the transparency. Also, the action-space comparison is a bundled change (tool availability + prompts + state tracking), so it does not isolate tool count alone. The authors note this.
Overall, this is honest, carefully executed work. The limitations are in plain view. The paper deserves a serious referee and will be useful to anyone building a coding agent. I would cite it for the context-management comparisons. For planning and action space, I would use the results with the caveat that they are only established at one budget.
Recommendation: send to peer review.
The strongest objection
The central claim that harness-component effectiveness is conditional on model capability, budget, and task type is only fully tested for context management. Planning and action-space ablations are performed exclusively under T4/128k, as the authors acknowledge in Limitations (§6). This is load-bearing because the reported patterns (planning as scaffold for weak models, cost saver for strong models; predefined tools helping bash-weak models) may themselves shift under tighter budgets or different management strategies. For example, at 32k, where context overflow dominates (T0 overflow rate 78.7% on SWE-Bench), the benefit of planning could diminish even for weak models because trajectories are truncated before planning can act. Similarly, bash-only for Nemotron-3 550B produces denser shell commands but also longer outputs; under a 32k budget the advantage could reverse if outputs push the window. Without evidence at other budgets, the claimed conditionality on context-window budget is unestablished for planning and action space.
The reader flagged external validity as the weakest assumption. I find a stronger internal limitation: the factorial design is incomplete for the two non-context components. The paper's central claim asserts conditional effects on context-window budget for planning and action space, yet those components are only evaluated under one budget-management pair. This gap directly affects the strength of the core argument, not just its generalizability. A conditional recommendation acknowledges that the main results for context management are well-supported (T0–T4 across four budgets), but the planning and action-space findings require validation under other budgets before the conditionality claim can be accepted without qualification.
The referee's report
This study is timely and methodologically rigorous. Coding harnesses are typically evaluated as monolithic systems, conflating multiple component effects. The authors' modular ablation design, controlled context-window sweep, use of four models spanning two families and three sizes, paired statistical tests with FDR control, and detailed trajectory analysis all set a high standard for empirical work on agent scaffolding. The findings have direct practical implications: they provide diagnostics for selecting harness components based on model capability, resource budget, and task type rather than adopting a one-size-fits-all design. The trajectory-level behavioral analysis (e.g., planning's differential effect on early termination vs. post-edit verification, action-space effects on granularity) is especially valuable for understanding mechanisms. The paper's transparency about limitations and its reliance on consistent patterns rather than over-interpretation of individual nonsignificant cells strengthen credibility.
Pith's reading of the paper (a machine's summary, not the authors')
The paper demonstrates that no single harness component universally improves coding agent performance. Context management prevents context-overflow failures and is critical under tight context-window budgets but offers diminishing returns as the window expands. Planning acts as an accuracy scaffold for weaker models, increasing success at higher cost, while for stronger models it primarily reduces cost by cutting redundant verification. The action space choice depends on the model's bash proficiency: predefined tools help models weak at bash, whereas bash-only interfaces lower cost for models that can compose complex shell commands, especially on command-line-centric tasks. These conditional effects mean harnesses should be tailored to the target model, budget, and task rather than adopted as a fixed design.
What carries the argument. The modular coding harness with a fixed ReAct execution loop, five context-management tiers (T0–T4: from no management to staged elision then summarization), a persistent planning scaffold via the update_plan tool, and two action spaces (predefined workspace tools vs. bash-only interface). The T4 strategy—applying cheap rule-based elision of stale observations before more expensive LLM-based summarization—is the central mechanism that delivers the lowest cost at comparable accuracy.
Core claim. The central claim is that each harness component's effectiveness is conditional, not universal. Through a systematic ablation of 176 settings across four models and two benchmarks, the paper establishes that: (1) context management's benefit stems almost entirely from avoiding context-overflow failures, so it matters most under tight budgets; (2) planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models; (3) predefined tools benefit models weaker at bash, while bash-only reduces cost for bash-capable models; and (4) lossless recall of elided content adds machinery that models rarely use and yields no accuracy gain. The paper also shows that a staged policy combining rule-based elision before LLM-based summarization (T4) achieves the best cost–accuracy trade-off among the context-management strategies.
The premise it rests on. The observed conditional effects are assumed to generalize beyond the two model families (Nemotron-3 and Mistral-Medium-3.5), the specific harness implementations (one planning prompt, one threshold-based context policy, bundled action-space changes), and the two task domains (Python-only SWE-Bench and small-scope Terminal-Bench), but the paper explicitly acknowledges this as a limitation.
If the paper is right (Pith's inferences, not the authors')
Harness designers can disable planning for strong models to reduce cost and enable it for weak models to improve success rates.
Context management should be tuned to the target context-window budget; at 128k windows, even no management may suffice for many settings, but at 32k it is essential.
Models with strong bash proficiency can use a bash-only interface to achieve lower cost, especially on shell-centric tasks, while models with weaker bash need predefined tools.
Lossless context management via recall_event adds complexity with no measurable accuracy gain; elision alone is sufficient, suggesting future harnesses should prefer cheap lossy compression.
The conditional effects imply that coding-agent evaluations should report model capability, context budget, and task type to enable fair comparison across harnesses.
As model capability continues to improve, harness components that currently scaffold performance may become unnecessary or even detrimental, motivating dynamic harnesses that adapt component usage to the model's observed skill level.
The rare invocation of recall_event suggests that lossless context management is not worth its engineering overhead for current models, but this could change if future models learn to actively retrieve historical information.
The task-type dependence (shell-centeredness) indicates that coding benchmarks should be categorized by interaction style—repository editing vs. command-line tasks—since the optimal action space differs between them.
The finding that T4 (elision before summarization) dominates other strategies suggests a general design principle for long-horizon agents: apply cheap deterministic compression first, and reserve expensive LLM-based compaction only when necessary.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.79
“These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.”
How the main claim is stated: asserted 0.97
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.34 · human participants 0.03 · controlled comparison 0.56 · randomized assignment 0.02 · states quantitative result 0.84 · compares with prior results 0.52 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. | gap 0.79 | 0.05 | 0.03 | asserted 1.00 |
| To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. | method 0.75 | 0.92 | 0.02 | asserted 0.92 |
| Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. | method 0.98 | 0.96 | 0.02 | asserted 0.96 |
| We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. | result 1.00 | 0.96 | 0.18 | asserted 1.00 |
| (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. | result 1.00 | 0.96 | 0.17 | asserted 1.00 |
| (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. | result 1.00 | 0.96 | 0.11 | asserted 1.00 |
| (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. | result 1.00 | 0.94 | 0.11 | asserted 1.00 |
| Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. | interpretation 0.97 | 0.93 | 0.12 | asserted 1.00 |
| These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components. | interpretation 0.79 | 0.62 | 0.22 | asserted 0.97 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,115 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
References (41)
- [1] Swe-bench: Can language models resolve real-world github issues? , author=. International Conference on Learning Representations , volume= on Pith
- [2] Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. International Conference on Learning Representations , volume= on Pith
- [3] Swe-agent: Agent-computer interfaces enable automated software engineering , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [4] Openhands: An open platform for ai software developers as generalist agents , author=. International Conference on Learning Representations , volume= on Pith
- [5] Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases , author=. arXiv preprint arXiv:2512.10398 , year= on PitharXiv
- [6] Qwen3-coder-next technical report , author=. arXiv preprint arXiv:2603.00729 , year= on PithPith's reviewarXiv
- [7] Agentless: Demystifying llm-based software engineering agents , author=. arXiv preprint arXiv:2407.01489 , year= on PithPith's reviewarXiv
- [8] More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffolding , author=. arXiv preprint arXiv:2605.05716 , year= on PithPith's reviewarXiv
- [9] Efficient streaming language models with attention sinks , author=. International Conference on Learning Representations , volume= on Pith
- [10] Llmlingua: Compressing prompts for accelerated inference of large language models , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages= on Pith
- [11] Memgpt: Towards llms as operating systems , author=. arXiv preprint arXiv:2310.08560 , year= on PithPith's reviewarXiv
- [12] Generative agents: Interactive simulacra of human behavior , author=. Proceedings of the 36th annual acm symposium on user interface software and technology , pages= on Pith
- [13] Recursively summarizing books with human feedback , author=. arXiv preprint arXiv:2109.10862 , year= on PithPith's reviewarXiv
- [14] Resum: Unlocking long-horizon search intelligence via context summarization , author=. arXiv preprint arXiv:2509.13313 , year= on PitharXiv
- [15] Acon: Optimizing context compression for long-horizon llm agents , author=. arXiv preprint arXiv:2510.00615 , year= on PithPith's reviewarXiv
- [16] Sculptor: Empowering llms with cognitive agency via active context management , author=. International Conference on Learning Representations , volume= on Pith
- [17] Memory as action: Autonomous context curation for long-horizon agentic tasks , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages= on Pith
- [18] Agentarch: A comprehensive benchmark to evaluate agent architectures in enterprise , author=. arXiv preprint arXiv:2509.10769 , year= on PitharXiv
- [19] Self-refine: Iterative refinement with self-feedback , author=. Advances in neural information processing systems , volume= on Pith
- [20] Large language models cannot self-correct reasoning yet , author=. International conference on learning representations , volume= on Pith
show 21 more
- [21] To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning , author=. International Conference on Learning Representations , volume= on Pith
- [22] Do advanced language models eliminate the need for prompt engineering in software engineering? , author=. ACM Transactions on Software Engineering and Methodology , volume=. 2026 , publisher= on Pith
- [23] Emergent abilities of large language models , author=. arXiv preprint arXiv:2206.07682 , year= on PithPith's reviewarXiv
- [24] Beyond resolution rates: Behavioral drivers of coding agent success and failure , author=. arXiv preprint arXiv:2604.02547 , year= on PithPith's reviewarXiv
- [25] React: Synergizing reasoning and acting in language models , author=. arXiv preprint arXiv:2210.03629 , year= on PithPith's reviewarXiv
- [26] Kimi k2. 5: Visual agentic intelligence , author=. arXiv preprint arXiv:2602.02276 , year= on PithPith's reviewarXiv
- [27] Glm-5: from vibe coding to agentic engineering , author=. arXiv preprint arXiv:2602.15763 , year= on PithPith's reviewarXiv
- [28] 2026 , howpublished= on Pith
- [29] Claude Code , author=. 2025 , howpublished= on Pith
- [30] Codex , author=. 2025 , howpublished= on Pith
- [31] opencode , author=. 2025 , howpublished= on Pith
- [32] Coder: Issue resolving with multi-agent and task graphs , author=. arXiv preprint arXiv:2406.01304 , year= on PithPith's reviewarXiv
- [33] Masai: Modular architecture for software-engineering ai agents , author=. arXiv preprint arXiv:2406.11638 , year= on PithPith's reviewarXiv
- [34] Inside the scaffold: A source-code taxonomy of coding agent architectures , author=. arXiv preprint arXiv:2604.03515 , year= on PithPith's reviewarXiv
- [35] Autocoderover: Autonomous program improvement , author=. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages= on Pith
- [36] Nvidia nemotron 3: Efficient and open intelligence , author=. arXiv preprint arXiv:2512.20856 , year= on PithPith's reviewarXiv
- [37] LCM: Lossless Context Management , author=. arXiv preprint arXiv:2605.04050 , year= on PithPith's reviewarXiv
- [38] LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception , author=. arXiv preprint arXiv:2606.30005 , year= on PithPith's reviewarXiv
- [39] Same Model, Different Harness: Different Coding-Agent Results , author=. arXiv preprint arXiv:2608.26218 , year= on PithPith's reviewarXiv
- [40] Executable code actions elicit better llm agents , author=. arXiv preprint arXiv:2402.01030 , year= on PithPith's reviewarXiv
- [41] Codeplan: Repository-level coding using llms and planning , author=. Proceedings of the ACM on Software Engineering , volume=. 2024 , publisher= on Pith
Cited by 11 Pith papers
Reviewed papers on Pith that cite this one. OpenAlex reports about 16 citations worldwide. The full citation record
- Hidden in Plain Sight: an Empirical Study about The Instruction Stacks of Coding Agents cs.SE · Oct 2026
- Agent Behavior Maps: Locating Agents by What They Would Do, Without Running Them cs.LG · Oct 2026
- When the harness decides success: state that cannot be re-observed cs.AI · Oct 2026
- SCARCE: Emergent Institutions in Long-Horizon Multi-Agent Coding Systems cs.MA · Oct 2026
- Anatomy of an Agent Harness: Decisions, Invariants and Corner Cases cs.CL · Nov 2026
- Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses cs.SE · Oct 2026
- Before the First Edit? Measuring When Coding-Agent Guidance Actually Arrives cs.SE · Oct 2026
- Honesty Testing of LLMs and AI Agents: A Vision for Software Engineering cs.SE · Oct 2026
- ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience? cs.LG · Oct 2026
- MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks cs.CL · Oct 2026
- Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done? cs.SE · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment