Artificial Intelligence · arXiv:2609.26048 v1 · submitted 22 Sep 2026 · 62 references
FIRE: Failure-Informed Runtime Engineering for Reliable Language-Model Agents
The paper says
“Language-model agents often reach a working solution and then fail to consistently deliver it.”
more of the abstract
We study runtime policies: targeted natural-language instructions and action denials applied by the agent harness at states that preceded observed failures, without changing model weights or the user prompt. With this, keeping capability constant, we observe a meaningful unlock in delivered reliability. Across the complete 87-task Terminal-Bench 2.1 suite, with two attempts per task, policies increase repeated success (pass^2) in all three GPT-5.6 tiers: 50.6% to 54.0% for Luna, 55.2% to 60.9% for Terra, and 64.4% to 73.6% for Sol. Sol's best-of-two success changes by 1.2 points while repeated success rises by 9.2, showing that policies chiefly convert reachable solutions into dependable delivery. We further cover 14 tasks under Terra's frozen portfolio. Policy-guided Terra reaches 71.4%, compared with 64.3% for unassisted Sol, at about half the cost, demonstrating how engineering around models could unlock dependability for a use case. To isolate the mechanism we run a randomized five-arm experiment: real policies reach 61% on eligible tasks, versus 39% without a policy, 36% with a timing-matched sham, and 39 to 43% with generic verification or reconsideration. The intended corrective behavior appears in 22 of 24 coded policy attempts, against at most 14 in any other arm. Runtime policies are therefore a practical reliability layer: they make capabilities an agent already possesses substantially more repeatable.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 24 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A careful, honest systems paper whose real contribution is the sham-controlled isolation of policy content from interruption timing; the central causal claim is plausible but not statistically settled, and the authors themselves scope it to the development distribution. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 62 references say.
The paper says, at §3, §4, Limitations [b154]
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The central causal panel may re-measure the development set rather than evaluate on a held-out population. Per §3, the portfolio was built by inspecting failed attempts, screening candidates 'on positive and nearby-negative tasks,' revising or retiring candidates over 13 evaluated Terra configurations, and auditing routing 'over all development tasks.'
more of objection 1
The 14 eligible panel tasks are then described as coming 'from the development population' (Limitations, [b154]). Because eligibility predicates and trigger conditions were derived from, and screened against, the failure archetypes of that same population, the primary contrast (real policy minus sham) can be inflated by policy selection on the evaluation distribution, even though the portfolio was frozen and hashed before the panel ran. The paper honestly states that the panel establishes a content effect 'on this distribution, not generalization,' but this does not fully discharge the concern: the problem is not only limited external generalization, it is that the 14 eligible tasks may have been part of the tuning/screening loop, making the panel a training-set re-measurement. Please either (i) show explicitly that none of the 14 eligible tasks was used to screen, revise, or retire policies, or (ii) add a held-out panel drawn from a population not used in development; and correspondingly qualify the abstract's unqualified sentence [b9] and the §1/§7 conclusions.
The paper says, at §5.1, Table 3, contribution bullet [b35]
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The prespecified primary contrast is real policy minus sham on eligible tasks: +25.0 points, 95% CI [7.1, 46.4], p=0.061. The paper reports this honestly as 'a large effect with a wide interval, not a settled one,' but the section heading ('Policy text, not timing or generic checking, drives the gain') and the contributions bullet present the content-attribution claim more strongly than the primary test supports.
more of objection 2
The baseline contrast (p=0.032) and the mechanism funnel are secondary evidence, and the paper does not correct for multiple secondary contrasts. Because the attribution claim is the paper's main causal contribution, the manuscript should lead with the fact that the confirmatory test was suggestive rather than significant at the 0.05 level, and should explicitly state that the attribution claim is supported but not confirmed by the panel alone.
The paper says, at §4, sham-arm exception for deny rules
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The sham arm does not test content versus denial for the two deny rules: the paper states that 'for deny rules the sham instructs instead of denying.' This means that for deployment-preservation and database-evidence families, the real-versus-sham comparison varies both the message content and the enforcement mechanism, so the conclusion that 'policy content, not timing' drives the gain is not established for these families.
more of objection 3
The exception is disclosed and fixed in advance, and it may be unavoidable, but the attribution claim should be scoped to the instruct-type rules, or a separate analysis should address whether denial itself, rather than the text of the denial, contributes to the effect.
5 minor objections
The paper says, at Figure 1
Pith says: minor objection 1 of 5
The legend and in-figure counts ('Real policy 25/28 0/32', 'Triggered sham 26/28 0/32', 'Always verify 28/28 29/32') are difficult to read because the two columns are not labeled inside the figure; please label the fired counts for eligible and silent tasks explicitly in the caption or legend.
The paper says, at Table 3
Pith says: minor objection 2 of 5
The column header 'p' is used for both the eligible and all-task comparisons, but the silent cohort has no p-value reported; the formatting should make clear that p-values are not computed or are omitted for that cohort, and which contrasts are secondary.
show 3 more
The paper says, at §5.2, Table 4
Pith says: minor objection 3 of 5
The behavior markers are described as deterministic extractors applied identically to every arm, which is a strength, but the extraction rules themselves are not specified in the main text or in the visible appendix material. Please include the extractor definitions or point to the exact artifact paths, since the 22/24 versus at-most-14 contrast is a load-bearing mechanism diagnostic.
The paper says, at §5.4
Pith says: minor objection 4 of 5
The complete-suite comparison is called an 'experiment,' but unlike the five-arm panel it is not randomized or sham-controlled within the same scheduled replicate. Consider describing it as a paired observational comparison or 'complete-suite evaluation' to keep the causal vocabulary aligned with the design.
The paper says, at Limitations [b156]
Pith says: minor objection 5 of 5
The assumption that the provider route did not materially change during the evaluation window is acknowledged but not verifiable; since the panel's five arms ran concurrently, this is less threatening for §5.1, but the cross-date complete-suite comparisons in §5.4 would be stronger if the paper stated whether baseline and policy cells for each tier were run in the same time window.
What would settle it
Hold out eligible tasks before authoring begins: reserve at least 14 Terminal-Bench tasks (or, better, archetype-matched tasks from a second benchmark such as τ-bench or OSWorld) whose trajectories are never inspected and never screened during policy authoring, then freeze the hashed Terra portfolio plus the triggered sham. Run the same two-attempt protocol and analysis on the held-out set, and compare the primary contrast (real minus sham) on held-out eligible tasks. If the 95% bootstrap lower bound stays positive and the point estimate replicates (roughly +20 to +25 points), the selection concern is retired and the general reliability claim stands; if the contrast collapses toward zero while the within-panel contrast stays at +25, the panel effect is a tuning artifact on the development population.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 24 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
What's actually new here is the measurement discipline, not the idea of runtime rules. AgentSpec and the guardrail literature already have rule layers, and the paper cites them fairly. The genuine step forward is the timing-matched sham control: same eligibility, same firing events, generic text. That lets you separate what a policy says from the fact that it interrupts the agent. The funnel data backs it up — the sham fires as often as the real policy, produces the behavior barely more often than baseline, and changes nothing. That is the contribution, and it's a real one. The pass2 versus pass@2 framing is also useful: Sol's repeated success jumps 9.2 points while reach moves 1.2, which is a clean way to show you're converting reachable solutions into dependable delivery rather than expanding capability.
The design is genuinely careful. Preregistered contrasts, hashed and frozen portfolios before the panel, deterministic behavior extractors applied identically to every arm (so blind-by-construction, no LLM judge), and honest reporting of the interval alongside the p-value rather than cherry-picking. The authors flag their own limitations clearly: the panel comes from the development population, the primary contrast is p=0.061 on 14 tasks, route stability is unverifiable, no leave-one-family-out ablation. I don't think the stress-test concern lands as a fatal flaw, because it's the very limitation the authors state in [b154]: the panel establishes a content effect on this distribution, not generalization. They say that in plain terms.
The soft spots are proportionate to what's claimed. The headline in [b9] talks about capabilities generally becoming "substantially more repeatable," but the causal evidence only covers 14 tasks from the same population the policies were screened on. That's a real gap between the abstract's general framing and the demonstrated scope, even though the limitations section walks it back. The sham's deny-rule exception — instructing instead of denying — is a minor deviation, but it was fixed before execution and it's disclosed. The 13 evaluated configurations before freeze limit overfitting capacity, but regex routing and trigger thresholds are high-dimensional, and 14 tasks leave little room to see selection effects.
Bottom line: this is an accept-shaped paper with specific, addressable concerns. The authors should add a held-out panel or explicitly scope the abstract's claim, and ideally an independent route-stability check. It deserves a serious referee, not a desk reject. The methodology is worth citing on its own.
The strongest objection
Per §3, the portfolio is built by inspecting failed attempts on the benchmark, screening candidates 'on positive and nearby-negative tasks,' and revising or retiring them under new identities across 13 evaluated Terra configurations; the eligibility predicates are lexical regexes whose routing was audited 'over all development tasks.' The causal panel then measures effect on 'the 14 tasks eligible for the Terra portfolio,' which come 'from the development population' (Limitations, [b154]). This is the load-bearing gap: both the scope predicate (which tasks fire) and the trigger predicate (which trajectory states fire) were selected on the exact failure archetypes of the population that supplies the evaluation tasks. The sham control genuinely isolates content from timing within that distribution — the real policy elicits the registered behavior in 22/24 coded attempts vs at most 14 in any other arm — but a repair kit tuned to a benchmark's recurring failure modes will show a content effect on the very tasks it was built for. So the panel supports 'a content effect on this distribution,' which the authors explicitly state, but not by itself the headline [b9] that policies generally 'make capabilities an agent already possesses substantially more repeatable.' The search was modest (13 configurations), which limits overfit capacity, but regex routing and trigger thresholds are high-dimensional, and 14 eligible tasks leave little room to observe selection effects. What is missing is evidence that the content advantage survives on tasks whose failures were never inspected and never screened.
The reader's weakest assumption — that the 13-configuration search over the development population did not overfit the evaluation distribution — is exactly the load-bearing concern, and I agree with it. I sharpened it: the concern is selection-based as much as statistical, because both the scope (eligibility regex over task descriptions) and the trigger (runtime regex over command history) of each policy were fitted to failure archetypes present in the population that also supplies the 14 eligible panel tasks ([b154]). The sham arm and the blind-coded behavior funnel are genuinely strong controls for timing and generic text within the tuned distribution, and I credit that: policy content does causal work on these tasks. The unresolved step is generalization beyond the tuned population, which the paper itself restricts ('a content effect on this distribution, not generalization') while stating the headline claim in general terms ([b9]). The 87-task complete suite is supporting evidence only: not randomized, cross-date, and reliant on route stability the authors cannot verify ([b156]); Sol's pass∧2-vs-pass@2 split is consistent with the conversion story but does not bear the selection burden. A held-out population test would settle the question and is cheap to run given the artifact release. None of this moves the verdict: the preregistration, hashes, frozen sources, and deterministic blind extractors are real strengths, and the authors flag the main weaknesses themselves ([b154], [b156], [b159]). I keep CONDITIONAL, with the conditions being held-out evaluation and a route-stability check, rather than overturning the paper.
The referee's report
If the central claim holds, the paper is a valuable systems contribution: it gives a concrete harness layer that improves delivery reliability without retraining, and it demonstrates a more careful attribution design than most guardrail evaluations, with a timing-matched sham, hashed and frozen portfolios, deterministic blind behavior extractors, task-level inference, and openly reported confidence intervals alongside p-values. The artifact release and preregistration-style commitments are genuine strengths. However, the headline generalization is currently bounded by the fact that the portfolio was developed and screened on the same development population that supplies the panel's eligible tasks, and the prespecified primary contrast does not reach conventional significance. The paper is therefore significant as a well-controlled demonstration on the distribution where the policies were built, and as a methodological template, but it does not yet establish the broad 'capabilities an agent already possesses substantially more repeatable' claim as a general property.
Pith's reading of the paper (a machine's summary, not the authors')
Language-model agents often reach a working solution and then lose it during delivery: a server dies with the agent's shell, a database is mutated before its evidence is copied, or a late formatter rewrites protected bytes. The paper claims that this failure is largely procedural, and that a lightweight harness layer—short natural-language instructions and action denials injected at states that preceded observed failures, without retraining or changing the user prompt—can make capabilities the model already possesses substantially more repeatable. The central evidence is a randomized five-arm panel in which the real policy portfolio beats a timing-matched sham by 25 points on eligible tasks, while generic verify and reconsider instructions fire on nearly every attempt yet do no better than baseline, showing that what the policy says, not the fact of interruption, drives the gain. If correct, this reframes part of agent reliability as a systems property rather than a model property.
What carries the argument. The central object is the runtime policy tuple $P=(E,G,A,R,S)$: an eligibility predicate $E$ reads the task description once to decide whether the policy is in scope, a runtime predicate $G$ reads the action history at harness events to detect a state that preceded an observed failure, an intervention $A$ either adds a short natural-language instruction or denies a proposed tool call, a release condition $R$ and a nudge limit prevent loops, and tracking rules update a policy state $S$. Separating $E$ from $G$ separates scope from timing: broad eligibility wastes tokens and distracts successful trajectories, while a late runtime predicate cannot prevent an irreversible action. This separation is what lets the evaluation attribute the gain to what the policy says rather than to the mere fact of an interruption.
Core claim. The central claim is that policy content, not interruption timing or generic checking, causes the reliability gain: runtime policies make capabilities an agent already possesses substantially more repeatable. On the 14 eligible panel tasks the real portfolio passes 17/28 attempts against 10/28 for a sham that fires at the same events with generic review text, and always-on verification and reconsideration instructions end at or near baseline. Across the full 87-task suite policies raise repeated success ($\mathrm{pass}^{\wedge 2}$) in all three model tiers, and for Sol this metric rises from 64.4% to 73.6% while best-of-two success rises only 1.2 points—the signature of converting reachable solutions into dependable delivery rather than expanding what is solved once.
The premise it rests on. The portfolios were produced by an iterative search that screened and revised candidates on positive and nearby-negative tasks drawn from the development population, and the causal panel then measures the effect on 14 eligible tasks from that same population, so the claim that the +25-point contrast reflects a general reliability property assumes this search did not overfit to the recurring failures of this specific benchmark; a secondary premise, which the authors state they cannot verify, is that the three GPT-5.6 routes were behaviorally stable during the evaluation window.
If the paper is right (Pith's inferences, not the authors')
On the full 87-task suite, policies raise repeated success ($\mathrm{pass}^{\wedge 2}$) in all three model tiers, and Sol's pattern—a 9.2-point rise in repeated success against a 1.2-point rise in best-of-two—shows the portfolio converts intermittent solutions into dependable delivery.
On the 14 tasks covered by Terra's frozen portfolio, policy-guided Terra reaches 71.4% versus 64.3% for unassisted Sol at roughly half the cost, placing a mid-tier model in the performance range of a costlier one on covered tasks.
A timing-matched sham and generic verify/reconsider instructions do not reproduce the gain, which rules out interruption and generic self-checking as the operative mechanism.
Policies do not add capability: on binary-equivalence tasks the registered corrective behavior appears but attempts still fail, because the policy cannot supply a missing reimplementation.
If the content-over-timing result generalizes, it suggests the reliability bottleneck is recall of the right procedure at the right moment rather than model competence: the model produces the corrective behavior when told specifically, but not when asked generically; a testable consequence is that other hand-authored or automatically generated specific instructions might recover a similar share of
Because policies cover only failure states that are observable in the harness, their reach is bounded by instrumentation; the authors note that no policy was written for several task classes because no observable intervention point was found, so richer state instrumentation (for example, inspecting generated artifacts) is the natural path to wider coverage.
The cost result points to a substitution effect the authors only gesture at: at the level of covered tasks, engineering around a cheaper model reproduced the reliability of a more expensive one at about half the cost, and whether that holds on a held-out distribution rather than the development population the policies were screened on is the open question that decides its practical value.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Language-model agents often reach a working solution and then fail to consistently deliver it. We study runtime policies: targeted natural-language instructions and action denials applied by the agent harness at states that preceded observed failures, without changing model weights or the user prompt. With this, keeping capability constant, we observe a meaningful unlock in delivered reliability. Across the complete 87-task Terminal-Bench 2.1 suite, with two attempts per task, policies increase repeated success (pass^2) in all three GPT-5.6 tiers: 50.6% to 54.0% for Luna, 55.2% to 60.9% for Terra, and 64.4% to 73.6% for Sol. Sol's best-of-two success changes by 1.2 points while repeated success rises by 9.2, showing that policies chiefly convert reachable solutions into dependable delivery. We further cover 14 tasks under Terra's frozen portfolio. Policy-guided Terra reaches 71.4%, compared with 64.3% for unassisted Sol, at about half the cost, demonstrating how engineering around models could unlock dependability for a use case. To isolate the mechanism we run a randomized five-arm experiment: real policies reach 61% on eligible tasks, versus 39% without a policy, 36% with a timing-matched sham, and 39 to 43% with generic verification or reconsideration. The intended corrective behavior appears in 22 of 24 coded policy attempts, against at most 14 in any other arm. Runtime policies are therefore a practical reliability layer: they make capabilities an agent already possesses substantially more repeatable.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-24
The authors' main claim, as the classifier picked it out 1.00
“Runtime policies are therefore a practical reliability layer: they make capabilities an agent already possesses substantially more repeatable.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.30 · human participants 0.04 · controlled comparison 0.98 · randomized assignment 0.76 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Language-model agents often reach a working solution and then fail to consistently deliver it. | background 0.98 | 0.14 | 0.03 | asserted 0.95 |
| We study runtime policies: targeted natural-language instructions and action denials applied by the agent harness at states that preceded observed failures, without changing model weights or the user prompt. | method 0.72 | 0.70 | 0.05 | asserted 0.76 |
| With this, keeping capability constant, we observe a meaningful unlock in delivered reliability. | result 0.71 | 0.92 | 0.37 | asserted 0.99 |
| Across the complete 87-task Terminal-Bench 2.1 suite, with two attempts per task, policies increase repeated success (pass^2) in all three GPT-5.6 tiers: 50.6% to 54.0% for Luna, 55.2% to 60.9% for Terra, and 64.4% to 73.6% for Sol. | result 1.00 | 0.96 | 0.18 | asserted 1.00 |
| Sol's best-of-two success changes by 1.2 points while repeated success rises by 9.2, showing that policies chiefly convert reachable solutions into dependable delivery. | interpretation 0.78 | 0.95 | 0.22 | asserted 0.98 |
| We further cover 14 tasks under Terra's frozen portfolio. | method 0.46 | 0.90 | 0.02 | asserted 0.87 |
| Policy-guided Terra reaches 71.4%, compared with 64.3% for unassisted Sol, at about half the cost, demonstrating how engineering around models could unlock dependability for a use case. | result 0.87 | 0.92 | 0.11 | asserted 0.80 |
| To isolate the mechanism we run a randomized five-arm experiment: real policies reach 61% on eligible tasks, versus 39% without a policy, 36% with a timing-matched sham, and 39 to 43% with generic verification or reconsideration. | result 0.51 | 0.97 | 0.07 | asserted 1.00 |
| The intended corrective behavior appears in 22 of 24 coded policy attempts, against at most 14 in any other arm. | result 0.99 | 0.95 | 0.06 | asserted 0.95 |
| Runtime policies are therefore a practical reliability layer: they make capabilities an agent already possesses substantially more repeatable. | interpretation 1.00 | 0.88 | 0.90 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.75 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-24: 1 request, 7,755 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (3)
References (62)
- [1] Terminal-Bench 2.1 Dataset and Evaluation Repository , year = on Pith
- [2] Harbor: A Framework for Running Agents in Containerized Environments , year = on Pith
- [4] Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , title =. International Conference on Learning Representations , year = on Pith
- [5] Liu, Xiao and Yu, Hao and Zhang, Hanchen and others , title =. International Conference on Learning Representations , year = on Pith
- [6] Xie, Tianbao and Zhang, Danyang and Chen, Jixuan and others , title =. Advances in Neural Information Processing Systems , year = on Pith
- [7] Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and others , title =. International Conference on Learning Representations , year = on Pith
- [8] Shinn, Noah and Cassano, Federico and Gopinath, Ashwin and Narasimhan, Karthik and Yao, Shunyu , title =. Advances in Neural Information Processing Systems , year = on Pith
- [9] Madaan, Aman and Tandon, Niket and Gupta, Prakhar and others , title =. Advances in Neural Information Processing Systems , year = on Pith
- [10] Yang, John and Jimenez, Carlos E. and Wettig, Alexander and others , title =. Advances in Neural Information Processing Systems , year = on Pith
- [11] Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral , title =. International Conference on Learning Representations , year = on Pith
- [13] Ruan, Yangjun and Dong, Honghua and Wang, Andrew and Pitis, Silviu and Zhou, Yongchao and Ba, Jimmy , title =. International Conference on Learning Representations , year = on Pith
- [14] Debenedetti, Edoardo and Zhang, Jie and Balunovi. Advances in Neural Information Processing Systems , year = on Pith
- [15] Bartocci, Ezio and Falcone, Yli. Introduction to Runtime Verification , booktitle =. 2018 , pages = on Pith
- [16] Schneider, Fred B. , title =. ACM Transactions on Information and System Security , volume =. 2000 , doi = on Pith
- [17] Sha, Lui , title =. IEEE Software , volume =. 2001 , doi = on Pith
- [18] Kapoor, Sayash and others , title =. International Conference on Learning Representations , year = on Pith
- [19] Hoover, Monte and Baherwani, Vatsal and Jain, Neel and Saifullah, Khalid and Vincent, Joseph and Jain, Chirag and Rad, Melissa Kazemi and Bruss, C. Bayan and Panda, Ashwinee and Goldstein, Tom , title =. International Conference on Learning Representations , year = on Pith
- [20] Merrill, Mike A. and Shaw, Alexander G. and Carlini, Nicholas and Li, Boxuan and Raj, Harsh and others , title =. International Conference on Learning Representations , year = on Pith
- [21] Patterson, David and Brown, Aaron and Broadwell, Pete and Candea, George and Chen, Mike and Cutler, James and Enriquez, Patricia and Fox, Armando and Kiciman, Emre and Merzbacher, Matthew and Oppenheimer, David and Sastry, Naveen and Tetzlaff, William and Traupman, Jonathan an... on Pith
- [22] Yao, Shunyu and Shinn, Noah and Razavi, Pedram and Narasimhan, Karthik , title =. International Conference on Learning Representations , year = on Pith
show 42 more
- [23] Zhang, Jiayi and Xiang, Jinyu and Yu, Zhaoyang and Teng, Fengwei and Chen, Xionghui and Chen, Jiaqi and Zhuge, Mingchen and Cheng, Xin and Hong, Sirui and Wang, Jinlin and Zheng, Bingnan and Liu, Bang and Luo, Yuyu and Wu, Chenglin , title =. International Conference on Learni... on Pith
- [24] Kapoor, Sayash and Stroebl, Benedikt and Siegel, Zachary S. and Nadgir, Nitya and Narayanan, Arvind , title =. Transactions on Machine Learning Research , year = on Pith
- [25] Wang, Haoyu and Poskitt, Christopher M. and Sun, Jun , title =. Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE) , year = on Pith
- [26] Xiang, Zhen and Zheng, Linzhi and Li, Yanjie and Hong, Junyuan and Li, Qinbin and Xie, Han and Zhang, Jiawei and Xiong, Zidi and Xie, Chulin and Yang, Carl and Song, Dawn and Li, Bo , title =. Proceedings of the 42nd International Conference on Machine Learning , year = on Pith
- [27] Huang, Jie and Chen, Xinyun and Mishra, Swaroop and Zheng, Huaixiu Steven and Yu, Adams Wei and Song, Xinying and Zhou, Denny , title =. International Conference on Learning Representations , year = on Pith
- [29] Berg-Kirkpatrick, Taylor and Burkett, David and Klein, Dan. An Empirical Investigation of Statistical Significance in NLP. Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 2012 on Pith
- [31] Yin, Wenpeng and Hay, Jamaal and Roth, Dan. Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La... on Pithdoi:10.18653/v1/d19-1404
- [35] Codex. 2026 , howpublished = on Pith
- [36] Ezio Bartocci, Yli è s Falcone, Adrian Francalanza, and Giles Reger. 2018. https://doi.org/10.1007/978-3-319-75632-5_1 Introduction to runtime verification . In Lectures on Runtime Verification, pages 1--33. Springer on Pithdoi:10.1007/978-3-319-75632-5_1
- [37] Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. https://aclanthology.org/D12-1091/ An empirical investigation of statistical significance in NLP . In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Nat... on Pith
- [38] Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.745 With little power comes great responsibility . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro... on Pithdoi:10.18653/v1/2020.emnlp-main.745
- [39] Mark Chen et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 on PithPith's reviewarXiv
- [40] Edoardo Debenedetti, Jie Zhang, Mislav Balunovi \'c , Luca Beurer-Kellner, Marc Fischer, and Florian Tram \`e r. 2024. https://arxiv.org/abs/2406.13352 AgentDojo : A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents . In Advances in Neural In... on PithPith's reviewarXiv
- [41] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. https://doi.org/10.18653/v1/P18-1128 The hitchhiker ' s guide to testing statistical significance in natural language processing . In Proceedings of the 56th Annual Meeting of the Association for Computational Lin... on Pithdoi:10.18653/v1/p18-1128
- [42] Harbor contributors . 2026. Harbor: A framework for running agents in containerized environments. https://github.com/harbor-framework/harbor. Accessed 17 August 2026 on Pith
- [43] Monte Hoover, Vatsal Baherwani, Neel Jain, Khalid Saifullah, Joseph Vincent, Chirag Jain, Melissa Kazemi Rad, C. Bayan Bruss, Ashwinee Panda, and Tom Goldstein. 2026. https://arxiv.org/abs/2509.02563 DynaGuard : A dynamic guardian model with user-defined policies . In Internat... on PitharXiv
- [44] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://openreview.net/forum?id=IkmD3fKBPQ Large language models cannot self-correct reasoning yet . In International Conference on Learning Representations on Pith
- [45] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench : Can language models resolve real-world GitHub issues? In International Conference on Learning Representations on Pith
- [46] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. https://doi.org/10.1162/tacl_a_00713 When can LLM s actually correct their own mistakes? a critical survey of self-correction of LLM s . Transactions of the Association for Computational Linguistics, 12:1417--1440 on Pithdoi:10.1162/tacl_a_00713
- [47] Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. 2025. https://arxiv.org/abs/2407.01502 AI agents that matter . Transactions on Machine Learning Research on PithPith's reviewarXiv
- [48] Sayash Kapoor et al. 2026. https://arxiv.org/abs/2510.11977 Holistic agent leaderboard: The missing infrastructure for AI agent evaluation . In International Conference on Learning Representations on PitharXiv
- [49] Xiao Liu, Hao Yu, Hanchen Zhang, et al. 2024. https://arxiv.org/abs/2308.03688 AgentBench : Evaluating LLMs as agents . In International Conference on Learning Representations on PithPith's reviewarXiv
- [50] Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.65 T ool S andbox: A stateful, conversational, interactive evaluation ... on Pithdoi:10.18653/v1/2025.findings-naacl.65
- [51] Aman Madaan, Niket Tandon, Prakhar Gupta, et al. 2023. https://arxiv.org/abs/2303.17651 Self-refine: Iterative refinement with self-feedback . In Advances in Neural Information Processing Systems on PithPith's reviewarXiv
- [52] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. 2026. https://arxiv.org/abs/2601.11868 Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces . In International Conference on Learning Representations on PithPith's reviewarXiv
- [53] OpenAI . 2026. Codex CLI : Lightweight coding agent that runs in your terminal. https://github.com/openai/codex. Version 0.146.0; Apache-2.0 license on Pith
- [54] David Patterson, Aaron Brown, Pete Broadwell, George Candea, Mike Chen, James Cutler, Patricia Enriquez, Armando Fox, Emre Kiciman, Matthew Merzbacher, David Oppenheimer, Naveen Sastry, William Tetzlaff, Jonathan Traupman, and Noah Treuhaft. 2002. Recovery-oriented computing (... on Pith
- [55] Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.40 N e M o guardrails: A toolkit for controllable and safe LLM applications with programmable rails . In Proceedings of the 2023 ... on Pithdoi:10.18653/v1/2023.emnlp-demo.40
- [56] Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, and Jimmy Ba. 2024. https://arxiv.org/abs/2309.15817 Identifying the risks of LM agents with an LM -emulated sandbox . In International Conference on Learning Representations on PithPith's reviewarXiv
- [57] Fred B. Schneider. 2000. https://doi.org/10.1145/353323.353382 Enforceable security policies . ACM Transactions on Information and System Security, 3(1):30--50 on PitharXiv
- [58] Lui Sha. 2001. https://doi.org/10.1109/52.936447 Using simplicity to control complexity . IEEE Software, 18(4):20--28 on Pithdoi:10.1109/52.936447
- [59] Tianneng Shi, Jingxuan He, Zhun Wang, Hongwei Li, Linyu Wu, Wenbo Guo, and Dawn Song. 2025. https://arxiv.org/abs/2504.11703 Progent : Securing AI agents with privilege control . arXiv preprint arXiv:2504.11703 on PithPith's reviewarXiv
- [60] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://arxiv.org/abs/2303.11366 Reflexion: Language agents with verbal reinforcement learning . In Advances in Neural Information Processing Systems on PithPith's reviewarXiv
- [61] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. https://arxiv.org/abs/2408.03314 Scaling LLM test-time compute optimally can be more effective than scaling model parameters . In International Conference on Learning Representations on PithPith's reviewarXiv
- [62] Terminal-Bench contributors . 2026. Terminal-bench 2.1 dataset and evaluation repository. https://github.com/harbor-framework/terminal-bench-2-1. Accessed 31 August 2026 on Pith
- [63] Haoyu Wang, Christopher M. Poskitt, and Jun Sun. 2026. https://arxiv.org/abs/2503.18666 AgentSpec : Customizable runtime enforcement for safe and reliable LLM agents . In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE) on PithPith's reviewarXiv
- [64] Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. 2025. https://icml.cc/virtual/2025/poster/46569 GuardAgent : Safeguard LLM agents via knowledge-enabled reasoning . In Proceedings of t... on Pith
- [65] Tianbao Xie, Danyang Zhang, Jixuan Chen, et al. 2024. https://arxiv.org/abs/2404.07972 OSWorld : Benchmarking multimodal agents for open-ended tasks in real computer environments . In Advances in Neural Information Processing Systems on PithPith's reviewarXiv
- [66] John Yang, Carlos E. Jimenez, Alexander Wettig, et al. 2024. https://arxiv.org/abs/2405.15793 SWE-agent : Agent-computer interfaces enable automated software engineering . In Advances in Neural Information Processing Systems on PithPith's reviewarXiv
- [67] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html -bench: A benchmark for tool-agent-user interaction in real-world domains . In International C... on Pith
- [68] Shunyu Yao, Jeffrey Zhao, Dian Yu, et al. 2023. https://arxiv.org/abs/2210.03629 ReAct : Synergizing reasoning and acting in language models . In International Conference on Learning Representations on PithPith's reviewarXiv
- [69] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. 2025. https://openreview.net/forum?id=z5uVAKwmjf AFlow : Automating agentic workflow generati... on Pith
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
- Harness Self-Evolution for Long-Horizon Terminal Agents cs.AI · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.



No comments yet.
Sign in to comment