Artificial Intelligence · arXiv:2610.06563 v1 · submitted 5 Oct 2026 · 129 references
HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention
The paper says
“Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists.”
more of the abstract
Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 6 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A genuinely new failure-driven co-evolution loop for agentic abstention with a smart transfer design, held back by LLM-only infeasibility certification and noise-level per-model numbers. This line is wrong
4 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 129 references say.
The paper says, at §3.1, Appendix B.3, Appendix D
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
The paper says, at §4, §5, Tables 1 and 3
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
The paper says, at §5, Table 1
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
The paper says, at §4, §5, Appendix F.1
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
6 minor objections
The paper says, at Appendix A.1
Pith says: minor objection 1 of 6
The paper says, at §3 and figures
Pith says: minor objection 2 of 6
show 4 more
The paper says, at Figure 2
Pith says: minor objection 3 of 6
The paper says, at §5.2, Appendix E.2
Pith says: minor objection 4 of 6
The paper says, at Appendix E.1, Table 5
Pith says: minor objection 5 of 6
The paper says, at §3.2, Appendix A.3
Pith says: minor objection 6 of 6
What would settle it
Re-verify all 60 HERA-BENCH 'infeasible' instances by an independent mechanical check: instrument the environment to enumerate tool calls with arguments drawn from reachable state values, and run a bounded breadth-first search up to the 30-turn/8192-token budget, checking each terminal state against the task verifier. If any such instance yields a satisfying path (or the search cannot rule it out within budget), the label is unsupported; recompute Abstain/Pair after correcting labels and observe whether the 83.3% claim survives. As a cheaper first pass, run the stronger rescue protocol with multiple independent LLMs (e.g., GPT-6-Astra, Claude Opus 5) under instructions to find any valid completion for each infeasible instance, and flag disagreements.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 6 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The genuinely new thing is the loop: the current harness's failures drive both harness code edits and generation of new matched feasible/infeasible pairs that target the diagnosed weakness. Prior work either kept environments fixed (AgentAbstain, Agentic Harness Engineering) or only adapted validation-task selection (Task-CoEvolve); nobody generated new abstention environments against the current harness's failures. That coupling is real, and the design choices around it are thoughtful: batch admission requires the current harness to fail on at least a fraction of new infeasible tasks, checkpoint selection on a disjoint fixed validation set requires no regression on both Abstain and Pair, and the 19-model frozen-harness transfer is a genuine external check. The cost analysis, a cheaper model with HERA matching a pricier base at roughly 85% lower cost, is practically important if it holds. The load-bearing soft spot is certification. Infeasibility labels on both the training pool and HERA-BENCH come from the same LLM-based semantic/rescue-policy pipeline the method optimizes, with no completeness certificate. The paper knows the right standard, Appendix B.3 is explicit that an unsuccessful rollout alone is insufficient, but the rescue-policy search is LLM reasoning and the human review is a plausibility check over 120 instances, not exhaustive. Because the environments are executable Python with small tool sets and a 30-turn budget, infeasibility is decidable in principle; an exhaustive or mechanised check would close the gap, and none is reported. A missed alternative path would both train and measure over-abstention, and the paper's over-abstention metric cannot catch it, since a mislabeled task is by construction labeled infeasible. The weather-style missing-evidence mutations are comparatively safe by construction; the risk concentrates in constraint-conflict and authorization mutations, and the paper does not quantify the mix. Second, the statistics: 60 pairs, one rollout, no error bars. On 60 binary instances a 10-point delta is about six instances, so per-model gains of 6.7 to 40.0 points are largely noise. The aggregate 15.3-point transfer across 19x60 instances is far more robust, and that is the claim I would trust. Third, a point neither reviewer flagged: on held-out HERA-BENCH, the task-generation-from-H0-failures baseline beats co-evolution on Pair, 73.3 versus 70.0, roughly two instances. Co-evolution wins decisively on the training pools, but the headline held-out Pair edge over that simpler baseline is absent, and the paper does not address it. No code, data, harness, or benchmark is shipped; the reproducibility statement promises detailed descriptions, not artifacts. This is for people building deployment harnesses for tool agents; the transfer and cost results matter to them. It deserves a serious referee: accept the paper for review, then press for seed-averaged error bars, artifact release, and exhaustive verification on at least the benchmark pairs.
The strongest objection
The central claim—that the evolved harness raises abstention accuracy to 83.3% on HERA-BENCH—presupposes that every 'infeasible' instance genuinely has no valid solution path. That label is produced by the §3.1 pipeline: semantic policy π_sem and rescue policy π_res, LLM-based reviewers, gate each mutation; Appendix B.3 states 'an unsuccessful solver rollout alone is insufficient: the label requires a reason grounded in the task constraints and the environment's available capabilities.' This sets the right standard but supplies no certificate of completeness. π_res searches for alternative routes by LLM reasoning; a route it misses is the failure mode. If any mutated instance retains a valid tool sequence, then (a) training signals reward abstention where acting is correct, and (b) HERA-BENCH's ground truth for 'Abstain' is wrong, so measured gains overcount over-abstention rather than correct abstention. The later human review (Section 4, Appendix D) checks plausibility and 'whether another allowed approach could still satisfy the request', but it is an unmechanised plausibility check over 120 instances, not exhaustive verification. Because the environment is executable Python with a bounded 30-turn budget, true infeasibility is decidable in principle; the paper does not report any exhaustive or independently mechanised check, so the load-bearing assumption remains unproven.
The reader's weakest assumption identifies the certification completeness of §3.1's infeasibility labels, and I agree this is the most load-bearing point. The central claim is an abstention gain on HERA-BENCH; if the benchmark's 'infeasible' ground truth is wrong in either direction, the metric is not abstention accuracy. The paper's own Appendix B.3 states the correct standard—'an unsuccessful solver rollout alone is insufficient'—and the human review in Appendix D is an independent check, but neither constitutes a completeness argument for π_res's alternative-route search. This is an unproven assumption inside an otherwise well-structured evaluation: training and test splits are disjoint, the validation set is fixed, and the 19-model transfer test is a genuine external check, which deserves credit. A single concrete experiment—mechanically searching the executable environments for any valid tool sequence for each allegedly infeasible instance—would settle whether the assumption fails. If labels hold, the conditional concerns about statistical noise and reproducibility remain but the central claim stands. If labels fail, over-abstention is built into both training and measurement, which would directly undercut the 83.3% headline. Given the executable, bounded nature of the environments, this test is feasible and should be run before the numbers are taken as established. The reader's CONDITIONAL verdict is the right one; this stress-test does not move it.
The referee's report
The co-evolution framing is timely, and the experimental design is stronger than is typical for this line of work: the held-out HERA-BENCH uses domains disjoint from training, checkpoint selection is done on a fixed validation set, and the 19-model transfer test is a genuine generalization check that rules out model-specific overfitting. The adversarial batch-admission rule, the paired Act/Abstain reporting, and the cost analysis are all useful design choices. If the ground-truth infeasibility labels can be certified, the central claim — that failure-driven environment evolution plus harness adaptation improves abstention without sacrificing feasible-task completion, and that the resulting harness transfers across models — is credible and worth publishing. However, the paper's linchpin, the correctness of the 'infeasible' labels in HERA-BENCH and in the training pool, rests on LLM-based semantic and rescue policies with no completeness certificate, and the headline numbers carry no uncertainty estimates.
Pith's reading of the paper (a machine's summary, not the authors')
HERA claims that an agent's ability to abstain — to recognize when a task cannot be completed and stop rather than act — is shaped by the harness around the model, and that the most reliable way to build that harness is to let the tasks themselves evolve alongside it. The framework constructs matched pairs of feasible and infeasible tasks by mutating environments so that the same request becomes unsolvable, then repeatedly uses the current harness's failures to both edit the harness code and generate new task pairs aimed at those failures. On a held-out set of 60 pairs, HERA's final harness lifts abstention success from 61.7% to 83.3%, paired success from 48.3% to 70.0%, and feasible-task completion from 68.3% to 76.7%, beating fixed-task, prompting-only, and harness-only baselines. The same frozen harness transfers to 19 other LLMs, raising abstention accuracy by an average of 15.3 points with no model-specific tuning. If correct, this means abstention can be engineered at the harness level rather than retrained into each model.
What carries the argument. The load-bearing object of the paper is the co-evolution loop itself, built on matched feasible–infeasible task pairs: each pair shares the same task instruction, and a controlled environment mutation determines whether execution is possible, so success and abstention can both be scored against a verified reference. The loop alternates two moves. Environment evolution uses a failure diagnosis $F_k$ from paired trajectory rollouts to guide an environment designer in modifying data, tool behavior, or constraints, after which new candidate pairs must pass a certification gate (semantic policy $\pi_{\mathrm{sem}}$ and rescue policy $\pi_{\mathrm{res}}$, checking consistency and hunting for unintended solution routes) and are admitted to the training pool only if the current harness fails to abstain on at least $\tau = 0.4$ of the five infeasible instances. Harness evolution, with the underlying model fixed, generates candidate harnesses $H_{k,i+1} = \mathrm{Optimize}(H_{k,i}, F_{k,i})$ over $n$ updates, each building on the preceding candidate, and adopts as the next checkpoint the candidate that on a disjoint validation set regresses in neither abstention nor paired success while strictly improving one. The paired rollout–diagnosis–update cycle is the mechanism claimed to keep the weakening-signal problem from stalling improvement.
Core claim. The paper's central claim is that optimizing a harness against a fixed set of tasks is inherently limiting for abstention, because as the harness improves, the same examples give less and less signal about where it still fails, so sustained progress requires the task distribution to adapt as well. HERA makes this concrete with a co-evolution loop: a pipeline that builds executable environments, validates a feasible task in each, and then applies controlled environment mutations — changing evidence availability, consistency of records, authorization scope, or interacting constraints — to turn that task into a matched infeasible variant requiring abstention, with a semantic policy and a rescue policy certifying each label; and a loop in which execution failures on the current harness drive both targeted edits to the harness (prompts, memory, tools, skills, sub-agents, control logic, with the model fixed) and directives for generating new environments and task pairs aimed at the diagnosed weaknesses. The claim is that this coupling, and not either piece alone, produces the reported results: abstention success rises from 61.7% to 83.3% and paired success from 48.3% to 70.0% on held-out HERA-BENCH, with feasible completion improving rather than falling, and a single harness generalizing across 19 other models. The paper further claims that a cheaper model can match a costlier one — GPT-5.6-Luna with the evolved harness reaches the paired success of base GPT-6-Astra at about 85% lower per-episode API cost.
The premise it rests on. The load-bearing assumption is that the certification gate in Section 3.1, using the semantic and rescue policies, correctly establishes that every mutated 'infeasible' instance truly has no valid solution path — the paper itself states (Appendix B.3) that an unsuccessful solver rollout alone is insufficient and a label requires a reason grounded in task constraints, but it does not prove that these two LLM-based review policies exhaust all possible completion routes, so a rescue path the policies miss would mislabel a feasible task as one requiring abstention and train the harness to over-abstain.
If the paper is right (Pith's inferences, not the authors')
Fixed-task harness optimization plateaus, while co-evolution keeps improving: because each accepted task batch is retained, HERA must improve on new tasks without degrading earlier ones, which the training-pool dynamics (86.0% abstain, 88.0% act on the final 50-pair pool) claim to achieve.
Abstention gains need not cost feasible-task performance: co-evolution raises Act to 76.7% while abstention prompting alone lowers it to 63.3%, so the abstain-versus-act trade-off is partly a harness-design artifact that this loop sidesteps.
Harness improvements transfer across models: a harness evolved on one model lifts abstention by 15.3 points on average across 19 others, including a 10-point gain on a stronger model (GPT-6-Astra), implying that abstention logic lives substantially in the harness rather than the model.
Cost parity becomes possible: a cheaper model running the evolved harness matches the paired success of a more expensive model's base harness at an estimated 85% reduction in per-episode API cost.
New task batches generated against the current harness start at lower performance and recover after harness adaptation (for example, batch $B_6$ goes from 60.0% to 80.0% after the $H_6$ update), which the paper reads as evidence that the co-evolution targets real weaknesses rather than adding redundant training data.
The abstention improvements are only as sound as the certification reviewers' coverage: because the 'infeasible' label rests on LLM policies whose completeness is not proven, the trained behavior could partly be learning those reviewers' blind spots, and the decisive check is to independently hunt for completion paths in ostensibly infeasible environments.
If the mechanism generalizes, the natural extension is direct transfer to real abstention settings — production tool environments where prerequisites, permissions, or data are missing — which HERA-BENCH approximates with synthetic mutations but does not directly test.
The round-over-round saturation the paper shows is likely to become the main risk beyond these rounds: co-evolution may overfit to the task-generator's style distribution, so a testable extension is whether abstention on held-out real-world-style infeasible tasks keeps rising or flattens as rounds grow.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists. Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-06
The authors' main claim, as the classifier picked it out 0.59
“On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.12 · human participants 0.03 · controlled comparison 0.83 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists. | background 0.80 | 0.06 | 0.02 | asserted 0.99 |
| Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. | gap 0.57 | 0.04 | 0.03 | asserted 1.00 |
| We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. | contribution 1.00 | 0.83 | 0.08 | asserted 0.99 |
| HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. | method 0.97 | 0.89 | 0.06 | asserted 1.00 |
| On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. | result 1.00 | 0.96 | 0.49 | asserted 1.00 |
| The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost. | result 0.98 | 0.94 | 0.48 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.93
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-06: 1 request, 5,265 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Figures are the authors', reproduced from the paper.
References (129)
- [1] Abstentionbench: Reasoning llms fail on unanswerable questions , author=. arXiv preprint arXiv:2506.09038 , year= on PithPith's reviewarXiv
- [2] AutoData: Agentic Search for Pre-training Data Selection , author= on Pith
- [3] Mind2web: Towards a generalist agent for the web , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [4] Webshop: Towards scalable real-world web interaction with grounded language agents , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [5] Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. arXiv preprint arXiv:2601.11868 , year= on PithPith's reviewarXiv
- [6] ALCUNA: Large language models meet new knowledge , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
- [7] BBQ: A hand-built bias benchmark for question answering , author=. Findings of the Association for Computational Linguistics: ACL 2022 , pages= on Pith
- [8] Beyond the imitation game: Quantifying and extrapolating the capabilities of language models , author=. Transactions on machine learning research , year= on Pith
- [9] The art of saying no: Contextual noncompliance in language models , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [10] Won’t get fooled again: Answering questions with false premises , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [11] Susbench: An online benchmark for evaluating dark pattern susceptibility of computer-use agents , author=. Proceedings of the 31st International Conference on Intelligent User Interfaces , pages= on Pith
- [12] Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification , author=. arXiv preprint arXiv:2601.16400 , year= on PithPith's reviewarXiv
- [13] Freshllms: Refreshing large language models with search engine augmentation , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages= on Pith
- [14] Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages= on Pith
- [15] Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [16] Evaluating the moral beliefs encoded in llms , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [17] The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
- [18] 2: Question answering with questionable assumptions , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [19] A dataset of information-seeking questions and answers anchored in research papers , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages= on Pith
- [20] SituatedQA: Incorporating extra-linguistic contexts into QA , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
show 80 more
- [21] Know what you don’t know: Unanswerable questions for SQuAD , author=. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages= on Pith
- [22] Worldsense: A synthetic benchmark for grounded reasoning in large language models , author=. arXiv preprint arXiv:2311.15930 , year= on PithPith's reviewarXiv
- [23] Gpqa: A graduate-level google-proof q&a benchmark , author=. First conference on language modeling , year= on Pith
- [24] Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year= on PithPith's reviewarXiv
- [25] Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year= on PithPith's reviewarXiv
- [26] Benchmarking hallucination in large language models based on unanswerable math word problem , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages= on Pith
- [27] Agentbench: Evaluating llms as agents , author=. arXiv preprint arXiv:2308.03688 , year= on PithPith's reviewarXiv
- [28] Simcse: Simple contrastive learning of sentence embeddings , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages= on Pith
- [29] The power of scale for parameter-efficient prompt tuning , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages= on Pith
- [30] Demystifying prompts in language models via perplexity estimation , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages= on Pith
- [31] Unnatural instructions: Tuning language models with (almost) no human labor , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [32] MuSiQue: Multihop Questions via Single-hop Question Composition , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher= on Pith
- [33] On evaluation of embodied navigation agents , author=. arXiv preprint arXiv:1807.06757 , year= on PithPith's reviewarXiv
- [34] The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year= on PithPith's reviewarXiv
- [35] Gemini 3 Flash Model Card , author =. 2025 , institution = on Pith
- [36] Grok 4.1 Model Card , author =. 2025 , institution = on Pith
- [37] Llama 3.3 70B Instruct , author =. 2024 , url = on Pith
- [38] Llama 4 Maverick , author =. 2025 , url = on Pith
- [39] gpt-oss-120b & gpt-oss-20b model card , author=. arXiv preprint arXiv:2508.10925 , year= on PithPith's reviewarXiv
- [40] Ministral 3 14B , author =. 2025 , howpublished = on Pith
- [41] MiniMax M2.5 , author =. 2026 , howpublished = on Pith
- [42] DeepSeek-R1-Distill-Llama-70B , author =. 2025 , howpublished = on Pith
- [43] Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year= on PithPith's reviewarXiv
- [44] s1.1-32B , author =. 2025 , howpublished = on Pith
- [45] Qwen3.5-9B , author =. 2026 , howpublished = on Pith
- [46] Qwen2.5 Technical Report , author=. 2025 , eprint= on Pith
- [47] Qwen2.5-32B-Instruct , author =. 2024 , howpublished = on Pith
- [48] Know your limits: A survey of abstention in large language models , author=. Transactions of the Association for Computational Linguistics , volume=. 2025 , publisher= on Pith
- [49] Natural questions: a benchmark for question answering research , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher= on Pith
- [50] Coqa: A conversational question answering challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher= on Pith
- [51] QuAC: Question answering in context , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages= on Pith
- [52] AmbigQA: Answering ambiguous open-domain questions , author=. Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages= on Pith
- [53] Do large language models know what they don’t know? , author=. Findings of the association for Computational Linguistics: ACL 2023 , pages= on Pith
- [54] Pubmedqa: A dataset for biomedical research question answering , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages= on Pith
- [55] Realtime qa: What's the answer right now? , author=. Advances in neural information processing systems , volume= on Pith
- [56] Alignment for honesty , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [57] Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [58] When not to trust language models: Investigating effectiveness of parametric and non-parametric memories , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages= on Pith
- [59] Simple entity-centric questions challenge dense retrievers , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
- [60] Realtoxicityprompts: Evaluating neural toxic degeneration in language models , author=. Findings of the association for computational linguistics: EMNLP 2020 , pages= on Pith
- [61] Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection , author=. Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages= on Pith
- [62] Latent hatred: A benchmark for understanding implicit hate speech , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages= on Pith
- [63] Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages= on Pith
- [64] Beavertails: Towards improved safety alignment of llm via a human-preference dataset , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [65] Cvalues: Measuring the values of chinese large language models from safety to responsibility , author=. arXiv preprint arXiv:2307.09705 , year= on PithPith's reviewarXiv
- [66] Xstest: A test suite for identifying exaggerated safety behaviours in large language models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages= on Pith
- [67] Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models , author=. arXiv preprint arXiv:2307.08487 , year= on PithPith's reviewarXiv
- [68] " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models , author=. Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pages= on Pith
- [69] Do-not-answer: Evaluating safeguards in LLMs , author=. Findings of the Association for Computational Linguistics: EACL 2024 , pages= on Pith
- [70] All languages matter: On the multilingual safety of large language models , author=. arXiv preprint arXiv:2310.00905 , year= on PithPith's reviewarXiv
- [71] Salad-bench: A hierarchical and comprehensive safety benchmark for large language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages= on Pith
- [72] Sorry-bench: Systematically evaluating large language model safety refusal , author=. arXiv preprint arXiv:2406.14598 , year= on PithPith's reviewarXiv
- [73] Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms , author=. Advances in neural information processing systems , volume= on Pith
- [74] SMART: Self-aware agent for tool overuse mitigation , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages= on Pith
- [75] Toward a theory of agents as tool-use decision-makers , author=. arXiv preprint arXiv:2506.00886 , year= on PithPith's reviewarXiv
- [76] Over-Searching in Search-Augmented Large Language Models , author=. 2026 , eprint= on Pith
- [77] Clarify when necessary: Resolving ambiguity through interaction with lms , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages= on Pith
- [78] CLAMBER: A benchmark of identifying and clarifying ambiguous information needs in large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [79] Active task disambiguation with llms , author=. arXiv preprint arXiv:2502.04485 , year= on PithPith's reviewarXiv
- [80] Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities , author=. arXiv preprint arXiv:2602.05073 , year= on PithPith's reviewarXiv
- [81] Position: Uncertainty quantification needs reassessment for large-language model agents , author=. arXiv preprint arXiv:2505.22655 , year= on PithPith's reviewarXiv
- [82] Flashrag: A modular toolkit for efficient retrieval-augmented generation research , author=. Companion Proceedings of the ACM on Web Conference 2025 , pages= on Pith
- [83] Text embeddings by weakly-supervised contrastive pre-training , author=. arXiv preprint arXiv:2212.03533 , year= on PithPith's reviewarXiv
- [84] MemGPT: Towards LLMs as Operating Systems , author=. 2024 , eprint= on Pith
- [85] React: Synergizing reasoning and acting in language models , author=. The eleventh international conference on learning representations , year= on Pith
- [86] Voyager: An open-ended embodied agent with large language models , author=. arXiv preprint arXiv:2305.16291 , year= on PithPith's reviewarXiv
- [87] MARVEL: Modular Abstention for Reliable and Versatile Expert LLMs , author= on Pith
- [88] Characterizing LLM abstention behavior in science QA with context perturbations , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages= on Pith
- [89] DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responses , author=. arXiv preprint arXiv:2512.02282 , year= on PitharXiv
- [90] Qwen3 Technical Report , author=. 2025 , eprint= on Pith
- [91] GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint= on Pith
- [92] Gemma 4 31B IT , year = on Pith
- [93] Agentic context engineering: Evolving contexts for self-improving language models , author=. arXiv preprint arXiv:2510.04618 , year= on PithPith's reviewarXiv
- [94] Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume= on Pith
- [95] Optimizing generative ai by backpropagating language model feedback , author=. Nature , volume=. 2025 , publisher= on Pith
- [96] Dynamic cheatsheet: Test-time learning with adaptive memory , author=. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [97] Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety , author=. arXiv preprint arXiv:2510.16492 , year= on PithPith's reviewarXiv
- [98] Agentic Abstention: Do Agents Know When to Stop Instead of Act? , author=. arXiv preprint arXiv:2606.28733 , year= on PithPith's reviewarXiv
- [99] AgentAbstain: Do LLM Agents Know When Not to Act? , author=. arXiv preprint arXiv:2607.10059 , year= on PithPith's reviewarXiv
- [100] Dspy: Compiling declarative language model calls into self-improving pipelines , author=. arXiv preprint arXiv:2310.03714 , year= on PithPith's reviewarXiv
The first 100 of 129 references are listed here. All of them are in this paper's citation record.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment