Pith.

Artificial Intelligence · arXiv:2610.06563 v1 · submitted 5 Oct 2026 · 129 references

HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention

Han Luo, Bingbing Wen, Guang Yang, Zora Zhiruo Wang, Pan Lu, Lucy Lu Wang

The paper says

“Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists.”

more of the abstract

Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 6 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A genuinely new failure-driven co-evolution loop for agentic abstention with a smart transfer design, held back by LLM-only infeasibility certification and noise-level per-model numbers. This line is wrong

4 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 129 references say.

The paper says, at §3.1, Appendix B.3, Appendix D

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The paper says, at §4, §5, Tables 1 and 3

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The paper says, at §5, Table 1

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The paper says, at §4, §5, Appendix F.1

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

6 minor objections

The paper says, at Appendix A.1

Pith says: minor objection 1 of 6

The paper says, at §3 and figures

Pith says: minor objection 2 of 6

show 4 more

The paper says, at Figure 2

Pith says: minor objection 3 of 6

The paper says, at §5.2, Appendix E.2

Pith says: minor objection 4 of 6

The paper says, at Appendix E.1, Table 5

Pith says: minor objection 5 of 6

The paper says, at §3.2, Appendix A.3

Pith says: minor objection 6 of 6

What would settle it

Re-verify all 60 HERA-BENCH 'infeasible' instances by an independent mechanical check: instrument the environment to enumerate tool calls with arguments drawn from reachable state values, and run a bounded breadth-first search up to the 30-turn/8192-token budget, checking each terminal state against the task verifier. If any such instance yields a satisfying path (or the search cannot rule it out within budget), the label is unsupported; recompute Abstain/Pair after correcting labels and observe whether the 83.3% claim survives. As a cheaper first pass, run the stronger rescue protocol with multiple independent LLMs (e.g., GPT-6-Astra, Claude Opus 5) under instructions to find any valid completion for each infeasible instance, and flag disagreements.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 6 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 58 s · 27,513 tokens read, 4,799 written
  3. Editor wrote the verdict above 35 s · 29,081 tokens read, 769 written
  4. Referee major revision: 4 major objections, 6 minor 41 s · 27,647 tokens read, 6,671 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The genuinely new thing is the loop: the current harness's failures drive both harness code edits and generation of new matched feasible/infeasible pairs that target the diagnosed weakness. Prior work either kept environments fixed (AgentAbstain, Agentic Harness Engineering) or only adapted validation-task selection (Task-CoEvolve); nobody generated new abstention environments against the current harness's failures. That coupling is real, and the design choices around it are thoughtful: batch admission requires the current harness to fail on at least a fraction of new infeasible tasks, checkpoint selection on a disjoint fixed validation set requires no regression on both Abstain and Pair, and the 19-model frozen-harness transfer is a genuine external check. The cost analysis, a cheaper model with HERA matching a pricier base at roughly 85% lower cost, is practically important if it holds. The load-bearing soft spot is certification. Infeasibility labels on both the training pool and HERA-BENCH come from the same LLM-based semantic/rescue-policy pipeline the method optimizes, with no completeness certificate. The paper knows the right standard, Appendix B.3 is explicit that an unsuccessful rollout alone is insufficient, but the rescue-policy search is LLM reasoning and the human review is a plausibility check over 120 instances, not exhaustive. Because the environments are executable Python with small tool sets and a 30-turn budget, infeasibility is decidable in principle; an exhaustive or mechanised check would close the gap, and none is reported. A missed alternative path would both train and measure over-abstention, and the paper's over-abstention metric cannot catch it, since a mislabeled task is by construction labeled infeasible. The weather-style missing-evidence mutations are comparatively safe by construction; the risk concentrates in constraint-conflict and authorization mutations, and the paper does not quantify the mix. Second, the statistics: 60 pairs, one rollout, no error bars. On 60 binary instances a 10-point delta is about six instances, so per-model gains of 6.7 to 40.0 points are largely noise. The aggregate 15.3-point transfer across 19x60 instances is far more robust, and that is the claim I would trust. Third, a point neither reviewer flagged: on held-out HERA-BENCH, the task-generation-from-H0-failures baseline beats co-evolution on Pair, 73.3 versus 70.0, roughly two instances. Co-evolution wins decisively on the training pools, but the headline held-out Pair edge over that simpler baseline is absent, and the paper does not address it. No code, data, harness, or benchmark is shipped; the reproducibility statement promises detailed descriptions, not artifacts. This is for people building deployment harnesses for tool agents; the transfer and cost results matter to them. It deserves a serious referee: accept the paper for review, then press for seed-averaged error bars, artifact release, and exhaustive verification on at least the benchmark pairs.

The strongest objection

The central claim—that the evolved harness raises abstention accuracy to 83.3% on HERA-BENCH—presupposes that every 'infeasible' instance genuinely has no valid solution path. That label is produced by the §3.1 pipeline: semantic policy π_sem and rescue policy π_res, LLM-based reviewers, gate each mutation; Appendix B.3 states 'an unsuccessful solver rollout alone is insufficient: the label requires a reason grounded in the task constraints and the environment's available capabilities.' This sets the right standard but supplies no certificate of completeness. π_res searches for alternative routes by LLM reasoning; a route it misses is the failure mode. If any mutated instance retains a valid tool sequence, then (a) training signals reward abstention where acting is correct, and (b) HERA-BENCH's ground truth for 'Abstain' is wrong, so measured gains overcount over-abstention rather than correct abstention. The later human review (Section 4, Appendix D) checks plausibility and 'whether another allowed approach could still satisfy the request', but it is an unmechanised plausibility check over 120 instances, not exhaustive verification. Because the environment is executable Python with a bounded 30-turn budget, true infeasibility is decidable in principle; the paper does not report any exhaustive or independently mechanised check, so the load-bearing assumption remains unproven.

The reader's weakest assumption identifies the certification completeness of §3.1's infeasibility labels, and I agree this is the most load-bearing point. The central claim is an abstention gain on HERA-BENCH; if the benchmark's 'infeasible' ground truth is wrong in either direction, the metric is not abstention accuracy. The paper's own Appendix B.3 states the correct standard—'an unsuccessful solver rollout alone is insufficient'—and the human review in Appendix D is an independent check, but neither constitutes a completeness argument for π_res's alternative-route search. This is an unproven assumption inside an otherwise well-structured evaluation: training and test splits are disjoint, the validation set is fixed, and the 19-model transfer test is a genuine external check, which deserves credit. A single concrete experiment—mechanically searching the executable environments for any valid tool sequence for each allegedly infeasible instance—would settle whether the assumption fails. If labels hold, the conditional concerns about statistical noise and reproducibility remain but the central claim stands. If labels fail, over-abstention is built into both training and measurement, which would directly undercut the 83.3% headline. Given the executable, bounded nature of the environments, this test is feasible and should be run before the numbers are taken as established. The reader's CONDITIONAL verdict is the right one; this stress-test does not move it.

The referee's report

The co-evolution framing is timely, and the experimental design is stronger than is typical for this line of work: the held-out HERA-BENCH uses domains disjoint from training, checkpoint selection is done on a fixed validation set, and the 19-model transfer test is a genuine generalization check that rules out model-specific overfitting. The adversarial batch-admission rule, the paired Act/Abstain reporting, and the cost analysis are all useful design choices. If the ground-truth infeasibility labels can be certified, the central claim — that failure-driven environment evolution plus harness adaptation improves abstention without sacrificing feasible-task completion, and that the resulting harness transfers across models — is credible and worth publishing. However, the paper's linchpin, the correctness of the 'infeasible' labels in HERA-BENCH and in the training pool, rests on LLM-based semantic and rescue policies with no completeness certificate, and the headline numbers carry no uncertainty estimates.

Pith's reading of the paper (a machine's summary, not the authors')

HERA claims that an agent's ability to abstain — to recognize when a task cannot be completed and stop rather than act — is shaped by the harness around the model, and that the most reliable way to build that harness is to let the tasks themselves evolve alongside it. The framework constructs matched pairs of feasible and infeasible tasks by mutating environments so that the same request becomes unsolvable, then repeatedly uses the current harness's failures to both edit the harness code and generate new task pairs aimed at those failures. On a held-out set of 60 pairs, HERA's final harness lifts abstention success from 61.7% to 83.3%, paired success from 48.3% to 70.0%, and feasible-task completion from 68.3% to 76.7%, beating fixed-task, prompting-only, and harness-only baselines. The same frozen harness transfers to 19 other LLMs, raising abstention accuracy by an average of 15.3 points with no model-specific tuning. If correct, this means abstention can be engineered at the harness level rather than retrained into each model.

What carries the argument. The load-bearing object of the paper is the co-evolution loop itself, built on matched feasible–infeasible task pairs: each pair shares the same task instruction, and a controlled environment mutation determines whether execution is possible, so success and abstention can both be scored against a verified reference. The loop alternates two moves. Environment evolution uses a failure diagnosis $F_k$ from paired trajectory rollouts to guide an environment designer in modifying data, tool behavior, or constraints, after which new candidate pairs must pass a certification gate (semantic policy $\pi_{\mathrm{sem}}$ and rescue policy $\pi_{\mathrm{res}}$, checking consistency and hunting for unintended solution routes) and are admitted to the training pool only if the current harness fails to abstain on at least $\tau = 0.4$ of the five infeasible instances. Harness evolution, with the underlying model fixed, generates candidate harnesses $H_{k,i+1} = \mathrm{Optimize}(H_{k,i}, F_{k,i})$ over $n$ updates, each building on the preceding candidate, and adopts as the next checkpoint the candidate that on a disjoint validation set regresses in neither abstention nor paired success while strictly improving one. The paired rollout–diagnosis–update cycle is the mechanism claimed to keep the weakening-signal problem from stalling improvement.

Core claim. The paper's central claim is that optimizing a harness against a fixed set of tasks is inherently limiting for abstention, because as the harness improves, the same examples give less and less signal about where it still fails, so sustained progress requires the task distribution to adapt as well. HERA makes this concrete with a co-evolution loop: a pipeline that builds executable environments, validates a feasible task in each, and then applies controlled environment mutations — changing evidence availability, consistency of records, authorization scope, or interacting constraints — to turn that task into a matched infeasible variant requiring abstention, with a semantic policy and a rescue policy certifying each label; and a loop in which execution failures on the current harness drive both targeted edits to the harness (prompts, memory, tools, skills, sub-agents, control logic, with the model fixed) and directives for generating new environments and task pairs aimed at the diagnosed weaknesses. The claim is that this coupling, and not either piece alone, produces the reported results: abstention success rises from 61.7% to 83.3% and paired success from 48.3% to 70.0% on held-out HERA-BENCH, with feasible completion improving rather than falling, and a single harness generalizing across 19 other models. The paper further claims that a cheaper model can match a costlier one — GPT-5.6-Luna with the evolved harness reaches the paired success of base GPT-6-Astra at about 85% lower per-episode API cost.

The premise it rests on. The load-bearing assumption is that the certification gate in Section 3.1, using the semantic and rescue policies, correctly establishes that every mutated 'infeasible' instance truly has no valid solution path — the paper itself states (Appendix B.3) that an unsuccessful solver rollout alone is insufficient and a label requires a reason grounded in task constraints, but it does not prove that these two LLM-based review policies exhaust all possible completion routes, so a rescue path the policies miss would mislabel a feasible task as one requiring abstention and train the harness to over-abstain.

If the paper is right (Pith's inferences, not the authors')

Fixed-task harness optimization plateaus, while co-evolution keeps improving: because each accepted task batch is retained, HERA must improve on new tasks without degrading earlier ones, which the training-pool dynamics (86.0% abstain, 88.0% act on the final 50-pair pool) claim to achieve.

Abstention gains need not cost feasible-task performance: co-evolution raises Act to 76.7% while abstention prompting alone lowers it to 63.3%, so the abstain-versus-act trade-off is partly a harness-design artifact that this loop sidesteps.

Harness improvements transfer across models: a harness evolved on one model lifts abstention by 15.3 points on average across 19 others, including a 10-point gain on a stronger model (GPT-6-Astra), implying that abstention logic lives substantially in the harness rather than the model.

Cost parity becomes possible: a cheaper model running the evolved harness matches the paired success of a more expensive model's base harness at an estimated 85% reduction in per-episode API cost.

New task batches generated against the current harness start at lower performance and recover after harness adaptation (for example, batch $B_6$ goes from 60.0% to 80.0% after the $H_6$ update), which the paper reads as evidence that the co-evolution targets real weaknesses rather than adding redundant training data.

The abstention improvements are only as sound as the certification reviewers' coverage: because the 'infeasible' label rests on LLM policies whose completeness is not proven, the trained behavior could partly be learning those reviewers' blind spots, and the decisive check is to independently hunt for completion paths in ostensibly infeasible environments.

If the mechanism generalizes, the natural extension is direct transfer to real abstention settings — production tool environments where prerequisites, permissions, or data are missing — which HERA-BENCH approximates with synthetic mutations but does not directly test.

The round-over-round saturation the paper shows is likely to become the main risk beyond these rounds: co-evolution may overfit to the task-generator's style distribution, so a testable extension is whether abstention on held-out real-world-style infeasible tasks keeps rising or flattens as rounds grow.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists. Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-06

The authors' main claim, as the classifier picked it out 0.59

“On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.12 · human participants 0.03 · controlled comparison 0.83 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists. background 0.80 0.06 0.02 asserted 0.99
Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. gap 0.57 0.04 0.03 asserted 1.00
We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. contribution 1.00 0.83 0.08 asserted 0.99
HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. method 0.97 0.89 0.06 asserted 1.00
On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. result 1.00 0.96 0.49 asserted 1.00
The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost. result 0.98 0.94 0.48 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.93

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-06: 1 request, 5,265 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (8)
Figure 1 of the paper
Figure 1. Harness–environment co-evolution. Given an environment pool of feasible and infea￾sible tasks and current harness, paired rollouts and failure diagnosis are used to iteratively evolve both the environment pool and harness. Environment evolution (top) constructs, validates, and se￾lects new feasible–infeasible task pairs to challenge the current agent and expand the training pool. Harness evolution (bottom) similarly refines the harness iteratively on the expanded task pool, with each updated harness selected from candidates using performance on a validation set. synthesize an…
Figure 2 of the paper
Figure 2. Performance on fixed task batches during co-evolution. From left to right, the panels show
Figure 3 of the paper
Figure 3. Abstention and paired success rates versus API cost on
Figure 4 of the paper
Figure 4. Example of harness evolution across successive updates. Starting from the base ex￾ecutor H0, each subsequent harness introduces or refines logic in response to diagnosed failures. H1 adds an evidence-based decision gate to distinguish recoverable uncertainty from conditions that warrant abstention; H2 introduces cross-source evidence reconciliation; H3 jointly verifies interact￾ing task constraints before consequential actions; and H4 extends this verification to authorization relationships and cumulative privacy constraints. Green regions indicate newly introduced or refined logic…
Figure 5 of the paper
Figure 5. End-to-End Environment–Task Construction Pipeline. The pipeline consists of three stages: environment construction, task generation, and environment mutation. A.2 MODELS AND INFERENCE SETTINGS Models and API access. We use GPT-5.6-Luna as the reference task-execution model during har￾ness development and in the main experiments, and evaluate the same final harness on 19 additional models. We evaluate GPT-5.6-Terra (OpenAI, 2026a), GPT-5.6-Sol (OpenAI, 2026a), GPT-6-Astra (OpenAI, 2026b), Claude Opus 5 (Anthropic, 2026a), Claude Sonnet 5 (Anthropic, 2026b), Gem￾ini 3.1 Pro (Preview)…
Figure 6 of the paper
Figure 6. Primary changes across rounds of environment evolution. Each bar summarizes the five
Figure 7 of the paper
Figure 7. Weather task and reference trajectory. The reference trajectory is obtained by executing a reference policy in the weather environment. The policy first looks up the Los Angeles weather point, then uses the returned county and forecast identifiers to retrieve the relevant stations and forecast periods. 22
Figure 8 of the paper
Figure 8. Weather task and reference trajectory in the infeasible instance. The task instruction and public tools remain unchanged. The forecast response marks the Saturday daytime precipita￾tion probability as unavailable. The reference trajectory still establishes the closest station and the available forecast evidence, but cannot determine whether the precipitation threshold is exceeded. Resource identifiers and observations are abbreviated for presentation. 23

Figures are the authors', reproduced from the paper.

References (129)
  1. [1] Abstentionbench: Reasoning llms fail on unanswerable questions , author=. arXiv preprint arXiv:2506.09038 , year= on PithPith's reviewarXiv
  2. [2] AutoData: Agentic Search for Pre-training Data Selection , author= on Pith
  3. [3] Mind2web: Towards a generalist agent for the web , author=. Advances in Neural Information Processing Systems , volume= on Pith
  4. [4] Webshop: Towards scalable real-world web interaction with grounded language agents , author=. Advances in Neural Information Processing Systems , volume= on Pith
  5. [5] Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. arXiv preprint arXiv:2601.11868 , year= on PithPith's reviewarXiv
  6. [6] ALCUNA: Large language models meet new knowledge , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
  7. [7] BBQ: A hand-built bias benchmark for question answering , author=. Findings of the Association for Computational Linguistics: ACL 2022 , pages= on Pith
  8. [8] Beyond the imitation game: Quantifying and extrapolating the capabilities of language models , author=. Transactions on machine learning research , year= on Pith
  9. [9] The art of saying no: Contextual noncompliance in language models , author=. Advances in Neural Information Processing Systems , volume= on Pith
  10. [10] Won’t get fooled again: Answering questions with false premises , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  11. [11] Susbench: An online benchmark for evaluating dark pattern susceptibility of computer-use agents , author=. Proceedings of the 31st International Conference on Intelligent User Interfaces , pages= on Pith
  12. [12] Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification , author=. arXiv preprint arXiv:2601.16400 , year= on PithPith's reviewarXiv
  13. [13] Freshllms: Refreshing large language models with search engine augmentation , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages= on Pith
  14. [14] Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages= on Pith
  15. [15] Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning , author=. Advances in Neural Information Processing Systems , volume= on Pith
  16. [16] Evaluating the moral beliefs encoded in llms , author=. Advances in Neural Information Processing Systems , volume= on Pith
  17. [17] The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
  18. [18] 2: Question answering with questionable assumptions , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  19. [19] A dataset of information-seeking questions and answers anchored in research papers , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages= on Pith
  20. [20] SituatedQA: Incorporating extra-linguistic contexts into QA , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
show 80 more
  1. [21] Know what you don’t know: Unanswerable questions for SQuAD , author=. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages= on Pith
  2. [22] Worldsense: A synthetic benchmark for grounded reasoning in large language models , author=. arXiv preprint arXiv:2311.15930 , year= on PithPith's reviewarXiv
  3. [23] Gpqa: A graduate-level google-proof q&a benchmark , author=. First conference on language modeling , year= on Pith
  4. [24] Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year= on PithPith's reviewarXiv
  5. [25] Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year= on PithPith's reviewarXiv
  6. [26] Benchmarking hallucination in large language models based on unanswerable math word problem , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages= on Pith
  7. [27] Agentbench: Evaluating llms as agents , author=. arXiv preprint arXiv:2308.03688 , year= on PithPith's reviewarXiv
  8. [28] Simcse: Simple contrastive learning of sentence embeddings , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages= on Pith
  9. [29] The power of scale for parameter-efficient prompt tuning , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages= on Pith
  10. [30] Demystifying prompts in language models via perplexity estimation , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages= on Pith
  11. [31] Unnatural instructions: Tuning language models with (almost) no human labor , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  12. [32] MuSiQue: Multihop Questions via Single-hop Question Composition , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher= on Pith
  13. [33] On evaluation of embodied navigation agents , author=. arXiv preprint arXiv:1807.06757 , year= on PithPith's reviewarXiv
  14. [34] The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year= on PithPith's reviewarXiv
  15. [35] Gemini 3 Flash Model Card , author =. 2025 , institution = on Pith
  16. [36] Grok 4.1 Model Card , author =. 2025 , institution = on Pith
  17. [37] Llama 3.3 70B Instruct , author =. 2024 , url = on Pith
  18. [38] Llama 4 Maverick , author =. 2025 , url = on Pith
  19. [39] gpt-oss-120b & gpt-oss-20b model card , author=. arXiv preprint arXiv:2508.10925 , year= on PithPith's reviewarXiv
  20. [40] Ministral 3 14B , author =. 2025 , howpublished = on Pith
  21. [41] MiniMax M2.5 , author =. 2026 , howpublished = on Pith
  22. [42] DeepSeek-R1-Distill-Llama-70B , author =. 2025 , howpublished = on Pith
  23. [43] Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year= on PithPith's reviewarXiv
  24. [44] s1.1-32B , author =. 2025 , howpublished = on Pith
  25. [45] Qwen3.5-9B , author =. 2026 , howpublished = on Pith
  26. [46] Qwen2.5 Technical Report , author=. 2025 , eprint= on Pith
  27. [47] Qwen2.5-32B-Instruct , author =. 2024 , howpublished = on Pith
  28. [48] Know your limits: A survey of abstention in large language models , author=. Transactions of the Association for Computational Linguistics , volume=. 2025 , publisher= on Pith
  29. [49] Natural questions: a benchmark for question answering research , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher= on Pith
  30. [50] Coqa: A conversational question answering challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher= on Pith
  31. [51] QuAC: Question answering in context , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages= on Pith
  32. [52] AmbigQA: Answering ambiguous open-domain questions , author=. Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages= on Pith
  33. [53] Do large language models know what they don’t know? , author=. Findings of the association for Computational Linguistics: ACL 2023 , pages= on Pith
  34. [54] Pubmedqa: A dataset for biomedical research question answering , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages= on Pith
  35. [55] Realtime qa: What's the answer right now? , author=. Advances in neural information processing systems , volume= on Pith
  36. [56] Alignment for honesty , author=. Advances in Neural Information Processing Systems , volume= on Pith
  37. [57] Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  38. [58] When not to trust language models: Investigating effectiveness of parametric and non-parametric memories , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages= on Pith
  39. [59] Simple entity-centric questions challenge dense retrievers , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
  40. [60] Realtoxicityprompts: Evaluating neural toxic degeneration in language models , author=. Findings of the association for computational linguistics: EMNLP 2020 , pages= on Pith
  41. [61] Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection , author=. Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) , pages= on Pith
  42. [62] Latent hatred: A benchmark for understanding implicit hate speech , author=. Proceedings of the 2021 conference on empirical methods in natural language processing , pages= on Pith
  43. [63] Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages= on Pith
  44. [64] Beavertails: Towards improved safety alignment of llm via a human-preference dataset , author=. Advances in Neural Information Processing Systems , volume= on Pith
  45. [65] Cvalues: Measuring the values of chinese large language models from safety to responsibility , author=. arXiv preprint arXiv:2307.09705 , year= on PithPith's reviewarXiv
  46. [66] Xstest: A test suite for identifying exaggerated safety behaviours in large language models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages= on Pith
  47. [67] Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models , author=. arXiv preprint arXiv:2307.08487 , year= on PithPith's reviewarXiv
  48. [68] " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models , author=. Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pages= on Pith
  49. [69] Do-not-answer: Evaluating safeguards in LLMs , author=. Findings of the Association for Computational Linguistics: EACL 2024 , pages= on Pith
  50. [70] All languages matter: On the multilingual safety of large language models , author=. arXiv preprint arXiv:2310.00905 , year= on PithPith's reviewarXiv
  51. [71] Salad-bench: A hierarchical and comprehensive safety benchmark for large language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages= on Pith
  52. [72] Sorry-bench: Systematically evaluating large language model safety refusal , author=. arXiv preprint arXiv:2406.14598 , year= on PithPith's reviewarXiv
  53. [73] Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms , author=. Advances in neural information processing systems , volume= on Pith
  54. [74] SMART: Self-aware agent for tool overuse mitigation , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages= on Pith
  55. [75] Toward a theory of agents as tool-use decision-makers , author=. arXiv preprint arXiv:2506.00886 , year= on PithPith's reviewarXiv
  56. [76] Over-Searching in Search-Augmented Large Language Models , author=. 2026 , eprint= on Pith
  57. [77] Clarify when necessary: Resolving ambiguity through interaction with lms , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages= on Pith
  58. [78] CLAMBER: A benchmark of identifying and clarifying ambiguous information needs in large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  59. [79] Active task disambiguation with llms , author=. arXiv preprint arXiv:2502.04485 , year= on PithPith's reviewarXiv
  60. [80] Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities , author=. arXiv preprint arXiv:2602.05073 , year= on PithPith's reviewarXiv
  61. [81] Position: Uncertainty quantification needs reassessment for large-language model agents , author=. arXiv preprint arXiv:2505.22655 , year= on PithPith's reviewarXiv
  62. [82] Flashrag: A modular toolkit for efficient retrieval-augmented generation research , author=. Companion Proceedings of the ACM on Web Conference 2025 , pages= on Pith
  63. [83] Text embeddings by weakly-supervised contrastive pre-training , author=. arXiv preprint arXiv:2212.03533 , year= on PithPith's reviewarXiv
  64. [84] MemGPT: Towards LLMs as Operating Systems , author=. 2024 , eprint= on Pith
  65. [85] React: Synergizing reasoning and acting in language models , author=. The eleventh international conference on learning representations , year= on Pith
  66. [86] Voyager: An open-ended embodied agent with large language models , author=. arXiv preprint arXiv:2305.16291 , year= on PithPith's reviewarXiv
  67. [87] MARVEL: Modular Abstention for Reliable and Versatile Expert LLMs , author= on Pith
  68. [88] Characterizing LLM abstention behavior in science QA with context perturbations , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages= on Pith
  69. [89] DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responses , author=. arXiv preprint arXiv:2512.02282 , year= on PitharXiv
  70. [90] Qwen3 Technical Report , author=. 2025 , eprint= on Pith
  71. [91] GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint= on Pith
  72. [92] Gemma 4 31B IT , year = on Pith
  73. [93] Agentic context engineering: Evolving contexts for self-improving language models , author=. arXiv preprint arXiv:2510.04618 , year= on PithPith's reviewarXiv
  74. [94] Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume= on Pith
  75. [95] Optimizing generative ai by backpropagating language model feedback , author=. Nature , volume=. 2025 , publisher= on Pith
  76. [96] Dynamic cheatsheet: Test-time learning with adaptive memory , author=. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  77. [97] Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety , author=. arXiv preprint arXiv:2510.16492 , year= on PithPith's reviewarXiv
  78. [98] Agentic Abstention: Do Agents Know When to Stop Instead of Act? , author=. arXiv preprint arXiv:2606.28733 , year= on PithPith's reviewarXiv
  79. [99] AgentAbstain: Do LLM Agents Know When Not to Act? , author=. arXiv preprint arXiv:2607.10059 , year= on PithPith's reviewarXiv
  80. [100] Dspy: Compiling declarative language model calls into self-improving pipelines , author=. arXiv preprint arXiv:2310.03714 , year= on PithPith's reviewarXiv

The first 100 of 129 references are listed here. All of them are in this paper's citation record.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment