Robotics · arXiv:2611.17020 v1 · submitted 12 Nov 2026 · 78 references
Maximizing LLM Utility While Managing LLM Fallibility
The paper says
“Large language models (LLMs) can supply autonomous systems with plans, predictions, and judgments, but these contributions are sometimes wrong. This review asks how a system can maximize an LLM's utility while managing its fallibility.”
more of the abstract
Our answer is to adapt the system's reliance on the LLM as evidence about the LLM's performance accumulates. Systems built from imperfect components, such as noisy sensors and faulty software, have long worked in this way, and the classical methods they use can be applied to LLM-based systems as well. A system can change its reliance on an LLM in several ways, depending on the LLM's role and on the other components in the system. Each change needs an estimate of the LLM's reliability, and the LLM's role and the kind of contribution it makes determine what to estimate. Classical methods from decision theory, estimation, and online learning then turn these estimates into reliance decisions. These decisions can reduce how often LLM errors affect the system. However, some errors matter even when they are rare. An independently verified mechanism can prevent such errors from being executed. Without one, adapting reliance using reliability estimates can still keep their risk within a bound. Studies on adapting reliance on LLMs are emerging but limited, and the reviewed comparisons suggest that adapting reliance can improve system performance.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A competent, honest review whose contribution is its taxonomy of LLM roles, reliance dimensions, reliability questions, and mapped classical methods; the transfer-from-classical-methods claim is plausible but empirically thin. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 78 references say.
The paper says, at Section 4 and Section 6
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.66):
“Second, a reliability estimate holds only for the LLM, the machine, and the kinds of inputs that produced its evidence.”
Pith says: Objection 1 of 3 · major
The review's central claim—that classical adaptive methods transfer to LLM-based systems—rests on an analogy that is asserted rather than argued in depth. The introduction states that LLM contributions are 'uncertain, context-dependent, and non-stationary' and that classical methods 'can be applied', and Section 4's four caveats (selective labels, input-regime specificity, correlated errors, rare large errors) are exactly the places where the analogy could break down.
more of objection 1
I do not think this is a fatal flaw for a review, and the authors do acknowledge the transfer risk; however, the paper would be strengthened by an explicit statement of the conditions under which the analogy holds (e.g., availability of a reference for the reliability quantity, outcome observability, error distributions that are estimable from the system's own experience) and by a sentence in the synthesis that marks the limits of the analogy. As written, the reader must assemble these conditions from Sections 3 and 4 themselves.
The paper says, at Section 5B
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The risk-bound claim ('adapting reliance using reliability estimates can still keep their risk within a bound') is presented in the abstract and Section 5B with only one example, KnowNo, and the text correctly notes that the guarantee holds 'under the conformal prediction assumptions'.
more of objection 2
The manuscript should state more explicitly that this is the only reviewed instance of the risk-bound claim, that the bound is on the coverage of a set of candidate actions rather than on task-level harm (the robot can still ask the user, and the guarantee is about correct-action coverage), and that no reviewed study demonstrates a risk bound for the more general case of reliability-estimate-driven adaptation. This is a load-bearing point because the abstract's risk-bound statement is broader than the evidence reviewed.
The paper says, at Section 4
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The discussion of selective labels (paragraph beginning 'To keep the estimates accurate') correctly identifies the problem and proposes continued exploration, random sampling, and propensity weighting. However, the paper does not note that propensity weighting alone does not fix the selection problem when exploration is not randomized or when the assignment policy depends on unobserved covariates; this is a known condition in the counterfactual-learning literature cited (Swaminathan and Joachims).
more of objection 3
A one-sentence qualification would make the advice accurate as stated, since the text says propensity weighting 'makes the reliability estimates represent all inputs' without stating the overlap/positivity and ignorability conditions. This is a minor-to-moderate precision issue, but it is located in the paper's practical guidance, so I flag it.
5 minor objections
The paper says, at Abstract and Section 1
Pith says: minor objection 1 of 5
The abstract says 'some errors matter even when they are rare' and Section 5 addresses this, but the distinction between 'independently verified mechanism' and 'fallible check' is introduced only in Section 5; a forward reference in the introduction would help readers who are deciding whether the framework applies to their setting.
The paper says, at Section 5B
Pith says: minor objection 2 of 5
The KnowNo example is described correctly, but the sentence 'the calibrated threshold thus determines when the robot relies on the LLM and when it seeks help' could note explicitly that asking the user is itself a reliance decision, linking the example back to Table III's assignment dimension; this would strengthen the coherence of the framework.
show 3 more
The paper says, at References
Pith says: minor objection 3 of 5
Several key sources are cited as arXiv preprints (CALIPER [19], CABS-D [20], ParetoBandit [59], and the safety survey [29]) rather than peer-reviewed publications. This is appropriate for an emerging area and I do not object, but the text should be consistent in marking which reviewed comparisons rest on preprints, since the synthesis says 'studies so far show that adapting reliance can improve system performance' without distinguishing the strength of the evidence.
The paper says, at Table IV
Pith says: minor objection 4 of 5
The row for 'Evaluator, Numerical estimate of a physical quantity' lists quantities 'bias, repeatability, uncertainty' but the reliability question asks only 'How does the estimate compare with a reference measurement'; a brief note on how to combine these into a single reliance decision (e.g., via a fused estimate or an acceptance threshold) would make the table's connection to Table V clearer.
The paper says, at Throughout
Pith says: minor objection 5 of 5
The paper uses 'reliability estimate' to cover both frequentist rates (valid-plan rate) and probabilistic quantities (calibrated confidence); a short terminological note at the start of Section 3 would prevent readers from expecting a single definition of reliability across all roles.
What would settle it
Re-run the Section 5B KnowNo-style conformal calibration on a deployment set drawn from a shifted distribution and compare the empirical failure rate with the specified conformal bound; if the bound fails materially, the abstract's risk-bound sentence would need an explicit distribution-shift qualifier.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should read this if you work on LLM-based autonomy in any capacity: it is the rare review whose tables are the contribution. Samani and Atkeson organize the space as eight LLM functional roles, six dimensions of reliance change (assignment, influence, invocation, oversight, persistence, latitude), reliability questions per role and interface, and classical methods mapped onto each dimension. Table III alone will save people time they currently spend rediscovering these distinctions.
The paper is honest in the right places. It states the four things that can break reliability estimates: selective labels (b224–b225), regime shift (b232), correlated LLM errors (b243), and rare large errors (b246). The refinement-loop example is well chosen: 40/100 valid single-shot plans versus 88/100 with the validator in the loop, which makes the point that reliability must be estimated from the contribution as produced, before other components act on it. Section 5 is the clearest part: it separates reducing error frequency from preventing rare critical errors, and it says plainly that without an independently verified mechanism the best you get is a risk bound under the conformal assumptions of the KnowNo-style calibration.
What is not here: any new technique, derivation, or data. It is a review by design, so I am not holding that against it. The softer spot is that the central claim—adapting reliance improves performance—rests on roughly six emerging studies, several of which use simpler rules than the classical machinery the paper maps onto them (BORA's ask-when-stalled rule is not value-of-information; the paper's own Table V says so). The paper is open about this being early, but the reader should not take the payoff as established; it is suggestive. The load-bearing assumption remains the sensor analogy: LLM errors are treated as being enough like noisy sensors and faulty software that classical methods transfer. The paper lists the conditions under which that fails, which I count as acknowledgement, but there is not yet much empirical evidence about how often those conditions hold in practice. That gap is a feature of the field, not a defect of this review.
Verdict: it deserves a serious referee and, in a venue that takes reviews, likely acceptance. The reviewer's job is to check fidelity of the cited results and coherence of the taxonomy, not to hunt for a missing proof. I would cite it as the entry point on adaptive reliance, and it could make a decent reading-group piece to argue with about the analogy.
The strongest objection
The paper's central claim is an existence/conditional claim—expressed as 'can reduce' and 'can keep risk within a bound'—rather than a universal theorem, and the conditions on which it depends are stated in the text. Reliability estimates require a trustworthy reference (b99), hold only for the tested input regime (b232), suffer from selective labels (b224, b225), correlated LLM errors (b243), and rare large errors (b246). The risk-bound claim in Section 5B is explicitly qualified by the conformal prediction assumptions of the KnowNo example. The transfer-from-classical-methods concern raised by the reader is real, but the paper acknowledges it and provides concrete mitigations, including continued exploration, propensity weighting, discounted bandits, and robust filters. I therefore do not find an unacknowledged assumption on which the central claim silently depends.
The reader accepted the paper with high confidence, and my independent pass agrees. The most exposed point is the abstract's statement that without an independently verified mechanism, adaptive reliance on reliability estimates can still keep risk within a bound. The body, however, immediately narrows this to the KnowNo conformal setting with stated assumptions, so the claim is not presented as unconditional. The other potentially load-bearing concern—that LLM errors differ from noisy sensors and faulty software—is acknowledged in the paper's own limitations and addressed with specific mechanisms rather than ignored. The paper is a review, so the absence of a new theorem or large-scale empirical study lowers novelty but does not make the central argument unsound. No internally inconsistent step or missing support for a cited result was found. The concrete test I propose would stress the one place where a reader might over-read the abstract, namely whether the risk bound survives distribution shift, but the paper already flags this by requiring inputs similar to those tested and by citing conformal assumptions.
The referee's report
The paper is a synthesis/review rather than a new empirical result, but it makes a useful and nontrivial conceptual contribution: it identifies a coherent design pattern—adaptive reliance based on accumulated performance evidence—and maps it across a broad space of LLM roles and system configurations. Its strengths are that it connects disparate emerging systems (BORA, CALIPER, LABO, KnowNo, routing and verification work) under one framework, that it is careful to state the conditions under which the approach works (a trustworthy reference for reliability estimates, suitable other components, continued exploration, and handling of correlated and rare errors), and that it is honest about what is not yet proven. The tables are a practical contribution for researchers building LLM-based systems, and the review's falsifiable organizing claim—that adaptive reliance can improve system performance—is supported by the cited comparisons, all of which the authors report accurately.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that an autonomous system should treat a large language model the way it already treats noisy sensors and faulty software: as an imperfect component whose reliability must be estimated and re-estimated, with the system adjusting how much it relies on the model as that evidence accumulates. It organizes the ways a system can change reliance into six dimensions across eight functional roles an LLM can play, and pairs each combination with a reliability quantity to estimate and a classical method for turning that estimate into a decision. The central claim is that such adaptive reliance reduces how often LLM errors affect the system, and that when errors matter even when rare, an independently verified mechanism can stop them from executing, or the system can keep their risk within a bound. The review matters because it reframes LLM fallibility as a component-reliability problem with a long toolkit behind it, rather than a new and unmanageable failure mode.
What carries the argument. The central device is a two-axis taxonomy linking functional roles of LLM contributions to dimensions of reliance change, with a reliability question assigned to each cell. Along one axis are the eight jobs an LLM can do in a system; along the other are the six ways a system can change how much it leans on the LLM: which source supplies the contribution (assignment), how much weight it gets (influence), when to ask for a new contribution (invocation), whether to check it first (oversight), whether a contribution in use stays in use (persistence), and which options the LLM may choose from (latitude). The role and the contribution interface fix what reliability quantity to estimate, and classical methods convert the estimate into a decision: adaptive Kalman or Bayesian fusion for influence, contextual and discounted bandits for assignment, value of information for invocation and oversight, sequential decision analysis for persistence, and trust-region-style resizing for latitude. The Blocksworld planning example, where a GPT-4 planner and a GPT-4 evaluator are each measured against a symbolic validator, carries the argument by showing how the estimates enter a cost-benefit calculation.
Core claim. The paper's central assertion is that a system's reliance on an LLM should not be fixed in advance but should adapt as evidence about the LLM's performance accumulates: assign a hard problem to a different source when the LLM's track record is poor, weight the LLM's contribution less when its errors accumulate, check its plans more often when they fail more often, and stop asking it when it stops helping. The machinery is a taxonomy: the LLM works in one of eight functional roles (formulator, candidate proposer, search shaper, consequence modeler, evaluator, selector or planner, coordinator, controller), and the system can change reliance along six dimensions (assignment, influence, invocation, oversight, persistence, latitude). Each role and contribution interface determines which reliability quantity to estimate, from a valid-plan rate for a planner to false-acceptance and false-rejection rates for an evaluator to prediction error for a consequence modeler, and established methods from decision theory, estimation, and online learning then turn those estimates into reliance decisions. The paper's worked Blocksworld example makes the logic concrete: a GPT-4 evaluator that accepts 38 of 45 invalid plans would prevent only seven invalid executions at the cost of 100 checks and one wrongly rejected valid plan, so whether checking pays off is a decision theory problem, not a matter of faith in the model. The synthesis claim is that adapting reliance can improve system performance and reduce how often LLM errors affect the system, while acknowledging that for errors that matter even when rare, adaptation alone can only bound risk and an independently verified mechanism is what prevents execution.
The premise it rests on. The argument rests on the premise that LLM failures are similar enough to noisy-sensor and faulty-software failures that classical adaptive methods transfer without major modification: if LLM errors are instead highly correlated across contexts or dominated by rare semantic failures that average reliability estimates cannot capture, the reduction in error incidence that the paper predicts may not materialize.
If the paper is right (Pith's inferences, not the authors')
If the central claim holds, a system designer's first question about any LLM component becomes empirical: what is its reliability on the inputs it will actually meet, measured against a reference, before the system decides how much to trust it.
Adaptation should be the default posture: systems that decide in advance to always use an LLM's contribution will underperform systems that test the contribution against observed outcomes and abstain when it hurts, as the CALIPER comparison illustrates.
The same data used to improve the LLM through prompting or fine-tuning can simultaneously be used to change reliance on it, so feedback loops serve two purposes at once.
Relying less on an LLM is only possible when other components can take over, so the value of adaptive reliance is bounded by the quality and availability of fallback sources, fallback controllers, validators, and humans.
For safety-critical contributions, reliance estimates alone are insufficient; preventing rare but serious errors requires an independently verified mechanism such as a validator, shield, control barrier function, or runtime monitor, and adaptation can only keep risk within a bound without one.
A reliability estimate is only usable if evidence is collected on inputs like those in use and on every alternative source, which means deliberately routing some inputs to alternatives and checking some contributions even when error rates are low.
A system's own decisions create gaps in what it observes: stopping checks after plans prove valid hides later degradation, and rarely using a low-reliability LLM hides improvement, so evidence collection must be designed around the selective-labels problem.
Estimates built from agreement between LLMs overstate reliability when the models share training data or context, so correlation of errors must be modeled rather than assumed away.
The framework implies that a large fraction of LLM integration work is really an instrumentation problem: the bottleneck is not the model but the quality of the reference signals used to estimate it, so progress will come from better validators, better simulators with characterized error, and better logging of original contributions before any other component acts on them. This is the authors' dir
The taxonomy suggests a design pattern: for any new LLM role a system designer wants to adopt, the first step is to place it on the role axis, pick which reliance dimensions are available given existing components, and then determine the cheapest reference that yields a usable reliability estimate. This is what the paper enables but does not itself spell out as a procedure.
A testable extension that follows from the argument: if the reliability estimates are accurate and the fallback components are decent, then a system using adaptive reliance should dominant a fixed-reliance baseline in expected cost, and this could be verified in a controlled benchmark where LLM error rates are known and shift mid-run.
An open question the authors flag but do not close is combining observed performance with prior domain knowledge about the LLM's contributions; a natural next step is a Bayesian prior over reliability that gets updated by outcomes, which the paper's own framework suggests but leaves to future work.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Large language models (LLMs) can supply autonomous systems with plans, predictions, and judgments, but these contributions are sometimes wrong. This review asks how a system can maximize an LLM's utility while managing its fallibility. Our answer is to adapt the system's reliance on the LLM as evidence about the LLM's performance accumulates. Systems built from imperfect components, such as noisy sensors and faulty software, have long worked in this way, and the classical methods they use can be applied to LLM-based systems as well. A system can change its reliance on an LLM in several ways, depending on the LLM's role and on the other components in the system. Each change needs an estimate of the LLM's reliability, and the LLM's role and the kind of contribution it makes determine what to estimate. Classical methods from decision theory, estimation, and online learning then turn these estimates into reliance decisions. These decisions can reduce how often LLM errors affect the system. However, some errors matter even when they are rare. An independently verified mechanism can prevent such errors from being executed. Without one, adapting reliance using reliability estimates can still keep their risk within a bound. Studies on adapting reliance on LLMs are emerging but limited, and the reviewed comparisons suggest that adapting reliance can improve system performance.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 0.51
“Our answer is to adapt the system's reliance on the LLM as evidence about the LLM's performance accumulates.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.69 · human participants 0.04 · controlled comparison 0.24 · randomized assignment 0.02 · states quantitative result 0.07 · compares with prior results 0.64 · mentions code or data availability 0.02
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Large language models (LLMs) can supply autonomous systems with plans, predictions, and judgments, but these contributions are sometimes wrong. | background 1.00 | 0.06 | 0.02 | hedged possibility 0.52 |
| This review asks how a system can maximize an LLM's utility while managing its fallibility. | contribution 0.45 | 0.17 | 0.06 | unclear 0.62 |
| Our answer is to adapt the system's reliance on the LLM as evidence about the LLM's performance accumulates. | contribution 0.76 | 0.58 | 0.86 | asserted 1.00 |
| Systems built from imperfect components, such as noisy sensors and faulty software, have long worked in this way, and the classical methods they use can be applied to LLM-based systems as well. | background 0.97 | 0.05 | 0.08 | asserted 0.98 |
| A system can change its reliance on an LLM in several ways, depending on the LLM's role and on the other components in the system. | method 0.68 | 0.14 | 0.05 | asserted 0.66 |
| Each change needs an estimate of the LLM's reliability, and the LLM's role and the kind of contribution it makes determine what to estimate. | method 0.90 | 0.19 | 0.07 | asserted 1.00 |
| Classical methods from decision theory, estimation, and online learning then turn these estimates into reliance decisions. | method 0.90 | 0.08 | 0.10 | asserted 0.99 |
| These decisions can reduce how often LLM errors affect the system. | result 0.58 | 0.13 | 0.14 | hedged possibility 0.73 |
| However, some errors matter even when they are rare. | limitation 0.50 | 0.07 | 0.03 | asserted 0.99 |
| An independently verified mechanism can prevent such errors from being executed. | method 0.72 | 0.09 | 0.07 | asserted 0.87 |
| Without one, adapting reliance using reliability estimates can still keep their risk within a bound. | interpretation 0.37 | 0.18 | 0.13 | hedged possibility 0.65 |
| Studies on adapting reliance on LLMs are emerging but limited, and the reviewed comparisons suggest that adapting reliance can improve system performance. | result 0.58 | 0.26 | 0.43 | hedged possibility 0.97 |
What kind of paper the classifier reports this is: review 1.00 · evidence from literature 0.91
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,636 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (1)
References (78)
- [1] A. Zenget al., “Socratic Models: Composing zero-shot multimodal reasoning with language,” inInt. Conf. Learn. Rep., 2023 on Pith
- [2] I. Singhet al., “ProgPrompt: Generating situated robot task plans using large language models,” inIEEE Int. Conf. Rob. Autom., 2023, pp. 11 523–11 530 on Pith
- [3] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” inInt. Conf. Mach. Learn., 2022, pp. 9118–9147 on Pith
- [4] Z. Liu, A. Bahety, and S. Song, “REFLECT: Summarizing robot experiences for failure explanation and correction,” inConf. Robot Learn., 2023, pp. 3468–3484 on Pith
- [5] Y . Kim, R. Arora, R. Martín-Martín, P. Stone, B. Abbatematteo, and Y . Sung, “Large-language-model-guided state estimation for partially observable task and motion planning,” inIEEE Int. Conf. Rob. Autom., 2026, pp. 6613–6620 on Pith
- [6] J. C. Knight and N. G. Leveson, “An experimental evaluation of the assumption of independence in multiversion programming,”IEEE Trans. Softw. Eng., vol. SE-12, no. 1, pp. 96–109, 1986 on Pith
- [7] R. Mehra, “On the identification of variances and adaptive Kalman filtering,”IEEE Trans. Autom. Control, vol. AC-15, no. 2, pp. 175–184, 1970 on Pith
- [8] L. Li, W. Chu, J. Langford, and R. E. Schapire, “A contextual-bandit approach to personalized news article recommendation,” inInt. Conf. World Wide Web, 2010, pp. 661–670 on Pith
- [9] A. Garivier and E. Moulines, “On upper-confidence bound policies for switching bandit problems,” inInt. Conf. Algorithmic Learn. Theory. Springer, 2011, pp. 174–188 on Pith
- [10] B. Ichteret al., “Do as I can, not as I say: Grounding language in robotic affordances,” inConf. Robot Learn., 2023, pp. 287–318 on Pith
- [11] OpenAI, “GPT-5.6—August Updates.” [Online]. Available: https: //deploymentsafety.openai.com/gpt-5-6-august-update/model-safety-tra ining-and-evaluation on Pith
- [12] K. Valmeekam, M. Marquez, and S. Kambhampati, “Can large language models really improve by self-critiquing their own plans?” inNeurIPS Workshop Found. Models Decis. Making, 2023 on Pith
- [13] W. Huanget al., “Inner Monologue: Embodied reasoning through planning with language models,” inConf. Robot Learn., 2023, pp. 1769– 1782 on Pith
- [14] K. Kobalczyk, Z. J. Lin, B. Letham, Z. Zhao, M. Balandat, and E. Bakshy, “LILO: Bayesian optimization with natural language feedback,” inInt. Conf. Mach. Learn., 2026, pp. 59 643–59 693 on Pith
- [15] L. Boyle, N. Baumann, P. Sivasothilingam, M. Magno, and L. Benini, “RobotxR1: Enabling embodied robotic intelligence on large language models through closed-loop reinforcement learning,” inConf. Robot Learn., 2025, pp. 4074–4092 on Pith
- [16] L. Ouyanget al., “Training language models to follow instructions with human feedback,” inAdv. Neural Inf. Process. Syst., 2022, pp. 27 730– 27 744 on Pith
- [17] A. Cissé, X. Evangelopoulos, V . V . Gusev, and A. I. Cooper, “Language- based Bayesian optimization research assistant (BORA),” inInt. Joint Conf. Artif. Intell., 2025, pp. 4967–4975 on Pith
- [18] Z. Chenet al., “LABO: LLM-accelerated Bayesian optimization through broad exploration and selective experimentation,” inInt. Conf. Mach. Learn., 2026, pp. 17 979–18 005 on Pith
- [19] J. Chen, Y . Ban, and T. Fu, “CALIPER: Evidence-gated LLM prior layers for multi-objective Bayesian optimization,”arXiv preprint arXiv:2606.01730v2, 2026 on PithPith's reviewarXiv
- [20] A. N. Sridhar, R. Singh, M. Mahdavi, and V . Narayanan, “Correlation- aware contextual bandits with surrogate rewards for LLM routing,”arXiv preprint arXiv:2607.09015v1, 2026 on PithPith's reviewarXiv
show 58 more
- [21] A. Paudel, A. Khanal, R. I. Arnob, S. Hossain, and G. J. Stein, “Object search in partially-known environments via LLM-informed model-based planning and prompt selection,” inIEEE/RSJ Int. Conf. Intell. Robots Syst., 2026 on Pith
- [22] S. Kiyani, S. Noorani, G. J. Pappas, and H. Hassani, “When to trust the cheap check: Weak and strong verification for reasoning,” inInt. Conf. Mach. Learn., 2026, pp. 59 388–59 414 on Pith
- [23] R. Firooziet al., “Foundation models in robotics: Applications, challenges, and the future,”Int. J. Robot. Res., vol. 44, no. 5, pp. 701–739, 2025 on Pith
- [24] O. Shorinwa, Z. Mei, J. Lidard, A. Z. Ren, and A. Majumdar, “A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions,”ACM Comput. Surv., vol. 58, no. 3, p. 63, 2026 on Pith
- [25] H. Wei, Z. Zhang, S. He, T. Xia, S. Pan, and F. Liu, “PlanGenLLMs: A modern survey of LLM planning capabilities,” inAnnu. Meeting Assoc. Comput. Linguistics, 2025, pp. 19 497–19 521 on Pith
- [26] M. Tantakoun, C. Muise, and X. Zhu, “LLMs as planning formalizers: A survey for leveraging large language models to construct automated planning models,” inFindings Assoc. Comput. Linguistics: ACL, 2025, pp. 25 167–25 188 on Pith
- [27] Y . Moslem and J. D. Kelleher, “Dynamic model routing and cascading for efficient LLM inference: A survey,”Trans. Mach. Learn. Res., 2026 on Pith
- [28] T. Kojimaet al., “A comprehensive survey on physical risk control in the era of foundation model-enabled robotics,” inInt. Joint Conf. Artif. Intell., 2025, pp. 10 517–10 527 on Pith
- [29] P. Dantas, L. Cordeiro, E. Nowroozi, and N. Tihanyi, “Toward safe LLM agents: A survey of specification, verification, and enforcement,”arXiv preprint arXiv:2608.14590v1, 2026 on PithPith's reviewarXiv
- [30] Y . J. Maet al., “Eureka: Human-level reward design via coding large language models,” inInt. Conf. Learn. Rep., 2024, pp. 26 516–26 560 on Pith
- [31] B. Liuet al., “LLM+P: Empowering large language models with optimal planning proficiency,”arXiv preprint arXiv:2304.11477, 2023 on PithPith's reviewarXiv
- [32] J. Lianget al., “Code as Policies: Language model programs for embodied control,” inIEEE Int. Conf. Rob. Autom., 2023, pp. 9493–9500 on Pith
- [33] W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxPoser: Composable 3D value maps for robotic manipulation with language models,” inConf. Robot Learn., 2023, pp. 540–562 on Pith
- [34] W. F. Reinhart and A. Statt, “Large language models design sequence- defined macromolecules via evolutionary optimization,”npj Comput. Mater., vol. 10, p. 262, 2024 on Pith
- [35] E. U. Samani and C. G. Atkeson, “Programming manufacturing robots with imperfect AI: LLMs as tuning experts for FDM print configuration selection,”arXiv preprint arXiv:2603.22118v1, 2026 on PithPith's reviewarXiv
- [36] A. Z. Renet al., “Robots that ask for help: Uncertainty alignment for large language model planners,” inConf. Robot Learn., 2023, pp. 661–682 on Pith
- [37] D. Jiang, X. Ren, and B. Y . Lin, “LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion,” inAnnu. Meeting Assoc. Comput. Linguistics, 2023, pp. 14 165–14 178 on Pith
- [38] A. Hafez, A. N. Akhormeh, A. Hegazy, and A. Alanwar, “Safe LLM- controlled robots with formal guarantees via reachability analysis,”arXiv preprint arXiv:2503.03911v1, 2025 on PithPith's reviewarXiv
- [39] Z. Wanget al., “Open-loop planning, closed-loop verification: Speculative verification for VLA,” inACM Int. Conf. Multimedia, 2026, pp. 11 220– 11 229 on Pith
- [40] Q. Meng, J. P. Zhou, K. Q. Weinberger, and H. Kress-Gazit, “INPROVF: Leveraging large language models to repair high-level robot controllers from assumption violations,” inIEEE Int. Conf. Autom. Sci. Eng., 2025, pp. 1680–1687 on Pith
- [41] J. Zhao, C. Shin, T.-H. Huang, S. S. S. Namburi GNVV , and F. Sala, “CARE: Confounder-aware aggregation for reliable LLM evaluation,” in Int. Conf. Mach. Learn., 2026, pp. 162 114–162 150. 9 on Pith
- [42] Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-Eval: NLG evaluation using GPT-4 with better human alignment,” inConf. Empirical Methods Natural Lang. Process., 2023, pp. 2511–2522 on Pith
- [43] P. Zave and M. Jackson, “Four dark corners of requirements engineering,” ACM Trans. Softw. Eng. Methodol., vol. 6, no. 1, pp. 1–30, 1997 on Pith
- [44] J. J. Shah, N. Vargas-Hernandez, and S. M. Smith, “Metrics for measuring ideation effectiveness,”Design Stud., vol. 24, no. 2, pp. 111–134, 2003 on Pith
- [45] J. K. Pugh, L. B. Soros, and K. O. Stanley, “Quality diversity: A new frontier for evolutionary computation,”Front. Robot. AI, vol. 3, p. 40, 2016 on Pith
- [46] L. Ljung, “Perspectives on system identification,”Annu. Rev. Control, vol. 34, no. 1, pp. 1–12, 2010 on Pith
- [47] M. C. Kennedy and A. O’Hagan, “Bayesian calibration of computer models,”J. Roy. Statist. Soc. Ser. B, vol. 63, no. 3, pp. 425–464, 2001 on Pith
- [48] T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,”J. Amer. Statist. Assoc., vol. 102, no. 477, pp. 359–378, 2007 on Pith
- [49] T. Gneiting, F. Balabdaoui, and A. E. Raftery, “Probabilistic forecasts, calibration and sharpness,”J. Roy. Statist. Soc. Ser. B, vol. 69, no. 2, pp. 243–268, 2007 on Pith
- [50] A. P. Dawid, “Present position and potential developments: Some personal views: Statistical theory: The prequential approach,”J. Roy. Statist. Soc. Ser. A (Gen.), vol. 147, no. 2, pp. 278–292, 1984 on Pith
- [51] M. G. Kendall, “A new measure of rank correlation,”Biometrika, vol. 30, no. 1–2, pp. 81–93, 1938 on Pith
- [52] Joint Committee for Guides in Metrology,Evaluation of measurement data—Guide to the expression of uncertainty in measurement, 2008, JCGM100:2008(E) on Pith
- [53] R. Howey, D. Long, and M. Fox, “V AL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL,” inIEEE Int. Conf. Tools Artif. Intell., 2004, pp. 294–301 on Pith
- [54] D. Shah, B. Osi ´nski, B. Ichter, and S. Levine, “LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action,” inConf. Robot Learn., 2023, pp. 492–504 on Pith
- [55] Y . Jadhav, P. Pak, and A. Barati Farimani, “LLM-3D print: Large language models to monitor and control 3D printing,”Addit. Manuf., vol. 114, p. 105027, 2025 on Pith
- [56] J. R. Rice, “The algorithm selection problem,” inAdvances in Computers. Academic Press, 1976, vol. 15, pp. 65–118 on Pith
- [57] I. Onget al., “RouteLLM: Learning to route LLMs from preference data,” inInt. Conf. Learn. Rep., 2025, pp. 34 433–34 448 on Pith
- [58] P. Auer, N. Cesa-Bianchi, Y . Freund, and R. E. Schapire, “The nonstochastic multiarmed bandit problem,”SIAM J. Comput., vol. 32, no. 1, pp. 48–77, 2002 on Pith
- [59] A. Taberner-Miller, “ParetoBandit: Budget-paced adaptive routing for non-stationary LLM serving,”arXiv preprint arXiv:2604.00136v1, 2026 on PithPith's reviewarXiv
- [60] P. Scerri, D. V . Pynadath, and M. Tambe, “Towards adjustable autonomy for the real world,”J. Artif. Intell. Res., vol. 17, pp. 171–228, 2002 on Pith
- [61] R. Parasuraman, T. B. Sheridan, and C. D. Wickens, “A model for types and levels of human interaction with automation,”IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 30, no. 3, pp. 286–297, 2000 on Pith
- [62] N. Littlestone and M. K. Warmuth, “The weighted majority algorithm,” Inf. Comput., vol. 108, no. 2, pp. 212–261, 1994 on Pith
- [63] Y . Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,”J. Comput. Syst. Sci., vol. 55, no. 1, pp. 119–139, 1997 on Pith
- [64] R. E. Kalman, “A new approach to linear filtering and prediction problems,”J. Basic Eng., vol. 82, no. 1, pp. 35–45, 1960 on Pith
- [65] C. J. Masreliez and R. D. Martin, “Robust Bayesian estimation for the linear model and robustifying the Kalman filter,”IEEE Trans. Autom. Control, vol. AC-22, no. 3, pp. 361–371, 1977 on Pith
- [66] R. A. Howard, “Information value theory,”IEEE Trans. Syst. Sci. Cybern., vol. SSC-2, no. 1, pp. 22–26, 1966 on Pith
- [67] J. O. Berger,Statistical Decision Theory and Bayesian Analysis, 2nd ed. Springer, 1985 on Pith
- [68] A. R. Conn, N. I. M. Gould, and P. L. Toint,Trust-Region Methods. SIAM, 2000 on Pith
- [69] X. Wan, V . Nguyen, H. Ha, B. Ru, C. Lu, and M. A. Osborne, “Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces,” inInt. Conf. Mach. Learn., 2021, pp. 10 663–10 674 on Pith
- [70] A. Schwankeet al., “Improving LLM-based global optimization with search space partitioning,” inInt. Conf. Learn. Rep., 2026, pp. 76 314– 76 344 on Pith
- [71] H. Lakkaraju, J. Kleinberg, J. Leskovec, J. Ludwig, and S. Mullainathan, “The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables,” inACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2017, pp. 275–284 on Pith
- [72] N. Alon, N. Cesa-Bianchi, O. Dekel, and T. Koren, “Online learning with feedback graphs: Beyond bandits,” inConf. Learn. Theory, 2015, pp. 23–35 on Pith
- [73] A. Swaminathan and T. Joachims, “Counterfactual risk minimization: Learning from logged bandit feedback,” inInt. Conf. Mach. Learn., 2015, pp. 814–823 on Pith
- [74] J.-A. Ting, E. Theodorou, and S. Schaal, “A Kalman filter for robust outlier detection,” inIEEE/RSJ Int. Conf. Intell. Robots Syst., 2007, pp. 1514–1519 on Pith
- [75] S. Kambhampatiet al., “Position: LLMs can’t plan, but can help planning in LLM-modulo frameworks,” inInt. Conf. Mach. Learn., 2024, pp. 22 895–22 907 on Pith
- [76] M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu, “Safe reinforcement learning via shielding,” inAAAI Conf. Artif. Intell., 2018, pp. 2669–2678 on Pith
- [77] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada, “Control barrier function based quadratic programs for safety critical systems,”IEEE Trans. Autom. Control, vol. 62, no. 8, pp. 3861–3876, 2017 on Pith
- [78] K. L. Hobbs, M. L. Mote, M. C. L. Abate, S. D. Coogan, and E. M. Feron, “Runtime assurance for safety-critical systems: An introduction to safety filtering approaches for complex control systems,”IEEE Control Syst., vol. 43, no. 2, pp. 28–65, 2023 on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.

No comments yet.
Sign in to comment