Pith.

Multiagent Systems · arXiv:2610.13870 v1 · submitted 9 Oct 2026 · 53 references

Ask Your Advisor: Learning When to Consult Stronger Models in Agentic Systems

Pascal Sowietzki, Steffen Holter, Yannick Metz, Emil Kasper, Raphael Baur, Mennatallah El-Assady

The paper says

“To balance task performance and computational cost, agentic workflows can use a lower-cost model for execution and selectively consult a stronger model.”

more of the abstract

In such executor-advisor systems, the executor decides when to seek advice, making escalation depend on self-assessment, although current language models struggle to detect where their own reasoning fails. The need for stronger reasoning emerges during execution and varies across workflow phases. We introduce ARIA, a meta-controller that learns when to consult by balancing task success against consultation cost. Our approach formulates consultation as a sequential resource-allocation problem. At workflow contact points, ARIA maps execution evidence to a compact state abstraction and selects SILENT or CONSULT, with action values learned by Monte Carlo control from cost-penalized outcomes. Across evaluated tasks of SWE-bench Verified, Terminal-Bench 2.1, and τ2-bench, ARIA recovers 44-81% of the executor-advisor success gap at less than half the advisor model's cost. ARIA achieves the best observed success-cost trade-off among ten other evaluated conditions, including routing and cascading methods, fixed and prompted consultation, and random placement at matched advice volume. The policies are compact and inspectable, with a single consultation-price parameter governing the success-cost trade-off without retraining either underlying model.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 12 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. ARIA is a carefully evaluated, honestly reported consultation-timing method whose main soft spot—outcome-screened training tasks—is real but disclosed and does not overturn the measured held-out results. This line is wrong

3 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 53 references say.

The paper says, at Appendix B.1, Table 7

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.81):

“This makes the development distribution easier than the evalu- ation distribution and can increase consultation pressure when the frozen policy is deployed on the harder held-out tasks.”

Pith says: Objection 1 of 3 · major

The development-task screening is outcome-dependent, and the transfer premise it creates is load-bearing but untested. The SWE-bench development pool (52 tasks from 100 candidates) retained only tasks solved by at least one of the two underlying models (stratum A: 21 executor-solved, stratum B: 31 advisor-solved), and the Terminal-Bench development partition removed 12 of 39 tasks never solved in six autonomous runs.

more of objection 1

Because the update target G_t = Y(tau) - kappa*N_c is observed only on episodes reaching a terminal outcome, the controller has no training experience on never-solved tasks, which are 30 percent (15 of 50) of the held-out Terminal-Bench set. The Q-difference Q(s,CONSULT) minus Q(s,SILENT) is therefore estimated conditional on a screening event that need not hold at deployment. The authors disclose this (b387, b394, b397) and argue that harder deployment tasks raise consultation pressure; that argument addresses cost direction, not placement calibration. On tasks that are unsolvable at either capability, consultation cannot change Y, so an unfiltered-trained controller might learn to skip wasted consultations and be cheaper; at the same time, tasks solvable only with advisor guidance are excluded because neither solo run solves them, and these are exactly where consultation is most valuable. A filtered training signal can distort the frozen policy in both directions, potentially exhausting the B=4 budget in states where advice cannot help and lowering PGR and ER. This is consistent with the weakest result occurring on Terminal-Bench (PGR=44.4 percent, ER=0.15), the benchmark whose development set explicitly removed never-solved tasks. I request a sensitivity analysis retraining a controller on the unfiltered Terminal-Bench development partition, or on a random split without outcome filtering, and reporting PGR, ER, and cost, plus a report of the frozen policy's consultation rate and success on the 15 never-solved held-out tasks versus the other 35. If this is infeasible, scope the conclusion and abstract to policies trained on screened development pools and soften the transfer claim in Section 6.

The paper says, at Appendix C.1, Tables 2 and 12

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The central contrasts are three-seed means without inference, and several decisive comparisons sit inside the reported seed noise. The paper says small differences should be read descriptively, but the main-text conclusions are not uniformly hedged. In Section 5.1, on Terminal-Bench ARIA and Heuristic both achieve 34.7 percent, with standard deviations of 5.0 and 3.1 percentage points, and the ARIA versus …

more of objection 2

… Random comparison is 34.7 (SD 5.0) against 30.7 (SD 5.0); on tau2-bench ARIA is 93.3 (SD 2.9) versus Random 85.8 (SD 5.2). The placement-matters claim therefore rests almost entirely on SWE-bench, where the ARIA-Random gap is 20 points. Similarly, the statements that no condition achieves higher success at equal or lower cost and that ARIA lies on the empirical Pareto frontier (b123-b124) are mean-only: on Terminal-Bench Always is 0.6 points higher in success at 12 percent higher cost, and ARIA ties Heuristic exactly, with all differences inside one standard deviation. Because the same task instances are evaluated under every condition, paired inference is feasible and cheap. I request per-task paired deltas with bootstrap or McNemar tests for ARIA versus Random, Heuristic, and Always, with intervals on PGR, CostShare, and ER, and a statement of which headline claims survive. At minimum, the seed-level dispersion should appear with the main table rather than only in Appendix C.1.

The paper says, at Section 5.2, Figure 2, Appendix A.3

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The headline operating point is selected among tested consultation prices, and the price sweep is an offline re-estimation rather than independent retraining. On SWE-bench, lambda=0.10 gives ER=0.37 while lambda=0.35 and 0.50 give ER=0.15 and 0.11, the latter two below Heuristic's 0.24; the claim that ARIA achieves the best observed success-cost trade-off among ten other conditions holds only for the selected price, and the abstract's Pareto-frontier phrasing (b7, b124) inherits that conditioning.

more of objection 3

The paper discloses the offline re-estimation in Appendix A.3, but the abstract should carry the same caveat. The sweep is reported only for SWE-bench, so the single-parameter trade-off is not demonstrated on the other two benchmarks. I request an explicit post hoc selection statement in Section 5.2 and the abstract, plus application of the offline lambda re-estimation to at least one other benchmark, which requires no new trajectories, to show that the Pareto-frontier claim is not an artifact of the chosen price.

7 minor objections

The paper says, at Table 3

Pith says: minor objection 1 of 7

The row labeled 'Sav. vs. equal-succ. mix' is not defined in Section 4; the text defines PGR, CostShare, and ER but not this savings measure, so the caption should state its formula or the row should be removed.

The paper says, at Table 2

Pith says: minor objection 2 of 7

Because seed-level dispersion is central to the claims, the standard deviations in Appendix C.1 should appear with the main table rather than only behind an appendix pointer.

show 5 more

The paper says, at Section 5.1

Pith says: minor objection 3 of 7

On tau2-bench, Random realizes only 87.8 percent of ARIA's consultation volume (Appendix B.3, Table 9); the comparison should be labeled as approximately rate-matched, because the success gap is partly confounded with the volume shortfall.

The paper says, at Appendix B.3

Pith says: minor objection 4 of 7

The Terminal-Bench Random probability is calibrated from reconstructed contact-point counts because complete logs are unavailable for 25 of 150 crash-path episodes (b442); the calibration should state how sensitive the resulting probability is to that reconstruction assumption.

The paper says, at Section 3.2

Pith says: minor objection 5 of 7

Because remaining budget is omitted from the state and CONSULT is forced to SILENT once B=4 is exhausted, the learned Q-values marginalize over the training-time budget process; a sentence in Section 6 noting that deployment at a different budget is a distribution shift would prevent the Q-tables from being read as budget-independent.

The paper says, at Figure 9 (Appendix C.5)

Pith says: minor objection 6 of 7

The figure mixes two outcome definitions, one-step error-free execution and episode success, in one panel; separating the two outcome types graphically would make the estimates easier to interpret.

The paper says, at References

Pith says: minor objection 7 of 7

Please verify that the DOIs for the AutoMix and MAST entries (10.52202 prefix) resolve to the correct records; several other entries would benefit from complete venue information.

What would settle it

Retrain the controller outcome-blind and re-measure on the same held-out tasks: for SWE-bench, run the identical ε-greedy Monte Carlo control procedure on the full 100-task candidate pool (or a fixed subsample) without the solve-based strata; for Terminal-Bench, run it on the 39-task development partition without removing the 12 never-solved tasks. Hold λ=0.10, B=4, the exploration schedule (Appendix A.3), and the three-seed protocol fixed, and compare frozen-policy PGR, ER, and realized cost against Table 2. If PGR drops by more than roughly 10 points or CostShare exceeds 50% on either benchmark, the headline recovery claim depends on the outcome-dependent screening; if the shifts stay within seed-level dispersion (±5 points), the transfer premise holds. This requires only about 150-160 additional training episodes per benchmark with the released framework.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 12 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 2 min 10 s · 30,813 tokens read, 15,863 written
  3. Editor wrote the verdict above 24 s · 31,925 tokens read, 2,150 written
  4. Referee major revision: 3 major objections, 7 minor 4 min 9 s · 33,936 tokens read, 7,438 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The thing to know: this is a solid systems paper, not a field-reorganizing result. ARIA learns a 24-state tabular policy that decides when a cheap executor should consult a stronger advisor at workflow contact points, trained by Monte Carlo on cost-penalized terminal outcomes. On three benchmarks it recovers 44–81% of the executor-advisor success gap at under half the advisor's cost, with positive excess recovery that beats the other conditions they tried.

What is genuinely new is the within-trajectory, state-conditioned allocation as opposed to task-level routing or fixed schedules. The state abstraction uses observable workflow evidence rather than self-reported confidence, and the policy tables are small enough to inspect. The evaluation is also careful: held-out tasks with verified zero overlap, three seeds, a random-placement control matched to consultation volume, and a cost reconciliation in Appendix B.4 that reproduces all 33 headline cost cells. That is real evidence, and more than most routing papers provide.

The main soft spot is the one they disclose. Development tasks were screened to keep tasks solved by at least one model (SWE-bench) and to drop tasks never solved in six autonomous runs (Terminal-Bench). The stress-test concern is that the Q-difference is estimated only on partially solvable tasks, so the consult/silent boundary could be miscalibrated exactly where deployment contains unsolvable tasks. I think that is a legitimate limitation, not a refutation. The measured numbers on the held-out splits are valid, and the authors' argument that the screening does not favor lower cost is sensible. What remains unresolved is placement transfer—whether the learned boundaries hold when the deployment distribution is harder. Terminal-Bench, where the filter was applied and 30% of held-out tasks are never-solved, is indeed ARIA's weakest result, which is consistent with the concern, though ARIA still matches the heuristic there at lower cost. An outcome-blind or unfiltered training set would settle this cleanly.

Two smaller points. Three seeds with overlapping standard deviations mean the nearer comparisons should be read descriptively, which the authors themselves say. And the headline lambda = 0.10 is selected among the tested prices rather than pre-registered. Both are minor given the size and consistency of the main effects.

This paper is for people building or evaluating cost-sensitive LLM agent deployments, especially executor-advisor systems. It deserves a serious referee: the claims are bounded, the method is simple to reproduce, and the appendices do the kind of accounting most papers skip. Send it to review.

The strongest objection

The measured headline numbers are valid on the evaluated splits: the held-out sets are unfiltered (Verified-Mini, the Terminal-Bench test partition, and the τ2 test split verbatim), the policy is frozen, and Appendix B.4 reconciles all 33 cost cells. The load-bearing premise is that consultation values learned on the development tasks transfer to the deployment distribution. That premise rests on a screened training signal: the SWE-bench development pool kept only tasks solved by at least one of the two models (21 executor-solved, 31 advisor-solved, from 100 candidates), and Terminal-Bench removed 12 tasks never solved in six autonomous runs (Appendix B.1). This is selection on the outcome the controller predicts: the Monte Carlo return Y(τ) − κ·N_c is observed only on tasks where success is achievable, so the Q-difference Q(s,CONSULT) − Q(s,SILENT) is estimated conditional on a screening event that need not hold at deployment. Never-solved tasks, the ones that would teach the controller that consultation is wasted effort, are absent from training. The authors disclose this (b387, b394, b397) and argue it does not favor lower cost because harder deployment tasks raise consultation pressure. That argument bounds the cost direction only; it does not address placement. If the value estimates are miscalibrated in the unsolvable region, the frozen policy can consult in the wrong states, exhaust the B=4 budget, and recover none of the gap, lowering PGR and ER rather than merely raising cost. The cross-benchmark pattern is consistent with the concern but not decisive: Terminal-Bench, where the filter was applied and 30% (15/50) of held-out tasks are never-solved, yields ARIA's weakest result (PGR 44.4%, ER 0.15, success tied with the unlearned Heuristic at 34.7%), while τ2, which was not filtered, yields the strongest result (PGR 80.6%), though its near-solvable tasks and interactive regime are confounds.

The central claim has two parts: ARIA recovers 44-81% of the executor-advisor success gap at under half the advisor's cost, and ARIA lies on the empirical success-cost Pareto frontier among evaluated conditions. For these to hold, three things must be true: (i) learned Q-values transfer from the tasks where they were estimated to the tasks where they are deployed; (ii) cost accounting and baseline contracts are fair; (iii) reported means are not seed artifacts. The paper handles (ii) well: Appendix B.4 reconstructs all 33 headline cost cells within $0.0049, and Appendix B.2 separates baseline contracts cleanly. Point (iii) is handled honestly: Table 12 reports per-seed dispersion, the text says small differences are interpreted descriptively, and the Pareto claim is stated by means and checks out arithmetically against Table 2. The least secure premise is (i). The outcome-dependent screening in Appendix B.1 means the training signal exists only on tasks solvable by at least one model; the authors disclose this and argue the bias does not favor lower cost, but that argument addresses the cost direction, not whether learned consultation placement transfers to tasks never solvable by either model. The cross-benchmark pattern is consistent with the concern (weakest ARIA result on the filtered, 30%-never-solved Terminal-Bench; strongest on the unfiltered τ2), though confounded by domain differences. The proposed outcome-blind retraining test settles the transfer question directly and is feasible with the released code. This supports rather than changes the reader's CONDITIONAL verdict: the measured claims are credible, the generalization premise is real but untested, and the remedy is an evaluation-design change the authors can run. I agree with the reader's weakest_assumption, which identifies the same screening/transfer premise; my refinement is that the disclosed cost-direction argument does not cover placement, and that the Terminal-Bench-versus-τ2 contrast provides a partial natural check. Credit where due: cost reconciliation, transparent splits, the within-state IPW analysis (Appendix C.5), human-validated MAST annotations, and the hedged means-based Pareto claim are genuine strengths; no machine-checked proof exists, but the empirical accounting is unusually auditable.

The referee's report

The paper has several genuine strengths. The evaluation is unusually auditable: Appendix B.4 reconstructs all 33 headline cost cells from raw billing artifacts with a maximum deviation of $0.0049 and reproduces the provider pricing for 100% of 106,662 calls; the held-out task sets are disjoint from development tasks with verified zero overlap; the deployment policy is frozen with no exploration; and the Random control approximates consultation volume rather than dollar cost. ARIA is also compact and inspectable: a 24-state tabular policy, no fine-tuning, and no access to model-internal confidence signals. The failure analysis is a substantive contribution, identifying ignored advisor input as the dominant residual failure mode once placement is learned, with a human-validated annotation sample. The main risks are outcome-dependent development-task screening, three-seed means without significance testing, and post hoc selection of the headline consultation price. These are all disclosed by the authors, and all are amenable to additional sensitivity analysis within the paper's scope.

Pith's reading of the paper (a machine's summary, not the authors')

A cheap model can run an agentic task end to end, and a frontier model can run it better but far more expensively; the paper's question is when, mid-task, the cheap model should pause and ask the frontier model for advice. ARIA, the proposed answer, is a small learned controller that ignores the executor's self-assessed confidence — known to be unreliable — and instead maps three observable signals (workflow stage, whether the latest step produced an error, and whether advice was already given) to a 24-state table that chooses SILENT or CONSULT, trained purely from final task success minus a per-consultation price. On SWE-bench Verified, Terminal-Bench 2.1, and τ²-bench, ARIA recovers 44–81% of the executor-advisor success gap while spending under half the advisor's cost, and it beats ten other strategies — routing, cascading, fixed triggers, prompt-based routing, and random placement at matched advice volume — on the success-cost trade-off. If this holds, a workflow does not need to run entirely on a frontier model to capture most of its capability; it needs the right moments of consultation, decided by a rule small enough to inspect.

What carries the argument. The load-bearing object is the 24-state consultation table learned by tabular Monte Carlo control. Each state $s_t = (p_t, e_t, h_t)$ compresses the trajectory into a temporal phase (FRAME, EXPLORE, REASON, or DELIVER, named after stages of problem solving), an execution-evidence class (NONE, CLEAN, or ERROR, derived from observable output such as tracebacks, non-zero exit codes, or tool results), and a bit recording whether advisor guidance has already entered the trajectory. The update target is the return-to-go $G_t = Y(\tau) - \kappa N_c^{\ge t}(\tau)$: terminal task success minus a fixed per-consultation penalty applied to consultations at or after the visit, with $\gamma = 1$ so the terminal outcome is not discounted against immediate cost, and $\kappa = \lambda \bar{c}/\Delta$ calibrated per domain to the real dollar cost of consulting. What carries the argument is that consultation value is estimated only from end-of-episode outcomes, pooled across the small state space, so the deployed greedy policy is a deterministic, inspectable lookup table requiring no model internals, no fine-tuning, and a single price parameter $\lambda$ that sweeps the success-cost frontier.

Core claim. ARIA treats consultation as a sequential resource-allocation problem: at each contact point — before a step, when progress stalls, and before final delivery — the controller chooses SILENT or CONSULT, and a consultation buys a natural-language assessment from the advisor that the executor may follow, adapt, or ignore. The state is the triple (phase, evidence, prior advice) over $4 \times 3 \times 2 = 24$ cells, where evidence is read from observable output (no output, clean output, or an explicit error signal such as a traceback or exit code) rather than from the executor's confidence. Values are learned by every-visit tabular Monte Carlo control on the cost-penalized terminal return $G(\tau) = Y(\tau) - \kappa N_c(\tau)$, with the penalty $\kappa$ calibrated to the measured dollar cost of a consultation, and a single price parameter $\lambda$ sets the operating point without retraining either model. On the three benchmarks ARIA reaches 48.0%, 34.7%, and 93.3% success at $22.09, $24.93, and $7.77, recovering 68.2%, 44.4%, and 80.6% of the executor-advisor success gap at 48.1%, 40.9%, and 34.5% of advisor-only cost, for an excess recovery (ER = PGR − CostShare) of 0.37, 0.15, and 0.65 — the best among the eleven evaluated conditions and on the empirical success-cost Pareto frontier. The learned tables select CONSULT in only 4–9 of the 24 states, concentrate advice at the delivery stage on SWE-bench and mid-trajectory on τ²-bench, and their dominant residual failure mode is no longer task verification but advice that reaches the executor and is not acted upon.

The premise it rests on. The load-bearing premise is that consultation values learned on easier, pre-screened development tasks transfer to harder held-out tasks: both training pools were filtered to keep tasks that at least one model could solve (SWE-bench by explicit screening, Terminal-Bench by dropping 12 never-solved tasks), and the authors disclose this asymmetry while arguing it does not favor lower cost, yet failure of that transfer would leave the table miscalibrated exactly where advice matters most.

If the paper is right (Pith's inferences, not the authors')

Cheap execution plus learned consultation replaces most of the frontier model: on SWE-bench Verified, ARIA solves 48.0% of tasks at $22.09, versus 62.0% at $45.90 for the advisor alone and 18.0% at $11.48 for the executor alone.

Timing drives the gain, not access or volume: ARIA and random placement use nearly the same number of consultations per episode (2.78 vs 2.53 on SWE-bench), yet ARIA is 20 percentage points more successful, and consulting at every opportunity at the same $B=4$ budget buys 14 points less success.

Letting the workflow's own orchestrator decide when to consult does not reproduce the learned behavior: the native router consults only 0.41–0.61 times per episode and places 64–77% of its calls in the final third of the trajectory.

The consultation price is a working operating-point dial: raising $\lambda$ from 0.10 to 0.50 cuts consultations from 2.78 to 1.55 per episode and cost from $22.09 to $16.63 on SWE-bench, but lowers excess recovery from 0.37 to 0.11, so the cheapest policy is not the most efficient one.

Residual failures expose the next bottleneck: as consultation placement improves, the dominant failure mode shifts from task verification to ignored agent input (58% of ARIA's SWE-bench failures), so improving whether advice is acted upon is a lever separate from improving when it is given.

Because the mechanism needs only observable workflow signals, outcome data, and a price, the same 24-state controller could be retrained for other executor-advisor pairs, other agent orchestration frameworks, or even human oversight channels without touching either model; the paper fixes the model pair and workflow, but nothing in the method depends on that choice.

A second signal tracking what the executor did with prior advice — not just that advice was given — could reclaim part of the ignored-input failures that remain under ARIA; the current abstraction deliberately collapses that response away.

The state space also drops the remaining consultation budget, and a controller that plans over its advice allowance might improve on heterogeneous tasks like Terminal-Bench, where the learned gain was smallest (PGR 44.4%).

If the known difficulty gap between ARIA's screened training pools and its held-out tests matters, the advantage should compress on the cohort of never-solved held-out tasks (15 of 50 on Terminal-Bench); reporting that cohort separately is a direct way to test whether learned timing helps in the regime where advice is the only hope.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

To balance task performance and computational cost, agentic workflows can use a lower-cost model for execution and selectively consult a stronger model. In such executor-advisor systems, the executor decides when to seek advice, making escalation depend on self-assessment, although current language models struggle to detect where their own reasoning fails. The need for stronger reasoning emerges during execution and varies across workflow phases. We introduce ARIA, a meta-controller that learns when to consult by balancing task success against consultation cost. Our approach formulates consultation as a sequential resource-allocation problem. At workflow contact points, ARIA maps execution evidence to a compact state abstraction and selects SILENT or CONSULT, with action values learned by Monte Carlo control from cost-penalized outcomes. Across evaluated tasks of SWE-bench Verified, Terminal-Bench 2.1, and τ2-bench, ARIA recovers 44-81% of the executor-advisor success gap at less than half the advisor model's cost. ARIA achieves the best observed success-cost trade-off among ten other evaluated conditions, including routing and cascading methods, fixed and prompted consultation, and random placement at matched advice volume. The policies are compact and inspectable, with a single consultation-price parameter governing the success-cost trade-off without retraining either underlying model.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-12

The authors' main claim, as the classifier picked it out 0.86

“ARIA achieves the best observed success-cost trade-off among ten other evaluated conditions, including routing and cascading methods, fixed and prompted consultation, and random placement at matched advice volume.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.15 · human participants 0.04 · controlled comparison 0.90 · randomized assignment 0.04 · states quantitative result 0.99 · compares with prior results 0.96 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
To balance task performance and computational cost, agentic workflows can use a lower-cost model for execution and selectively consult a stronger model. background 1.00 0.06 0.05 hedged possibility 0.86
In such executor-advisor systems, the executor decides when to seek advice, making escalation depend on self-assessment, although current language models struggle to detect where their own reasoning fails. background 0.54 0.07 0.03 asserted 0.99
The need for stronger reasoning emerges during execution and varies across workflow phases. background 0.95 0.17 0.03 asserted 1.00
We introduce ARIA, a meta-controller that learns when to consult by balancing task success against consultation cost. contribution 1.00 0.93 0.14 asserted 1.00
Our approach formulates consultation as a sequential resource-allocation problem. method 0.87 0.83 0.04 asserted 1.00
At workflow contact points, ARIA maps execution evidence to a compact state abstraction and selects SILENT or CONSULT, with action values learned by Monte Carlo control from cost-penalized outcomes. method 1.00 0.91 0.07 asserted 1.00
Across evaluated tasks of SWE-bench Verified, Terminal-Bench 2.1, and $\tau^2$-bench, ARIA recovers 44-81% of the executor-advisor success gap at less than half the advisor model's cost. result 1.00 0.96 0.50 asserted 1.00
ARIA achieves the best observed success-cost trade-off among ten other evaluated conditions, including routing and cascading methods, fixed and prompted consultation, and random placement at matched advice volume. result 1.00 0.95 0.43 asserted 1.00
The policies are compact and inspectable, with a single consultation-price parameter governing the success-cost trade-off without retraining either underlying model. result 0.58 0.92 0.15 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.95

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-12: 1 request, 6,900 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (10)
Figure 1 of the paper
Figure 1. ARIA setup. At each contact point, the meta controller chooses between S
Figure 2 of the paper
Figure 2. Success-cost operating points on SWE-bench for ARIA policies with
Figure 3 of the paper
Figure 3. Primary failure mode across baselines and benchmarks grouped into system-design, inter
Figure 4 of the paper
Figure 4. Task success and consultation frequency for ARIA,
Figure 5 of the paper
Figure 5. Training dynamics across three benchmarks and three controller seeds.
Figure 6 of the paper
Figure 6. Learned action values for SWE-bench Verified across three independently trained con
Figure 7 of the paper
Figure 7. Learned action values for Terminal-Bench 2.1 across three independently trained con
Figure 8 of the paper
Figure 8. Learned action values for τ 2 -bench Telecom across three independently trained con￾troller seeds. Left: compact numerical tables reporting state visit counts and action values, where QS = Q(s, SILENT) and QC = Q(s, CONSULT). State notation is phase/evidence/prior consul￾tation: F/E/R/D denote FRAME, EXPLORE, REASON, and DELIVER; N/O/F denote NONE, OK, and FAILED. Structurally unreachable states are omitted, while reachable states with N = 0 are retained. Right: the corresponding learned value tables for the three seeds visualized as heatmaps. Deployment selects the action with the…
Figure 9 of the paper
Figure 9. Propensity-weighted effect of CONSULT versus SILENT within each state cell. Points show Hajek IPW estimates with 95% episode-cluster bootstrap intervals. Marker shape indicates ´ the greedy action of the reference Q-table; circled markers use the episode-level outcome because the one-step outcome is constant or unavailable. Faded intervals include zero. Parentheses report the number of observed decisions in each cell. Mid-trajectory effects are more heterogeneous. On SWE-bench, REASON/NONE/k ≥ 1 and REASON/ERROR/k ≥ 1 have negative one-step estimates of −0.35 [−0.59, −0.03] and…
Figure 10 of the paper
Figure 10. Secondary failure modes across baselines and benchmarks grouped into system-design,

Figures are the authors', reproduced from the paper.

References (53)
  1. [1] P. How to solve it:. 1945 , publisher = on Pith
  2. [2] Altman, Eitan , year =. Constrained on Pith
  3. [3] A fast learning algorithm for deep belief nets , author =. Neural Computation , volume =. 2006 , number = on Pith
  4. [4] Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling learning algorithms toward. 2007 , doi = on Pith
  5. [5] Torrey, Lisa and Taylor, Matthew E. , booktitle =. Teaching on a budget:. 2013 , pages = on Pith
  6. [6] Near optimal behavior via approximate state abstraction , author =. 2016 , publisher = on Pith
  7. [7] Learning with rejection , author =. Proceedings of the International Conference on Algorithmic Learning Theory , pages =. 2016 , publisher = on Pith
  8. [8] Deep learning , author =. 2016 , publisher = on Pith
  9. [9] Sutton, Richard S. and Barto, Andrew G. , edition =. Reinforcement learning:. 2018 , publisher = on Pith
  10. [10] Huang, Wenlong and Xia, Fei and Xiao, Ted and Chan, Harris and Liang, Jacky and Florence, Pete and Zeng, Andy and Tompson, Jonathan and Mordatch, Igor and Chebotar, Yevgen and Sermanet, Pierre and Jackson, Tomas and Brown, Noah and Luu, Linda and Levine, Sergey and Hausman, Ka... on Pith
  11. [11] Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle = iclr, year = on Pith
  12. [12] Qingyun Wu and Gagan Bansal and Jieyu Zhang and Yiran Wu and Beibin Li and Erkang Zhu and Li Jiang and Xiaoyun Zhang and Shaokun Zhang and Jiale Liu and Ahmed Hassan Awadallah and Ryen W White and Doug Burger and Chi Wang , booktitle=colm, year= on Pith
  13. [13] Fourney, Adam and Bansal, Gagan and Mozannar, Hussein and Tan, Cheng and Salinas, Eduardo and Zhu, Erkang and Niedtner, Friederike and Proebsting, Grace and Bassman, Griffin and Gerrits, Jack and Alber, Jacob and Chang, Peter and Loynd, Ricky and West, Robert and Dibia, Victor... on PithPith's reviewarXiv
  14. [14] Wang, Xingyao and Chen, Yangyi and Yuan, Lifan and Zhang, Yizhe and Li, Yunzhu and Peng, Hao and Ji, Heng , booktitle = icml, pages =. Executable code actions elicit better. 2024 , volume= on Pith
  15. [15] Agashe, Saaket and Han, Jiuzhou and Gan, Shuyu and Yang, Jiachen and Li, Ang and Wang, Xin , booktitle = iclr, year =. Agent on Pith
  16. [16] Automated design of agentic systems , author =. 2025 , url = on Pith
  17. [17] Zhang, Jiayi and Xiang, Jinyu and Yu, Zhaoyang and Teng, Fengwei and Chen, Xiong-Hui and Chen, Jiaqi and Zhuge, Mingchen and Cheng, Xin and Hong, Sirui and Wang, Jinlin and Zheng, Bingnan and Liu, Bang and Luo, Yuyu and Wu, Chenglin , booktitle = iclr, year = on Pith
  18. [18] Luo, Xufang and Zhang, Yuge and He, Zhiyuan and Wang, Zilong and Zhao, Siyun and Li, Dongsheng and Qiu, Luna K. and Yang, Yuqing , year =. Agent. arXiv preprint arXiv:2508.03680 , doi = on PithPith's reviewarXiv
  19. [19] Mozannar, Hussein and Bansal, Gagan and Tan, Cheng and Fourney, Adam and Dibia, Victor and Chen, Jingya and Gerrits, Jack and Payne, Tyler and Maldaner, Matheus Kunzler and Grunde-McLaughlin, Madeleine and Zhu, Eric and Bassman, Griffin and Alber, Jacob and Chang, Peter and Lo... on PithPith's reviewarXiv
  20. [20] Kon, Patrick Tser Jern and Pradeep, Archana and Chen, Ang and Ellis, Alexander P. and Hunt, Warren and Wang, Zijian and Yang, John and Thompson, Samuel , year =. arXiv preprint arXiv:2602.22124 , doi = on PitharXiv
show 33 more
  1. [21] Shao, Yijia and Samuel, Vinay and Jiang, Yucheng and Yang, John and Yang, Diyi , booktitle = iclr, year = on Pith
  2. [22] Cemri, Mert and Pan, Melissa Z and Yang, Shuyi and Agrawal, Lakshya A and Chopra, Bhavya and Tiwari, Rishabh and Keutzer, Kurt and Parameswaran, Aditya and Klein, Dan and Ramchandran, Kannan and Zaharia, Matei A and Gonzalez, Joseph E. and Stoica, Ion , booktitle = neurips, do... on Pith
  3. [23] Zhang, Shaokun and Yin, Ming and Zhang, Jieyu and Liu, Jiale and Han, Zhiguang and Zhang, Jingyang and Li, Beibin and Wang, Chi and Wang, Huazheng and Chen, Yiran and Wu, Qingyun , booktitle = icml, pages =. Which agent causes task failures and when?. 2025 , url = on Pith
  4. [24] Yang, Wei and Cao, Defu and Pang, Jiacheng and Weng, Muyan and Liu, Yan , booktitle = iclr, year =. Adaptive collaboration with humans: on Pith
  5. [25] Hay, Nicholas and Russell, Stuart and Tolpin, David and Shimony, Solomon Eyal , booktitle =. Selecting computations:. 2012 , isbn = on Pith
  6. [26] Chen, Lingjiao and Zaharia, Matei and Zou, James , journal=tmlr, year=. Frugal on Pith
  7. [27] Aggarwal, Pranjal and Madaan, Aman and Anand, Ankit and Potharaju, Srividya Pranavi and Mishra, Swaroop and Zhou, Pei and Gupta, Aditya and Rajagopal, Dheeraj and Kappaganthu, Karthik and Yang, Yiming and Upadhyay, Shyam and Faruqui, Manaal and Mausam , booktitle = neurips, vo... on Pith
  8. [28] Belcak, Peter and Heinrich, Greg and Diao, Shizhe and Fu, Yonggan and Dong, Xin and Muralidharan, Saurav and Lin, Yingyan Celine and Molchanov, Pavlo , year =. Small language models are the future of agentic. arXiv preprint arXiv:2506.02153 , doi = on PithPith's reviewarXiv
  9. [29] Damani, Mehul and Shenfeld, Idan and Peng, Andi and Bobu, Andreea and Andreas, Jacob , booktitle = iclr, year =. Learning how hard to think: on Pith
  10. [30] Rational Metareasoning for Large Language Models , author=. NeurIPS Workshop on Behavioral Machine Learning , year= on Pith
  11. [31] Ong, Isaac and Almahairi, Amjad and Wu, Vincent and Chiang, Wei-Lin and Wu, Tianhao and Gonzalez, Joseph E. and Kadous, M Waleed and Stoica, Ion , booktitle = iclr, year = on Pith
  12. [32] Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral , booktitle = iclr, year =. Scaling on Pith
  13. [33] Confidence-guided stepwise model routing for cost-efficient reasoning , author =. 2026 , doi = on Pith
  14. [34] Teaching Large Language Models to Self-Debug , author=. 2024 , url= on Pith
  15. [35] Reflexion: language agents with verbal reinforcement learning , author =. 2023 , volume= on Pith
  16. [36] Large Language Models Cannot Self-Correct Reasoning Yet , author=. 2024 , url= on Pith
  17. [37] Is Self-Repair a Silver Bullet for Code Generation? , author=. 2024 , url= on Pith
  18. [38] Tyen, Gladys and Mansoor, Hassan and Carbune, Victor and Chen, Peter and Mak, Tony , booktitle = aclfindings, year = on Pith
  19. [39] Xiong, Miao and Hu, Zhiyuan and Lu, Xinyang and Li, Yifei and Fu, Jie and He, Junxian and Hooi, Bryan , booktitle = iclr, year =. Can on Pith
  20. [40] Kazemnejad, Amirhossein and Aghajohari, Milad and Portelance, Eva and Sordoni, Alessandro and Reddy, Siva and Courville, Aaron and Le Roux, Nicolas , title =. 2025 , booktitle = icml, url= on Pith
  21. [41] Liu, Zihan and Yang, Zhuolin and Chen, Yang and Lee, Chankyu and Shoeybi, Mohammad and Catanzaro, Bryan and Ping, Wei , booktitle = iclr, year = on Pith
  22. [42] Chowdhury, Neil and Aung, James and Chan, Jun Shern and Jaffe, Oliver and Sherburn, Dane and Starace, Giulio and Mays, Evan and Dias, Rachel and Aljubeh, Marwan and Glaese, Mia and Jimenez, Carlos E. and Yang, John and Ho, Leyton and Patwardhan, Tejal and Liu, Kevin and Madry,... on Pith
  23. [43] Kapoor, Sayash and Stroebl, Benedikt and Siegel, Zachary S. and Nadgir, Nitya and Narayanan, Arvind , journal =. 2025 , issn = on Pith
  24. [44] Kwa, Thomas and West, Ben and Becker, Joel and Deng, Amy and Garcia, Katharyn and Hasin, Max and Jawhar, Sami and Kinniment, Megan and Rush, Nate and Von Arx, Sydney and Bloom, Ryan and Broadley, Thomas and Du, Haoxing and Goodrich, Brian and Jurkovic, Nikola and Miles, Luke H... on Pith
  25. [45] Wei, Jason and Sun, Zhiqing and Papay, Spencer and McKinney, Scott and Han, Jeffrey and Fulford, Isa and Chung, Hyung Won and Passos, Alex Tachard and Fedus, William and Glaese, Amelia , year =. arXiv preprint arXiv:2504.12516 , doi = on PithPith's reviewarXiv
  26. [46] Barres, Victor and Dong, Honghua and Ray, Soham and Si, Xujie and Narasimhan, Karthik , year =. ^2 -. arXiv preprint arXiv:2506.07982 , doi = on PithPith's reviewarXiv
  27. [47] Merrill, Mike A. and others , booktitle = iclr, year = on Pith
  28. [48] Building effective agents , author =. 2024 , month = dec, howpublished = on Pith
  29. [49] New tools for building agents , author =. 2025 , month = mar, howpublished = on Pith
  30. [50] The. 2025 , month = feb, howpublished = on Pith
  31. [51] The advisor strategy:. 2026 , month = apr, howpublished = on Pith
  32. [52] Mohan, Varun and Kilpatrick, Logan , year =. Building the agentic future: on Pith
  33. [53] Almeida, Diogo , year =. Introducing on Pith
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment