Pith.

Artificial Intelligence · arXiv:2610.24871 v1 · submitted 17 Oct 2026 · 46 references

Bellman Language Models: Solving the Recurring Decisions of LLM Agents

Adarsh Vatsa, William Eiers

The paper says

“A general observation in agentic workflows, especially bounded ones, is that the decision space of the LLM, when we separate a decision from generation, starts approaching saturation: the agent meets the same states again and again, and after enough conversations it stops meeting new ones.”

more of the abstract

LLMs generally don't treat these decisions as an optimization problem and instead re-derive them from their context/prompt on every turn. We show that treating decision making within reasonable bounds as an optimization problem, solving it once, and reusing the solved decisions can often make an agent faster, cheaper and more reliable. Our approach, which we call Discretize, Then Decide, starts from data such as the workflow's own logs or, when none exists, runs of the LLM agent itself. It reads each conversation into discrete states and estimates the probability of each outcome and the cost of each move, until no new states appear and the probabilities stop changing. It then solves the resulting Markov decision process. The solution runs as a controller around the model, sending moves whose content is fixed in the state, such as a tool call or a templated reply, and any state it hasn't seen is sent back to the LLM. We call an LLM run this way a Bellman language model (BLM), after Richard Bellman's equation for the best move in each state. On τ2-bench telecom, the controller with no model solves 79% of tasks. As a BLM, Qwen3.8-27B, an open model, goes from 29% to 96% with 93% fewer model calls. For Claude Opus 5.5, which solves 96% of tasks on its own, running it as a BLM cuts model calls and cost per task by about 98% and lets it solve 2 more tasks. On InventoryBench, the solved policy with no model beats every published method. Where the hard part is writing content rather than choosing moves, such as resolving SymPy issues, few decisions can be solved in advance, and a BLM helps little.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 20 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A careful, honest paper: solving recurring agent decisions once and executing them cuts model calls 93–98% on its benchmark, and real-customer generalization is the acknowledged, unquantified risk. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 46 references say.

The paper says, at §5.3, Observation 7, Table 5

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.72):

“The other three domains show how far the claims carry, but there we kept test data apart from development less strictly (Section 5.3).”

Pith says: Objection 1 of 3 · major

The InventoryBench result is not a heldout comparison. Section 5.3 states that InventoryBench has no separate test data, and Section 11 states that the demand forecasts and the rollout were developed while looking at results on the whole benchmark. Observation 7 and Table 5 nevertheless present the no-model policy as beating every published method.

more of objection 1

The real/synthetic split is a partial control, but it does not protect the components that were selected by whole-benchmark outcomes. This claim should be either re-evaluated with settings frozen before seeing the real-demand batches, or explicitly reported as a development-set result; as written, it is load-bearing for the abstract's claim that the solved policy with no model beats every published method.

The paper says, at §11, Observations 1/3/4, Proposition 4

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.70):

“Our strongest evidence comes from telecom, where LLMs play the customers, who may vary less than real ones.”

Pith says: Objection 2 of 3 · major

The paper openly acknowledges that its strongest evidence comes from telecom with LLM-simulated customers. The remaining problem is that the quantitative headline—93-98% model-call reductions—is a function of the fallback share f, and f is measured only on that simulated distribution.

more of objection 2

A real-customer distribution with more off-script turns would raise f, and Proposition 4 shows that the reduction ceiling is 1/(rho+f+a). The Opus-as-customer replication is reassuring but still operates inside the same fault grammar. Please add either a sensitivity analysis (for example, the reduction as f ranges from 1% to 20%) or a measurement of the decidable share on human conversations; without one, the abstract's unqualified generalization about model-call reduction goes beyond the evidence.

The paper says, at §3.6, §4.2, Figure 8, Observation 1

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.60):

“The third condition is where saturation comes in, and it depends on choosing the state well.”

Pith says: Objection 3 of 3 · major

Observation 1's saturation is partly definitional. The decision state includes 'the first check in the solved order not yet made', so its value space is bounded by the number of checks in the fixed order; by construction it cannot grow beyond that finite set.

more of objection 3

The interesting claim is that this state suffices for good decisions, and the supporting evidence is the no-model controller's 90/114 success in Table 4. That evidence is valid for the tau2-simulated distribution, but the paper should separate the finite-by-construction claim from the empirical adequacy claim, and state explicitly that saturation is a property of the chosen abstraction rather than a property of the workflow independent of that choice. Section 11's caveat about state choice helps, but Observation 1 is phrased as an empirical discovery, not as a property of the chosen state representation.

4 minor objections

The paper says, at §5.3 and Appendix A

Pith says: minor objection 1 of 4

The main text says registrations were committed before any result, but Appendix A shows that on test set 1 the runs began 28 seconds before the registration was committed and that reruns used an unregistered rule; adding this qualification to Section 5.3 would prevent a misreading of the registration claim.

The paper says, at §4.5, Eq. (1)

Pith says: minor objection 2 of 4

The symbol a is used both for a move and for the double-check share in Proposition 4; renaming one of them would remove avoidable confusion.

show 2 more

The paper says, at Abstract and §6

Pith says: minor objection 3 of 4

The abstract's '93%' and '98%' figures are test-set-3/Opus figures; naming the specific test set in the abstract would help readers avoid over-generalizing the numbers to all three telecom controllers.

The paper says, at §9

Pith says: minor objection 4 of 4

The phrase 'utiledistinctions' is missing a space and should read 'utile distinctions'.

What would settle it

Replay the frozen third telecom controller (registered hash 51db041) and its reader over human-origin conversations: either real anonymized telecom support transcripts covering similar device problems, or a Wizard-of-Oz study in which human participants respond to the controller while following the same 19-fault task states. Count (i) the share of agent turns whose incoming message the reader fails to parse, (ii) the number of new decision states per 100 conversations, and (iii) the model calls per conversation that would result from the BLM fallback path. If the parsed/decidable share stays above roughly 90% and the new-state rate stabilizes, the 93–98% reduction transfers to real users; if the fallback share reaches 10–20% of turns, the headline reduction is specific to the simulated-customer distribution and should be re-stated with a sensitivity curve over f.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 20 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 26 s · 43,642 tokens read, 11,142 written
  3. Editor wrote the verdict above 50 s · 44,640 tokens read, 7,900 written
  4. Referee major revision: 3 major objections, 4 minor 1 min 4 s · 44,090 tokens read, 8,092 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The punchline: this is the most honest agent-systems paper I've read in a while, and its central claim — bounded workflows saturate, so solving the recurring decisions once and executing them with a controller removes most model calls — is well-supported on the benchmark it's tested on. The real open question is how the numbers survive real customers, and the paper says so itself (Section 11, b109) without quantifying the risk.

What's actually new: the ingredients aren't. POMDP dialogue managers, state abstraction, plan caching, trajectory compilation are all cited. What's new is the working recipe — read logs into discrete states, count outcomes and costs, solve the MDP once, execute with LLM fallback — plus two concrete deliverables: the decidable-share measurement (replay a frozen controller over existing logs to predict savings before deploying) and Proposition 4, which correctly casts the fallback share as an Amdahl serial fraction. Observation 6 is the cleanest experiment in the paper: telling the model the policy gets accuracy but not efficiency; executing gets both.

The methodology is strong. Controllers frozen and hashed before test draws, public randomness, separate test sets, honest reporting of missed targets — two of the first three telecom primaries missed and they say so. CI and SymPy are reported as the borderline results they are; the simulated CI numbers are explicitly not counted as evidence.

Soft spots, in proportion. The 93–98% call-reduction headline rests on LLM-simulated customers in one benchmark. The stress-test arithmetic is right: 0.25 calls over ~31 turns implies a fallback share around 1%, and 10–20% fallback with real users would drop the reduction to roughly 80–60%. That is a load-bearing external-validity gap, acknowledged but not quantified. The mitigation is that the decidable-share replay is exactly the instrument a practitioner would use to check on real logs first, so the recipe is self-diagnosing. Second, the InventoryBench "beats every published method" claim carries development exposure on the full benchmark (acknowledged); the cross-split setting selection is partial mitigation, and the claim reads stronger than the evidence. Third, test set 1's unregistered reruns of infrastructure errors (84 of 114 Opus runs) are disclosed in Appendix A but still a blemish; the later test sets were clean. Minor: the "small model reaches frontier accuracy" headline holds on test set 3 and the unseen-fault sets; the paper reports the two earlier sets where it fell short.

Who it's for: practitioners building bounded workflows get a concrete recipe with a pre-deployment cost-benefit test; researchers get a clean decomposition of where LLM calls are wasted and a formal target — the fallback share — for future work. I agree with the conditional verdict. I'd bring it to reading group, cite the decidable-share and cost-law framing, and send it to a serious referee, with real-customer replication as the expected next step.

The strongest objection

The central quantitative claim is Observations 3 and 4: BLMs remove 93–98% of model calls while holding or improving accuracy. That figure depends on two empirical facts measured only on τ2-bench telecom with LLM-simulated customers: the decision-state space stops growing (Observation 1, Figure 8) and the reader parses agent/customer turns well enough to keep the fallback share tiny. The paper acknowledges this directly in Section 11 (sentence b109): 'Our strongest evidence comes from telecom, where LLMs play the customers, who may vary less than real ones.' The Opus-as-customer replication is the right kind of stress test and is reassuring, but Opus still behaves inside the benchmark's hidden fault grammar; it is not a source of free-form human refusals, ambiguous answers, misunderstood instructions, or off-script questions, all of which the controller's fallback path hands to the LLM. That fallback share is exactly the Amdahl serial fraction f in Proposition 4. For a ~31-turn Opus BLM conversation, the observed 0.25 model calls imply f≈1%; keeping at least a 90% call reduction (vs. 16.9 calls for Opus alone) requires f≲5% of turns. A real-customer fallback of even 10–20% of turns would lower the call reduction to roughly 80–60%, turning the headline 'most model calls are unnecessary' from a strong quantitative claim into a much weaker qualitative one. The paper's internal controls—preregistered hashes, frozen controllers, separate test draws, honest reporting of missed targets—protect against test-set selection and overfitting, not against a different customer distribution. That external-validity gap is the load-bearing weakness; it is acknowledged but not quantified.

I read the paper as making a strong but carefully scoped claim: in bounded workflows whose decision states saturate, solving the recurring decisions once and executing them removes most model calls without losing accuracy. The telecom evidence is unusually well controlled: frozen controllers hashed before test draws, preregistered analyses, separate test sets, honest reporting of missed targets, and a replication with a stronger simulated customer. Those controls make the result internally credible. The reader's conditional verdict already centers on the same weak point I find: the saturation and reader-coverage measurements come from LLM-simulated customers, and the paper explicitly concedes those customers may vary less than real ones. I agree with that diagnosis. Real users can produce off-script turns, refusals, ambiguous statements, and actions that the reader does not recognize; each such turn becomes a model call, and the fallback share is precisely the quantity that caps the savings in the paper's own cost law. The Opus-as-customer result narrows the gap but does not close it, because Opus still plays inside the benchmark's hidden fault grammar. I did not find a separate internal inconsistency or a more load-bearing flaw: the InventoryBench exposure and the CI/SymPy limits are acknowledged and are secondary to the main telecom claim. Because the concern is the same one the reader already conditioned on, my stress-test does not change the verdict; it sharpens the condition by giving the fallback threshold that would need to hold for the headline reduction to transfer.

The referee's report

If the results hold, the paper makes a useful empirical and engineering contribution: it gives a concrete recipe for converting recurring LLM-agent decisions into an executed policy and shows large cost reductions in a realistic support-workflow benchmark. The formalization in Propositions 2-4 is standard but cleanly connects state abstraction, estimation error, and fallback rates to the measured savings. The paper's empirical discipline is a genuine strength: frozen controllers with recorded hashes, test tasks drawn after freezing, separate reporting of test sets, confidence intervals, a public registration repository, and explicit reporting of missed targets, negative results, and acknowledged limitations. The main caveat is that the headline quantitative claims rest on one benchmark with LLM-simulated customers and on a non-heldout InventoryBench comparison, so the breadth of the conclusions exceeds the cleanest evidence.

Pith's reading of the paper (a machine's summary, not the authors')

An agent working a bounded job — support tickets, failed-test triage, inventory restocking — keeps meeting the same decision states even as the words it generates change every time, and the paper argues that an LLM paid to re-derive those decisions on every turn is paying again for decisions it has already made. The method, Discretize, Then Decide, reads conversations into small discrete states, counts outcome probabilities and move costs from the workflow's own logs, solves the resulting Markov decision process once, and runs the solution as a controller that sends fixed templates and tool calls with no model call, sending only unseen states back to the LLM. In the strongest results, on a telecom troubleshooting benchmark, a small open model run this way rose from 29% to 96% of tasks solved with 93% fewer model calls and came within one task of a frontier model's accuracy, while the frontier model itself cut calls and cost by about 98% and solved slightly more tasks; a controller with no model at all beat every published method on an inventory benchmark. The paper states four claims: the decision states saturate, LLM agents do not optimize (a solver given their own probability estimates decides better), the gains come from executing rather than from telling, and the agent's own logs predict where the approach pays. If these claims hold, most model calls in bounded workflows are unnecessary for the decisions the solver already covers.

What carries the argument. The load-bearing object is the decision state: a small tuple of discrete facts the controller reads from each message with simple rules such as regular expressions — in telecom, the kind of problem, whether it is fixed yet, anything stopping a fix, and the first check not yet made in the solved order — so that different wordings collapse into the same state and the decision space saturates while the text does not. On top of that state sits a Markov decision process: moves cost a stated unit (customer turns, dollars, tokens), outcome probabilities are counts from logs, and the best move in each state comes from Bellman's equation, solved once by dynamic programming, which gives the method its name; in the worked example the solver is the classical probability-per-turn rule for ordering tests. The theory around it has three pieces: Proposition 2 bounds the cost of grouping conversations into states by $2\varepsilon$ per decision via the performance difference lemma, Proposition 3 shows an estimate error costs nothing unless a decision threshold lies between estimate and truth, and Proposition 4 gives the cost law — savings converge to $1/(\rho+f+a)$, where $\rho$ is the cost of reading a turn, $f$ the share of decisions falling back to the LLM and $a$ an audit share — so the fallback share sets the ceiling on how much cheaper deciding becomes. The controller executes the policy: covered turns are sent as fixed templates or tool calls, uncovered turns go to the LLM with the task's own prompt and tools, and the LLM's replies are read back into the state.

Core claim. Separate a decision from the text that carries it out, and a bounded workflow's decision space saturates: over 1,721 training conversations the paper's telecom reader met only 69 distinct decision states, none of them new in the final 179 conversations, while the agent's messages kept growing to 18,254 distinct texts. Because the state holds only what the decision depends on — the kind of problem, whether it is fixed yet, anything blocking a fix, and the next check in the solved order — a policy computed once from estimated probabilities suffices on almost every turn, and executing it is what removes the model calls. The paper's most direct demonstration is that execution, not knowledge, carries the gain: telling the model the solved policy in the prompt raised a weak model's accuracy but left its 18 to 23 calls per conversation unchanged, while executing the same policy reached higher accuracy at five or fewer calls; and a small open model as a BLM matched a frontier model's task accuracy, including on faults never seen in training, because the checks come from the workflow's guide rather than from the data. Two formal results carry the argument: discretizing conversations into states costs at most $2\varepsilon$ per decision (Proposition 2), and an estimate error matters only near a decision threshold (Proposition 3), which is why a policy computed from an LLM's own rough estimates matched one computed from true probabilities while the LLM choosing for itself did worse. The paper is explicit that its strongest evidence is measured against LLM-simulated customers, who may vary less than real ones.

The premise it rests on. The central claim collapses if real users, rather than the LLM-simulated customers the paper concedes 'may vary less than real ones,' keep producing decision states the controller has not seen, because the reported 93 to 98 percent model-call reductions are exactly the share of turns the controller can cover.

If the paper is right (Pith's inferences, not the authors')

In bounded workflows where most turns are decisions, executing a solved policy removes most model calls, cost and waiting time at equal or better accuracy: 93 to 98 percent fewer calls in telecom, and 0.25 to 0.61 calls per conversation for a frontier model run as a BLM.

A small open model as a BLM can reach a frontier model's accuracy in such workflows without fine-tuning on that model's conversations; a fine-tuned Qwen3.8-27B solved fewer tasks than the same model as a BLM and still called the model on every turn.

The method survives faults never seen in training, because checks come from the workflow's guide and the state records the kind of problem rather than the specific fault: on tasks whose fault types were absent from training, the open model as a BLM solved 42 of 60 versus 22 alone, and in a three-attempt setting solved 93.3 percent of tasks in all three attempts versus 40.0 percent alone.

The decidable share — the share of an agent's turns a draft controller's state can fix, measured by replaying the controller over existing logs — predicts where the approach pays before it is built: about 98 percent of a frontier model's telecom turns against 15 to 22 percent of a coding agent's SymPy turns, where a BLM helped little.

Where the whole task is deciding, the solved policy needs no model at all: on the InventoryBench benchmark a no-model controller scored 0.625, ahead of every published method (best published, 0.538), including an LLM running the best published strategy.

The cost law implies the approach is self-limiting in a useful way: once the build cost is amortized, savings cap at $1/(\rho+f+a)$, so the method itself identifies its best targets — workflows with long runs, stable recurring states and cheap reading rules — and candidate workflows could be ranked by measuring the decidable share first.

The paper's compiler analogy points to a testable extension it leaves implicit: like a tracing just-in-time compiler, a BLM could promote newly recurring states into the solved policy online, which the paper lists as future work, turning a static controller into one that keeps compiling what becomes recurrent.

A distributional corollary of saturation: as bounded workflows are automated for longer, the marginal value of each additional model call should decline, so the crossover point where a small model plus controller beats a frontier model alone should be predictable from the decidable share and the per-call price difference.

Because the strongest evidence uses LLM-simulated customers, a cautious reader should treat the telecom numbers as an upper bound on savings; applying the paper's own replay methodology to real human-agent logs would show whether human novelty outpaces the simulator and inflates the fallback share.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

A general observation in agentic workflows, especially bounded ones, is that the decision space of the LLM, when we separate a decision from generation, starts approaching saturation: the agent meets the same states again and again, and after enough conversations it stops meeting new ones. LLMs generally don't treat these decisions as an optimization problem and instead re-derive them from their context/prompt on every turn. We show that treating decision making within reasonable bounds as an optimization problem, solving it once, and reusing the solved decisions can often make an agent faster, cheaper and more reliable. Our approach, which we call Discretize, Then Decide, starts from data such as the workflow's own logs or, when none exists, runs of the LLM agent itself. It reads each conversation into discrete states and estimates the probability of each outcome and the cost of each move, until no new states appear and the probabilities stop changing. It then solves the resulting Markov decision process. The solution runs as a controller around the model, sending moves whose content is fixed in the state, such as a tool call or a templated reply, and any state it hasn't seen is sent back to the LLM. We call an LLM run this way a Bellman language model (BLM), after Richard Bellman's equation for the best move in each state. On τ2-bench telecom, the controller with no model solves 79% of tasks. As a BLM, Qwen3.8-27B, an open model, goes from 29% to 96% with 93% fewer model calls. For Claude Opus 5.5, which solves 96% of tasks on its own, running it as a BLM cuts model calls and cost per task by about 98% and lets it solve 2 more tasks. On InventoryBench, the solved policy with no model beats every published method. Where the hard part is writing content rather than choosing moves, such as resolving SymPy issues, few decisions can be solved in advance, and a BLM helps little.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-20

The authors' main claim, as the classifier picked it out 1.00

“We show that treating decision making within reasonable bounds as an optimization problem, solving it once, and reusing the solved decisions can often make an agent faster, cheaper and more reliable.”

How the main claim is stated: asserted 0.80

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.96 · human participants 0.04 · controlled comparison 0.89 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.96 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
A general observation in agentic workflows, especially bounded ones, is that the decision space of the LLM, when we separate a decision from generation, starts approaching saturation: the agent meets the same states again and again, and after enough conversations it stops meeting new ones. background 0.98 0.22 0.03 asserted 0.93
LLMs generally don't treat these decisions as an optimization problem and instead re-derive them from their context/prompt on every turn. background 0.67 0.08 0.03 asserted 0.99
We show that treating decision making within reasonable bounds as an optimization problem, solving it once, and reusing the solved decisions can often make an agent faster, cheaper and more reliable. contribution 0.88 0.78 0.88 asserted 0.80
Our approach, which we call Discretize, Then Decide, starts from data such as the workflow's own logs or, when none exists, runs of the LLM agent itself. method 0.96 0.70 0.04 asserted 0.96
It reads each conversation into discrete states and estimates the probability of each outcome and the cost of each move, until no new states appear and the probabilities stop changing. method 1.00 0.86 0.05 asserted 1.00
It then solves the resulting Markov decision process. method 1.00 0.76 0.05 asserted 0.95
The solution runs as a controller around the model, sending moves whose content is fixed in the state, such as a tool call or a templated reply, and any state it hasn't seen is sent back to the LLM. method 1.00 0.73 0.07 asserted 0.99
We call an LLM run this way a Bellman language model (BLM), after Richard Bellman's equation for the best move in each state. contribution 0.52 0.63 0.06 asserted 0.92
On $\tau^2$-bench telecom, the controller with no model solves 79% of tasks. result 1.00 0.91 0.06 asserted 1.00
As a BLM, Qwen3.8-27B, an open model, goes from 29% to 96% with 93% fewer model calls. result 1.00 0.93 0.11 asserted 1.00
For Claude Opus 5.5, which solves 96% of tasks on its own, running it as a BLM cuts model calls and cost per task by about 98% and lets it solve 2 more tasks. result 1.00 0.92 0.09 asserted 1.00
On InventoryBench, the solved policy with no model beats every published method. result 1.00 0.88 0.09 asserted 1.00
Where the hard part is writing content rather than choosing moves, such as resolving SymPy issues, few decisions can be solved in advance, and a BLM helps little. limitation 0.99 0.79 0.05 asserted 0.94

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 9,726 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (13)
Figure 1 of the paper
Figure 1. Discretize, Then Decide. Top: built once, run every turn. The key point is that when the state fixes the move, the turn never reaches the model. Bottom: tasks solved against agent tokens in telecom, where up and to the left is better. Telling the model the solved policy moves it up. Executing it moves it up and left. Other variants
Figure 2 of the paper
Figure 2. Step 1 on the worked example: reading conversations into states. Customers word the same fact in many ways. Notice that the reader keeps only the fact, so different conversations end up in the same state. abroad, roaming off airplane mode on network mode 2G only 1 2 3 4 5 6 7 8 eight past tasks 6 of 8 0.75 4 of 8 0.50 4 of 8 0.50 probability 1 1 2 turns to check
Figure 3 of the paper
Figure 3. Step 2 on the worked example: estimating. Each column is a past task, with a dot for each fault it had. Counting the dots gives the probability of each fault. Notice that roaming is off in six of the eight, the most common fault. 3.3 Step 2: estimate the probability of each outcome and the cost of each move Next, we count. For a check, the outcome that matters is whether it finds a fault
Figure 4 of the paper
Figure 4. Step 3 on the worked example: solving. Left: the solved policy asks first about the fault with the highest probability per turn. Right: the total turns each of the six orders takes over the eight past tasks, where fewer is better. Notice that the solved order takes the fewest turns, and asking about travel last takes the most. the highest probability per turn, the classical rule for ordering tests (Kadane and Simon, 1977). Travel comes first at 0.75 per turn, then airplane mode at 0.5, then the network mode at 0.25. The rule is exact when there’s one fault to find. With several at…
Figure 5 of the paper
Figure 5. Step 4 on the worked example: running. A new customer has all three faults. When the state fixes the move, the controller sends it with no model call. Notice that only the off-script question goes to the LLM, and its reply is read back into the state. call. Because every uncovered turn goes to the LLM, the solved policy doesn’t have to cover every state. It only has to be right about the turns it covers. The controller also looks at what the LLM sends. Suppose the model tries to hand the customer to a human while checks remain and nothing is stopping a fix, as a locked SIM card…
Figure 6 of the paper
Figure 6. The real task the worked example comes from, abridged from the logged transcripts. Dark tags mark each fault an agent fixed. Notice that alone, Qwen3.8-27B never asks about travel and hands the customer to a human. In the BLM, the controller sends every move and fixes all three faults with no model call. The turns come from Opus’s conversations. A check’s cost is the number of customer turns from the one where Opus first asked about it to the one where it asked about the next. We averaged this over the 229 solved conversations among the first 233 to finish. Asking about travel took…
Figure 7 of the paper
Figure 7. The cost law (Proposition 4) for an example with 120 decision states, 4 moves each, 3 decisions per conversation, 𝜌 = 0.02 and 𝑎 = 0.02, and building costing one model call per state and move. Each curve is a fallback share. Notice that savings level off at 1/(𝜌 + 𝑓 + 𝑎): about 14, 7 and 3 times cheaper for 3%, 10% and 25% fallback
Figure 8 of the paper
Figure 8. Distinct values seen as training conversations accumulate, as a share of the number seen by the end: 1,721 telecom training conversations that ran without a solved policy, replayed in the order they were collected through the third controller’s reader. The decision state the solved policy acts on levels off. The full record of checks made and the agent’s messages keep growing to the end. doesn’t follow the fault rates in its own conversations (
Figure 9 of the paper
Figure 9. Every agent turn on test set 1 with Claude Opus 5.5. Each row is one conversation (114, sorted by model calls) and each cell one agent turn, colored by what produced it. Left: Opus alone calls the model on every turn. Right: notice that in the Bellman model almost every turn is an executed lookup or template, and 82 of the 114 conversations contain no model call at all. Conversations with the BLM have more turns, because each check is a separate instruction. from $0.43–0.49 to $0.012–0.024. On test sets 1 and 2, the customer’s total waiting time per conversation fell from 77.8 and…
Figure 10 of the paper
Figure 10. Where InventoryBench’s gain comes from. Notice that better beliefs alone bring the published rule to about the best published level, and the solved decision rule does the rest. Where the gains come from
Figure 11 of the paper
Figure 11. The decidable share, measured from the LLM agent’s own conversations, against the model calls BLM removed. Horizontal: the share of the LLM agent’s turns where a frozen controller’s state fixes the content, found by replaying the controller over those conversations (InventoryBench: 100% by construction). Vertical: model calls per conversation removed by the BLM. Telecom: one point per test set. The bar for SymPy spans two ways of matching the agent’s turns to the controller’s moves. Notice that points above the diagonal remove more calls than the share predicts, because the solved…
Figure 12 of the paper
Figure 12. Calibration matters only where it flips a decision. Left: expected cost of the three moves that end a CI triage, as a function of the belief 𝑏 that a failure is a regression; the cheapest move changes at 𝑏 = 0.125 and 𝑏 = 0.75. Right: the controller’s beliefs before calibration, against outcomes, on 198 validation failures. Notice that its most confident bin, where it backs out changes, was right 72% of the time (51 of 71), below the 0.75 needed for backing out to pay. 8 Building a BLM for a new workflow How would you build a BLM for a workflow of your own? The recipe is short. Use…
Figure 13 of the paper
Figure 13. Tasks solved by number of faults, for each test set with its own frozen controller. Points are success rates with Wilson 95% intervals. Notice that alone, Qwen3.8-27B does worse as faults are added, and as a BLM it declines much less. J A simulated world for comparing decisions To compare an LLM’s decisions with optimal ones exactly, we used a simulated world of a coding agent with hidden variables, in which the expected cost of any controller can be computed. It has 46 reachable states. An LLM given a one-page description of the task estimated, in one pass, the probabilities of…

Figures are the authors', reproduced from the paper.

References (46)
  1. [3] For test set 1, which the first controller ran, we replay the second controller; replaying the third instead changes no share by more than 6.4 points. Table 9 gives the shares. The controller’s state fixes nearly all of Opus’s turns in telecom, and a small share of the SymPy a... on Pith
  2. [5] Tom Bocklisch, Thomas Werkmeister, Daksh Varshneya, and Alan Nichol. Task-oriented dialogue with in-context learning. arXiv:2402.12234, on PithPith's reviewarXiv
  3. [6] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. FireAct: Toward language agent fine-tuning.arXiv preprint arXiv:2310.05915, on PithPith's reviewarXiv
  4. [7] Revision 7b928d8; non-autoregressive decision model, ModernBERT-large backbone, 421M parameters. Wenxuan Ding, Nicholas Tomlin, and Greg Durrett. Calibrate-then-act: Cost-aware exploration in LLM agents. arXiv:2602.16699, on PithPith's reviewarXiv
  5. [10] doi: 10.1287/msom.2021.1064. Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in Markov decision processes.Artificial Intelligence, 147(1–2):163–223, on Pithdoi:10.1287/msom.2021.1064
  6. [11] were each hashed before their single scoring run on the test window, and registered afterwards, in the amendment. The hashed file of the first was later edited in place into the second, so its hash record and its results remain, but the file doesn’t. The two registrations also... on Pith
  7. [14] doi: 10.1109/FOCS61266.2024.00024. Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in LLMs. InAdvances in Neural Information ... on Pithdoi:10.1109/focs61266.2024.00024
  8. [15] doi: 10.52202/079017-0762. InventoryBench. InventoryBench leaderboard. https://tianyipeng.github.io/ InventoryBench/leaderboard.html, on Pithdoi:10.52202/079017-0762
  9. [16] Accessed 2026-10-08. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), on Pith
  10. [20] The first random number we took was 3s older than the freeze, so before any test ran the tasks were drawn again from one published two minutes later, as the registration records. As on test set 1, conversations without an outcome were rerun (87 of Opus alone’s 114), and three ... on Pith
  11. [22] doi: 10.1016/B978-1-55860-377-6.50052-9. Ollie Liu, Deqing Fu, Dani Yogatama, and Willie Neiswanger. DeLLMa: Decision making under uncertainty with large language models. InInternational Conference on Learning Representations (ICLR), on Pithdoi:10.1016/b978-1-55860-377-6.50052-9
  12. [25] doi: 10.1016/B978-1-55860-377-6.50055-4. Atif Memon, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. Taming Google-scale continuous testing. InProceedings of the International Conference on Software Engineering: Software Engineering in Prac... on Pithdoi:10.1016/b978-1-55860-377-6.50055-4
  13. [28] doi: 10.1145/3476105. Peng Qi, Chunliang Lyu, Gang Li, Fabian Chan, Cheng Chang, Ignacio Cases, and Will Lu. Agent behavior as code: Efficient and robust LLM agents with programmatic specifications. arXiv:2610.04824, on PithPith's reviewarXiv
  14. [29] Revision 1d4bf0f. Release blog: https://qwen.ai/blog?id= qwen3.8. Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. NeMo guardrails: A toolkit for controllable and safe LLM applications with pro- grammable rails. InProceedings of... on Pith
  15. [30] doi: 10.18653/v1/2023. emnlp-demo.40. Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, and Pang Wei Koh. Context language models. arXiv:2609.37725, on PithPith's reviewarXiv
  16. [31] Commit 5bfa7e3. Pascal Sowietzki, Steffen Holter, Yannick Metz, Emil Kasper, Raphael Baur, and Mennatallah El-Assady. Ask your advisor: Learning when to consult stronger models in agentic systems. arXiv:2610.13870, on PithPith's reviewarXiv
  17. [33] doi: 10.18653/v1/P17-1062. Bellman Language Models 40 Caleb Winston, Ron Yifeng Wang, Azalia Mirhoseini, and Christos Kozyrakis. Agent JIT compila- tion for latency-optimizing web agent planning and scheduling. InProceedings of the 43rd International Conference on Machine Lear... on Pithdoi:10.18653/v1/p17-1062
  18. [34] arXiv:2605.21470. Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. StateFlow: Enhancing LLM task-solving through state-driven workflows. InFirst Conference on Language Modeling (COLM), on PithPith's reviewarXiv
  19. [35] Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. FlowBench: Revisiting and benchmarking workflow-guided planning for LLM-based agents. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 10883–10900, on Pith
  20. [36] doi: 10.18653/v1/2024.findings-emnlp.638. Linwei Xin. Technical note—understanding the performance of capped base-stock policies in lost-sales inventory models.Operations Research, 69(1):61–70, on Pithdoi:10.18653/v1/2024.findings-emnlp.638
show 26 more
  1. [37] doi: 10.1287/opre. 2020.2019. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.𝜏-bench: A benchmark for tool-agent-user interaction in real-world domains. InInternational Conference on Learning Representations (ICLR), on Pithdoi:10.1287/opre
  2. [39] doi: 10.18653/v1/2024. findings-acl.181. Alex L. Zhang, Tim Kraska, and Omar Khattab. Recursive language models. arXiv:2512.24601, 2025a. Chenyu Zhang, Wonbin Kweon, and Jiawei Han. How strongly should task state influence an LLM agent? arXiv:2609.25686, on PithPith's reviewarXiv
  3. [40] Qizheng Zhang, Michael Wornow, and Kunle Olukotun. Agentic plan caching: Test-time memory for fast and cost-efficient LLM agents. InAdvances in Neural Information Processing Systems (NeurIPS), volume 38, pages 114372–114398, 2025b. doi: 10.52202/085713-3451. Bellman Language M... on Pithdoi:10.52202/085713-3451
  4. [44] Against a simpler rule taken from the written policy, which blocks every hand-off unless the reader sees a reason in the written policy for it, such as a locked SIM card:−3.1 [−9.6, 3.5], so no better. The token forecast missed:+32,103 billed tokens per conversation, agent and... on Pith
  5. [45] with rank 16 on the attention, linear-attention and MLP projections, which gives 108.8 million trainable parameters (0.40% of the model). We used AdamW at a learning rate of10−4 with cosine decay, eight conversations per update and three epochs, with the loss on the agent’s tu... on Pith
  6. [151] While four comparisons on test set 2 were running, before any had finished, we fixed in writing (9 October 2026, 16:15 UTC) how the paper would be framed. The four were Qwen with the checklist, with thinking, with thinking and the checklist, and with the oracle plan. If the be... on Pith
  7. [1951] doi: 10.2307/1906813. Jackie Baek, Yaopeng Fu, Will Ma, and Tianyi Peng. AI agents for inventory control: Human-LLM- OR complementarity.arXiv preprint arXiv:2602.12631, on PithPith's reviewarXiv
  8. [1977] doi: 10.1214/aos/1176343791. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains.Artificial Intelligence, 101(1–2):99–134, on Pithdoi:10.1214/aos/1176343791
  9. [1983] doi: 10.1001/jama.1983. 03330370053031. Bellman Language Models 37 David Heckerman, John S. Breese, and Koos Rommelse. Decision-theoretic troubleshooting. Communications of the ACM, 38(3):49–57, on Pithdoi:10.1001/jama.1983
  10. [1991] doi: 10.1287/moor.16.3.580. Vishvesh Bhat, Asad Aali, and Muhammad Ahmed Mohsin. Compiling expert procedures into verified programs: When a small model is enough. arXiv:2610.19148, on PithPith's reviewarXiv
  11. [1992] doi: 10.1145/143095.143114. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations (ICLR), on Pithdoi:10.1145/143095.143114
  12. [1995] doi: 10.1145/203330.203341. Urs Hölzle, Craig Chambers, and David Ungar. Debugging optimized code with dynamic deopti- mization. InACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 32–43, on Pithdoi:10.1145/203330.203341
  13. [1998] doi: 10.1016/S0004-3702(98)00023-X. Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. InProceedings of the International Conference on Machine Learning (ICML), pages 267–274, on Pithdoi:10.1016/s0004-3702(98)00023-x
  14. [2002] Seongjae Kang, Taehyung Yu, and Sung Ju Hwang. PolicyGuide: From guarding one action to guiding the whole workflow for policy-compliant LLM agents. arXiv:2608.19861, on PithPith's reviewarXiv
  15. [2007] doi: 10.1016/j.csl.2006. 06.008. Jason D. Williams, Kavosh Asadi, and Geoffrey Zweig. Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. InProceedings of the 55th Annual Meeting of the Association for Computation... on Pithdoi:10.1016/j.csl.2006
  16. [2008] doi: 10.1111/j.1541-0420.2007.00888.x. National Institute of Standards and Technology. NIST randomness beacon, version 2.0.https: //beacon.nist.gov/home, on Pithdoi:10.1111/j.1541-0420.2007.00888.x
  17. [2009] doi: 10.1145/1542476.1542528. Joren Gijsbrechts, Robert N. Boute, Jan A. Van Mieghem, and Dennis J. Zhang. Can deep reinforcement learning improve inventory management? performance on lost sales, dual- sourcing, and multi-echelon problems.Manufacturing & Service Operations Man... on Pithdoi:10.1145/1542476.1542528
  18. [2013] doi: 10.1109/JPROC.2012.2225812. Min-Young Yu, Tony Kim, and Jang Won Choi. NOMOS: Compiling written policies into statically verified tool-call gates for LLM agents. arXiv:2610.11030, 2026a. Ziwen Yu, Ivan Koychev, Elizabeth Coulthard, Ting Zhou, Bolin Chen, Dian Hong, Zinuo ... on PithPith's reviewarXiv
  19. [2014] doi: 10.1145/2635868.2635920. Mateusz Machalica, Alex Samylkin, Meredith Porth, and Satish Chandra. Predictive test selection. InProceedings of the International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 91–100, on Pithdoi:10.1145/2635868.2635920
  20. [2017] doi: 10.1109/ICSE-SEIP.2017.16. Satoshi Morita, Peter F. Thall, and Peter Müller. Determining the effective sample size of a parametric prior.Biometrics, 64(2):595–602, on Pithdoi:10.1109/icse-seip.2017.16
  21. [2019] doi: 10.1109/ICSE-SEIP.2019.00018. R. Andrew McCallum. Instance-based utile distinctions for reinforcement learning with hidden state. InInternational Conference on Machine Learning (ICML), pages 387–395, on Pithdoi:10.1109/icse-seip.2019.00018
  22. [2021] doi: 10.1145/3468264. 3473931. Lihong Li, Thomas J. Walsh, and Michael L. Littman. Towards a unified theory of state abstraction for MDPs. InInternational Symposium on Artificial Intelligence and Mathematics (ISAIM), on Pithdoi:10.1145/3468264
  23. [2022] doi: 10.1145/3510003.3510233. NormFerns,PrakashPanangaden,andDoinaPrecup. MetricsforfiniteMarkovdecisionprocesses. InConference on Uncertainty in Artificial Intelligence (UAI), pages 162–169, on Pithdoi:10.1145/3510003.3510233
  24. [2023] doi: 10.1145/3600006.3613165. Johannes Lampel, Sascha Just, Sven Apel, and Andreas Zeller. When life gives you oranges: Detecting and diagnosing intermittent job failures at Mozilla. InProceedings of the ACM Joint European Software Engineering Conference and Symposium on the F... on Pithdoi:10.1145/3600006.3613165
  25. [2024] doi: 10.52202/079017-4164. Eitan Altman.Constrained Markov Decision Processes. Chapman and Hall/CRC, on Pithdoi:10.52202/079017-4164
  26. [2026] Introduces the InventoryBench benchmark. Ali Baheri and Peng Wei. Controller-agnostic runtime safety certificates for tool-using AI agents via data-driven robust reachability. arXiv:2610.16865, on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment