Pith. sign in

REVIEW 3 major objections 5 minor 14 references

Four frontier LLMs see hidden supply-chain stress almost perfectly and still fail to act on it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:36 UTC pith:ZHKPO4UT

load-bearing objection A genuinely fair-oracle benchmark with a real, fixable confound: the skill score conflates missing world-model parameters with poor control, so the 'failures are in control, not perception' headline is not yet proven. the 3 major comments →

arxiv 2607.13618 v1 pith:ZHKPO4UT submitted 2026-07-15 cs.AI cs.LG

STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle

Sagar Deb , Ashwanth Krishnan This is my paper
classification cs.AI cs.LG
keywords knowing-doing gapLLM agentsPOMDPfair oracleBayes filtersupply-chain replenishmentstate estimationcontrol
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

STOCKTAKE is a 26-week inventory-replenishment benchmark cast as a factored partially observable Markov decision process with six hidden factor processes, built so that a fair reference policy is computable. That reference — an exact Bayes filter per factor driving a rollout on the identical observation stream the LLM receives — lets the authors attribute the gap between an agent and the oracle to acting, not to privileged information. Across fifty seeds, four frontier LLMs all detect 84–88% of hidden stress episodes within about half a week, yet skill scores span 0.62 to −0.23, with two models ending below a symptom-blind base-stock policy on roughly half their seeds. The paper's claim: on this task, frontier-model failures are concentrated in control, not state estimation, and the knowing-doing gap runs in both directions — under-responding to persistent stress and over-responding with costly protective actions.

Core claim

On STOCKTAKE, detection is nearly uniform while costs diverge sharply: all four frontier LLMs name hidden-factor stress episodes 84–88% of the time, with mean detection lag under half a week, so the 'seeing' part of the task is close to solved. Yet skill scores — where 0 is a symptom-blind base-stock floor and 1 is the fair oracle's mean — range from 0.62 to −0.23, and two of four models finish below the floor on roughly half their seeds. The fair oracle is the load-bearing construction: exact Bayes filters per factor, conditioned on exactly the weekly observation dictionary the LLM sees, so shortfall cannot be blamed on withheld information (the one disclosed asymmetry is that the oracle us

What carries the argument

The fair Bayes-filter oracle and the skill score. The oracle is one exact predict–correct filter per hidden factor, updated on the same weekly observation dictionary serialized into the LLM's prompt, then a rollout over about 200 sampled futures scores a menu of candidate actions (order quantities, routes, suppliers, and mitigation levers gated on posterior risk); it never reads hidden state and is executed through the same action API as the LLM. Because the hidden dynamics are factored and action-independent, the posterior splits into six small filters and the reference is computable. The skill score locates each run on a scale anchored by 0 (a symptom-blind base-stock policy) and 1 (the or

Load-bearing premise

The clean split between seeing and doing assumes the qualitative description in the prompt is enough for an LLM to reconstruct the exact quantitative cost structure; if it isn't, part of the measured 'action gap' is really a gap about exact numbers.

What would settle it

Give the LLM agents the same quantitative generative parameters the oracle uses (transition probabilities, regime means, and the full cost table). If skill scores rise toward 1, the control gap is substantially a knowledge gap about numbers; if the spread from 0.62 to −0.23 persists, the gap is genuinely in acting on correct beliefs.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the knowing-doing gap dominates, interventions aimed at turning a correct diagnosis into a calibrated order — planning scaffolds, explicit contingency rules, action-menu support — should raise skill more than better inference tooling.
  • Benchmarks for partially observable LLM-agent tasks should report perception and control metrics separately, or they will misattribute failure causes.
  • A symptom-blind base-stock rule can beat a frontier LLM on a substantial share of tapes, a concrete deployment flag that default policies should sometimes be simple rules rather than capable models.
  • The knowing-doing rate is a severity-dependent diagnostic, not a model ranking; it should be read alongside severity-matched baselines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the disclosed asymmetry: give the LLM the same quantitative generative parameters (transition tables, regime means, exact costs) the oracle uses; if skill scores jump toward 1, part of the 'action gap' is really an epistemic gap about numbers, not purely control.
  • The construction generalizes beyond supply chains: any factored POMDP whose exact filter is computable can become a knowing-doing benchmark for other hidden-state decision problems, such as portfolio management or clinical logistics.
  • The severity confound in the knowing-doing rate suggests a matching protocol — pairing diagnosed and undiagnosed stress weeks on severity features — to estimate the causal effect of diagnosis on stockouts.
  • Each model–seed cell is a single run; repeating runs per cell would tighten the estimate of how much of the cross-model skill spread is systematic versus single-run noise.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces STOCKTAKE, a 26-week supply-chain replenishment benchmark built as a factored POMDP with six hidden factor processes. Each weekly decision requires the agent to read noisy dashboard symptoms, write a rationale, and place an order. The authors propose two reference policies: a symptom-blind base-stock floor and a 'fair oracle' consisting of six exact Bayes filters driving a rollout policy on the same observation stream the LLM receives. Skill is defined as the agent's cost improvement over the floor, normalized by the oracle's improvement. Belief-side metrics—detection lag and knowing-doing rate—are derived from an LLM grader that labels which hidden factors each rationale names. The paper reports that four frontier models detect 84–88% of stress episodes with sub-week lags, yet skill scores range from 0.62 to −0.23, with two models below the floor on roughly half their seeds. The central claim is that failures are concentrated in control, not state estimation, and that the 'fair oracle' makes this split measurable.

Significance. The benchmark construction is genuinely useful: the event-tape protocol fixes the hidden trajectory as a pure function of the seed, the Bayes-filter reference is computable, all cost and transition parameters are fully specified in Appendix A, and the stress-profile grouping (ISOLATED/PERSISTENT/COMPOUND) makes the interaction of hidden failures a controlled variable. The paper is also unusually candid in Section 6 about limitations such as single runs, grader audit scope, and the severity confound. However, the central inference—that the skill gap measures a control failure rather than a model-knowledge gap—is undermined by the disclosed asymmetry that the oracle uses the true generative parameters while the LLM receives only a qualitative description. Because the belief metrics only grade factor names and not posterior accuracy, the paper's own evidence does not cleanly separate 'knowing' from 'acting.' If the authors can supply a reference policy that does not use privileged quantitative parameters, or substantially narrow their claim, the benchmark and the empirical comparison would be a solid contribution to LLM-agent evaluation.

major comments (3)
  1. [§4, §6, Eq. (1)] The central attribution—that skill shortfall equals a control failure—does not follow from the disclosed asymmetry. The oracle's filters use the true transition tables P_i and regime means behind Q_i (Appendix B), while the LLM prompt (Appendix E) gives exact costs but only qualitative descriptions of dynamics: no onset rates, persistence probabilities, age caps, or regime means. An agent can name a factor and still under-order because it does not know that port congestion persists with probability 0.85 or that a demand decline is sticky; under the paper's own POMDP framing, that is a state-estimation/model-knowledge gap, not a knowing-doing gap. The detection metrics only require the factor to be named, so they cannot rule out such a gap. The paper calls the oracle 'fair' and says shortfall is 'attributable to acting' (§1, §6), but by its own fairness criterion ('denied exactly the info
  2. [§6, Appendix C] All belief-side metrics for the full fifty-seed, four-model sample depend on the gpt-5-mini grader, but the audits (a 30-row manual read and a second-grader agreement check on 100 sampled weeks) were performed only on the original twenty-seed grading run; the thirty expansion seeds and the later models were not re-audited (§6). Since detection and knowing-doing rates are central to the 'seeing is uniform' result, an un-audited grader on a large fraction of the episodes leaves open the possibility that label drift changes the reported detection and KD values. Please report audit statistics for the full dataset, or at minimum per-seed/model grader breakdowns; otherwise the perception-side uniformity claim is under-supported.
  3. [§5, Table 4] The knowing-doing rate is presented as evidence that 'correct diagnosis coexists with empty shelves' (§4), but Table 4 shows that diagnosed stress weeks stock out more often than undiagnosed stress weeks for every model, and the text concedes this is a severity confound. With a confound of this size left unmodeled, the pooled KD values (0.24–0.43) cannot by themselves support the claim that agents knew and failed to act; they may simply reflect that severe weeks are both more noticeable and more costly. Please provide a severity-matched comparison (e.g., stockout rates on diagnosed vs. undiagnosed weeks within matched severity bands) or a causal decomposition. Without it, the knowing-doing claim rests on anecdotal trace readings (Appendix D) rather than on the metric named in Eq. (2).
minor comments (5)
  1. [Abstract] The phrase 'naming factors slightly faster than the two that beat it' is ambiguous; Table 3 reports mean detection lag in weeks, and the direction should be stated explicitly (shorter lag).
  2. [Table 2] The table note says the seed with negative headroom is excluded from skill means, but the 'beats floor' count includes that seed; the column headers and note should clarify which denominator applies to which statistic to avoid confusion.
  3. [Appendix B] The oracle's 'small menu' is described as order quantities {0, four weeks, six weeks of expected demand}, but the exact expected-demand definition is not given; please specify how 'expected demand' is computed from the belief.
  4. [§4] Eq. (1) is undefined when C_base = C_orc; the text excludes one such seed but does not say how many seeds are left in each group for Table 2. The group counts (11/15/23) are helpful, but a note that the excluded seed belongs to which group would improve transparency.
  5. [Appendix D] The trace excerpts are compelling but are presented as anecdotal support; the sentence 'In every one of Claude Sonnet 5's nine below-floor seeds, its spending on air expedite alone exceeds its entire excess over the floor' is quantitative and should be moved into the main results with a footnote.

Circularity Check

0 steps flagged

No circularity: the skill-score construction is an external cost comparison; the disclosed oracle asymmetry is a validity limitation, not a circular step.

full rationale

The paper's derivation chain is a benchmark measurement, not a derivation of predictions from fitted inputs. The skill score (Eq. 1) is defined as a linear placement of the agent's total cost between a fixed base-stock floor and a Bayes-filter oracle cost, both computed on the same seed. The oracle's filter and rollout use the true generative parameters, which is disclosed as the 'one disclosed asymmetry' (Section 4; Section 6). This is a real threat to the interpretation that shortfall against the oracle is purely a control gap: an agent that lacks the exact transition probabilities and regime means can under-order for epistemic reasons, not purely action reasons. But that is a construct-validity / confound issue, not a circularity: the paper does not fit any parameter to agent outcomes and then rename it a prediction; no equation reduces to itself; and the cross-model comparison (uniform detection, divergent skill) is an independent empirical observation. There are no self-citations and no imported uniqueness theorems. The paper also explicitly qualifies its own claim in Section 6 and in the severity confound check (Table 4), which further indicates the metrics are not circularly defined. Therefore, score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

No new entities, forces, or quantities are postulated. The benchmark is a synthetic factorial simulation. The main hidden costs are: (1) design choices in the oracle's candidate menu and risk thresholds, (2) hand-chosen seed selection with thresholds described qualitatively, and (3) the disclosed asymmetry that the oracle uses true generative parameters. The environment is fully specified in the appendices, so the ledger is comparatively clean.

free parameters (4)
  • Oracle candidate-menu thresholds = 0.3 (freight lock, air expedite), 0.5 (inspection)
    Appendix B: the oracle gates mitigation levers on posterior risk thresholds; these thresholds are chosen by the authors and affect the oracle's cost, hence the skill-score scale.
  • Base-stock safety factor z
    Derived from the holding/stockout cost ratio (z≈1.67) rather than fitted to data, so it does not really count as a free parameter; but the floor policy's S is a design choice defining the skill scale's zero.
  • Seed-selection thresholds
    50 seeds are selected by thresholds on stress features from a pool of 200; the thresholds that separate ISOLATED/PERSISTENT/COMPOUND and discard benign seeds are specified only qualitatively in Section 3.3, so the stress-axis results depend on those choices.
  • Oracle replication count and trajectory budget = 20 replications; 191-210 sampled futures
    Appendix B: these are practical approximations; the oracle is a stochastic rollout policy, not an exact optimum, so its mean defines the skill-score unit.
axioms (3)
  • domain assumption The six hidden factor processes are action-independent and their dynamics and observations factor exactly as specified.
    This is what makes the exact per-factor Bayes filter a valid posterior and makes the oracle 'fair' (Section 3.2). If factors were coupled or action-dependent, the reference would be approximate.
  • domain assumption The LLM grader reliably maps written rationales to named factors.
    All belief-side metrics (detection lag, knowing-doing rate) depend on gpt-5-mini's binary judgments, audited only on the original twenty-seed run; the paper reports an 89% agreement with a second grader on the metric-relevant component.
  • standard math Cost parameters and transition tables in Appendix A faithfully specify the world the LLM faces.
    Classical inventory-control background: base-stock policy, newsvendor critical ratio, Bayes-filter predict-correct update. No ad hoc mathematics; the innovation is the application.

pith-pipeline@v1.3.0-alltime-deepseek · 17105 in / 6969 out tokens · 52980 ms · 2026-08-02T04:36:47.674811+00:00 · methodology

0 comments
read the original abstract

LLM agents are increasingly evaluated on multi-week decision tasks in which the state that drives cost is never directly observed. On such tasks the final cost cannot say why an agent failed: it may have misread the world, or read it correctly and still failed to act (the knowing-doing gap). Existing evaluations cannot separate these two failures; their reference policies either read privileged information the agent never sees, or are missing altogether. We introduce STOCKTAKE, a 26-week supply-chain replenishment benchmark built as a factored partially observable Markov decision process with six hidden factor processes, designed so that a fair reference policy is computable: an exact Bayes filter per factor drives a rollout policy on the identical observation stream the agent receives. Scoring each run between a symptom-blind base-stock floor (0) and this oracle (1) yields a skill score, and grading each week's written rationale yields a stated-belief detection lag and a knowing-doing rate, so state estimation and control are measured separately. On fifty seeds with curated stress profiles, Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5 detect 84-88% of hidden failures, typically within a week of onset, yet span skill scores from 0.62 to -0.23: two of the four end below the symptom-blind floor while naming factors slightly faster than the two that beat it. The failure has two faces. Where stress persists, 34-43% of correctly diagnosed stress weeks still end in stockout for every model, a rate that partly reflects the severity of the weeks models notice. That rate also runs opposite to skill: the two models under the floor stock out least on diagnosed weeks, so under-response is only one face of the gap, and their traces point to the other, responses whose cost exceeds what they protect. STOCKTAKE measures both directions of that failure.

Figures

Figures reproduced from arXiv: 2607.13618 by Ashwanth Krishnan, Sagar Deb.

Figure 1
Figure 1. Figure 1: Measuring the knowing-doing gap with a fair oracle. Six hidden factor processes evolve on a fixed, action-independent event tape (left; dark segments mark stress regimes) and emit noisy symptoms into a weekly dashboard. The identical observation stream feeds both the LLM agent and a Bayes-filter reference policy (one exact filter per factor driving a rollout), so neither sees the hidden state. The agent’s … view at source ↗
Figure 2
Figure 2. Figure 2: The hidden world behind the dashboard. The hidden state is six independent Markov [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Seeing is uniform; doing is not. (a) All four models detect 84–88% of their hidden-factor stress episodes (283 per model over the fifty seeds), typically within a week of onset. (b) Group-mean skill by stress profile (lines), with per-seed scores as faint dots. Costs diverge across models despite panel (a): DeepSeek-V4-Pro and Grok 4.5 finish below the symptom-blind floor overall [PITH_FULL_IMAGE:figures/… view at source ↗
Figure 4
Figure 4. Figure 4: The freeze, week by week (Claude Sonnet 5, seed 1). Top: on-hand inventory; the shaded band is the true port-congestion window (weeks 17–24) and the rugs above mark the true demand windows, none of which the agent observes directly. Bottom: units ordered by sea and by air each week. The agent diagnoses the congestion in week 16, its sea orders pile up behind the blocked port, and in weeks 22–24 it stops or… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 10 linked inside Pith

  1. [1]

    Backlund and L

    A. Backlund and L. Petersson. Vending-Bench: A benchmark for long-term coherence of autonomous agents. arXiv:2502.15840, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Zhang, J

    L. Zhang, J. Wang, J. Wu, and Z. Zhang. RetailBench: Evaluating long-horizon autonomous decision- making and strategy stability of LLM agents in realistic retail environments. arXiv:2603.16453, 2026

    Pith/arXiv arXiv 2026

  3. [3]

    X. Zhao, Y . Xie, C. Chen, and Y . Sun. AIM-Bench: Evaluating decision-making biases of agentic LLM as inventory manager. arXiv:2508.11416, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Quan and Z

    Y . Quan and Z. Liu. InvAgent: A large language model based multi-agent system for inventory management in supply chains. arXiv:2407.11384, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Schmied, J

    T. Schmied, J. Bornschein, J. Grau-Moya, M. Wulfmeier, and R. Pascanu. LLMs are greedy agents: Effects of RL fine-tuning on decision-making abilities. arXiv:2504.16078, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Sobotka, M

    J. Sobotka, M. O. Karabag, and U. Topcu. Why do LLMs struggle in strategic play? Broken links between observations, beliefs, and actions. arXiv:2605.00226, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    Cheng, C

    Y . Cheng, C. Fan, M. JafariRaviz, K. Rezaei, and S. Feizi. Model-adaptive tool necessity reveals the knowing-doing gap in LLM tool use. arXiv:2605.14038, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    Z. Jin, Y . Chen, F. Leeb, L. Gresele, O. Kamal, Z. Lyu, K. Blin, F. Gonzalez Adauto, M. Kleiman-Weiner, M. Sachan, and B. Schölkopf. CLadder: Assessing causal reasoning in language models. arXiv:2312.04350,

    Pith/arXiv arXiv

  9. [9]

    Zeˇcevi´c, M

    M. Zeˇcevi´c, M. Willig, D. S. Dhami, and K. Kersting. Causal parrots: Large language models may talk causality but are not causal. arXiv:2308.13067, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Samanta, A

    A. Samanta, A. Magesh, T. Lancewicki, A. Jain, Y . Yu, P. Sajda, K. Hassani, A. Modi, D. R. Jiang, and Y . Efroni. BayesBench: Evaluating LLM belief trajectories under multi-turn evidence accumulation. arXiv:2606.30850, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    stockout

    E. Bayraktar and M. Ludkovski. Inventory management with partially observed nonstationary demand. Annals of Operations Research, 176(1):7–39, 2010. 10 Appendix A World and Cost Parameters All values below are read directly from the environment source; they fully specify the weekly cost function and the six hidden factor processes of Section 3. Costs.Each ...

    2010

  12. [12]

    freight remains cheap

    RULED-OUT or BENIGN mentions do NOT count. "freight remains cheap" = no freight. "the weak fill looks noisy rather than confirmed deterioration" = no supplier. "lanes normal" = no disruption. Naming a factor to say it is fine is the OPPOSITE of naming it as a problem

  13. [13]

    AQL reject -- inspected batch

    FACTUAL symptom reports DO count even without editorializing. "AQL reject -- inspected batch" = quality. "port congestion (berth_wait 16) delayed arrivals longer than expected" = port. The manager asserts a problem exists 12 by reporting its symptom as the cause of something

  14. [14]

    I’ll keep using the freight lock

    PRECAUTIONS without a current problem do NOT count. "I’ll keep using the freight lock" alone = no freight. "avoiding spot until it stabilizes" after calling the data noise = no supplier. But acting BECAUSE of an asserted current problem ("air-expediting because the port is holding arrivals") = count that factor. Worked example -- rationale: "Freight lock ...