{"id":"4c19111e-6a52-48a1-b3af-b26e59646e18","arxiv_id":"2607.13618","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Frontier LLM agents detect hidden supply-chain stress almost equally well (84–88% of episodes) but vary from skill 0.62 to −0.23, with two of four models acting worse than ignoring symptoms.","lead":"STOCKTAKE is a 26-week simulated supply-chain test that scores AI agents on how well they both read hidden problems and act on what they read, using a fair computer policy as the reference. It found the AI models all noticed hidden faults about equally well but differed hugely in cost—two of the four did worse than a rule that ignores all symptoms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fair oracle's true-parameter advantage may conflate missing quantitative world model with action failure; the control-vs-estimation claim is not yet clean.","rationale":"The reader's verdict identifies the oracle's true-parameter asymmetry as the weakest assumption and returns CONDITIONAL. I agree this is the most load-bearing concern. The paper's headline claim—that the fair oracle makes the split between perception and action measurable—depends on the premise that the oracle's advantage over the LLM is solely in how it acts on beliefs, not in what it knows about the world. But the oracle knows the exact generative model, and the LLM does not. This is not a 'privileged' oracle in the sense of seeing hidden state, but it is a privileged model. The skill score measures the gap between the agent and a model-optimal reference; if the agent lacks the model, part of that gap is estimation (posterior calibration), not control. The detection and knowing-doing metrics are too coarse to separate this, because they only ask whether the factor is named, not whether the posterior probability is accurate. The paper's internal consistency is high, and the trace excerpts (e.g., DeepSeek's flailing, Sonnet's freezing) suggest genuine action failures, but those are anecdotal and not a measured decomposition. A parameter-visible prompt arm is the direct test: if giving the LLM the same quantitative parameters as the oracle reduces the skill gap, then the 'control' claim is inflated. If it does not, the action-gap interpretation is robust. This test is cheap, uses the existing harness, and would settle the concern.","tokens_in":17450,"tokens_out":6365,"duration_ms":59074,"concrete_test":"Run a second prompt arm ('parameter-visible') in which the rules-only prompt is augmented with the exact quantitative parameter table from Appendix A—transition probabilities, onset/persistence rates, age caps, regime means, and the newsvendor critical ratio—while the observation stream and action API remain identical. Use the same four models on the same fifty seeds, one run per cell (or ideally 3 runs per cell to reduce noise). Compare overall skill scores and the floor-beating counts to the original arm. Decision rule: if the two below-floor models (DeepSeek-V4-Pro, Grok 4.5) improve by more than 0.2 in mean skill or cross from negative to positive on a substantial fraction of seeds, the original skill gap is partly attributable to missing quantitative model knowledge, so the 'failures concentrated in control' claim requires qualification. If skill scores do not move materially, the c","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—failures are concentrated in control, not state estimation—rests on the Bayes-filter oracle being a fair reference in the sense that its cost gap is attributable to acting, not to privileged knowledge. The paper discloses one asymmetry: the oracle's filters use the true generative parameters (transition tables P_i, regime means behind Q_i) and the rollout uses exact costs and lead times, while the LLM receives only a qualitative world description (Section 4, Appendix E). This asymmetry is not merely a philosophical caveat; it directly threatens the attribution. The belief-side metrics measure binary factor detection from rationale text, and the knowing-doing rate conditions on naming a factor. But the skill score—the quantitative basis for the 'control' claim—is computed against a reference that knows the exact probabilities of persistence, the exact regime means, and the exact cost structure. The LLM prompt gives exact costs (S4 sea, S15 air, S20 stockout) but not the transition probabilities, onset rates, persistence rates, age caps, or regime means (compare Appendix A vs Appendix E). A model that correctly names a port congestion may still under-order because it does not know the congestion persists with probability 0.85 per week and lasts up to 8 weeks; the oracle's rollout does. That is an epistemic gap about the world model, not a pure failure to act on a correct belief. The 'seeing' metrics are too coarse to capture this: a model can 'see' the factor and still lack the calibrated posterior that the oracle uses for control. Therefore part of the skill gap may be a knowledge gap about numbers, and the central claim 'failures are concentrated in control' is not unambiguously supported by the current experiment. The paper's own limitations note a clairvoyant ceiling to bound the oracle's optimality gap, but that does not address the parameter-knowledge confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces STOCKTAKE, a 26-week supply-chain replenishment benchmark built as a factored POMDP with six hidden factor processes. Each weekly decision requires the agent to read noisy dashboard symptoms, write a rationale, and place an order. The authors propose two reference policies: a symptom-blind base-stock floor and a 'fair oracle' consisting of six exact Bayes filters driving a rollout policy on the same observation stream the LLM receives. Skill is defined as the agent's cost improvement over the floor, normalized by the oracle's improvement. Belief-side metrics—detection lag and knowing-doing rate—are derived from an LLM grader that labels which hidden factors each rationale names. The paper reports that four frontier models detect 84–88% of stress episodes with sub-week lags, yet skill scores range from 0.62 to −0.23, with two models below the floor on roughly half their seeds. The central claim is that failures are concentrated in control, not state estimation, and that the 'fair oracle' makes this split measurable.","tokens_in":17773,"tokens_out":5706,"duration_ms":59455,"significance":"The benchmark construction is genuinely useful: the event-tape protocol fixes the hidden trajectory as a pure function of the seed, the Bayes-filter reference is computable, all cost and transition parameters are fully specified in Appendix A, and the stress-profile grouping (ISOLATED/PERSISTENT/COMPOUND) makes the interaction of hidden failures a controlled variable. The paper is also unusually candid in Section 6 about limitations such as single runs, grader audit scope, and the severity confound. However, the central inference—that the skill gap measures a control failure rather than a model-knowledge gap—is undermined by the disclosed asymmetry that the oracle uses the true generative parameters while the LLM receives only a qualitative description. Because the belief metrics only grade factor names and not posterior accuracy, the paper's own evidence does not cleanly separate 'knowing' from 'acting.' If the authors can supply a reference policy that does not use privileged quantitative parameters, or substantially narrow their claim, the benchmark and the empirical comparison would be a solid contribution to LLM-agent evaluation.","major_comments":[{"comment":"The central attribution—that skill shortfall equals a control failure—does not follow from the disclosed asymmetry. The oracle's filters use the true transition tables P_i and regime means behind Q_i (Appendix B), while the LLM prompt (Appendix E) gives exact costs but only qualitative descriptions of dynamics: no onset rates, persistence probabilities, age caps, or regime means. An agent can name a factor and still under-order because it does not know that port congestion persists with probability 0.85 or that a demand decline is sticky; under the paper's own POMDP framing, that is a state-estimation/model-knowledge gap, not a knowing-doing gap. The detection metrics only require the factor to be named, so they cannot rule out such a gap. The paper calls the oracle 'fair' and says shortfall is 'attributable to acting' (§1, §6), but by its own fairness criterion ('denied exactly the info","section":"§4, §6, Eq. (1)"},{"comment":"All belief-side metrics for the full fifty-seed, four-model sample depend on the gpt-5-mini grader, but the audits (a 30-row manual read and a second-grader agreement check on 100 sampled weeks) were performed only on the original twenty-seed grading run; the thirty expansion seeds and the later models were not re-audited (§6). Since detection and knowing-doing rates are central to the 'seeing is uniform' result, an un-audited grader on a large fraction of the episodes leaves open the possibility that label drift changes the reported detection and KD values. Please report audit statistics for the full dataset, or at minimum per-seed/model grader breakdowns; otherwise the perception-side uniformity claim is under-supported.","section":"§6, Appendix C"},{"comment":"The knowing-doing rate is presented as evidence that 'correct diagnosis coexists with empty shelves' (§4), but Table 4 shows that diagnosed stress weeks stock out more often than undiagnosed stress weeks for every model, and the text concedes this is a severity confound. With a confound of this size left unmodeled, the pooled KD values (0.24–0.43) cannot by themselves support the claim that agents knew and failed to act; they may simply reflect that severe weeks are both more noticeable and more costly. Please provide a severity-matched comparison (e.g., stockout rates on diagnosed vs. undiagnosed weeks within matched severity bands) or a causal decomposition. Without it, the knowing-doing claim rests on anecdotal trace readings (Appendix D) rather than on the metric named in Eq. (2).","section":"§5, Table 4"}],"minor_comments":[{"comment":"The phrase 'naming factors slightly faster than the two that beat it' is ambiguous; Table 3 reports mean detection lag in weeks, and the direction should be stated explicitly (shorter lag).","section":"Abstract"},{"comment":"The table note says the seed with negative headroom is excluded from skill means, but the 'beats floor' count includes that seed; the column headers and note should clarify which denominator applies to which statistic to avoid confusion.","section":"Table 2"},{"comment":"The oracle's 'small menu' is described as order quantities {0, four weeks, six weeks of expected demand}, but the exact expected-demand definition is not given; please specify how 'expected demand' is computed from the belief.","section":"Appendix B"},{"comment":"Eq. (1) is undefined when C_base = C_orc; the text excludes one such seed but does not say how many seeds are left in each group for Table 2. The group counts (11/15/23) are helpful, but a note that the excluded seed belongs to which group would improve transparency.","section":"§4"},{"comment":"The trace excerpts are compelling but are presented as anecdotal support; the sentence 'In every one of Claude Sonnet 5's nine below-floor seeds, its spending on air expedite alone exceeds its entire excess over the floor' is quantitative and should be moved into the main results with a footnote.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong benchmark core and the authors are transparent about limitations. The main issue is not lack of disclosure but that the disclosed asymmetry breaks the central inference: the skill score compares the LLM against a reference that knows quantitative model parameters the LLM never receives, so the 'control, not state estimation' claim is not yet established. The grader audit gap is also fixable. With a re-framed central claim or an additional parameter-free oracle variant, I would be supportive; as is, the paper overclaims its headline result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, STOCKTAKE is a genuinely new benchmark construction: an exact Bayes-filter rollout policy computed on the identical observation stream the LLM receives, anchored between a symptom-blind base-stock floor and the oracle mean, with belief-side metrics graded from weekly rationales. That is a real step forward in separating state estimation from control, and the implementation is careful — factored action-independent dynamics, exact per-factor filters, and an explicit disclosure of the one asymmetry. Second, the headline claim — failures are concentrated in control, not state estimation — is stronger than the current experiment supports. The disclosed asymmetry is load-bearing: the oracle's filters use the true transition probabilities, regime means, onset rates, persistence rates, and age caps, while the LLM prompt gives only qualitative descriptions plus exact costs. A model that names 'port congestion' in its rationale may still lack the calibrated posterior the oracle uses to decide whether to order through the congestion, because it does not know that congestion persists with probability 0.85 and caps at eight weeks. That is an epistemic gap about the world model, not purely a failure to act on a correct belief. The detection metrics are too coarse to separate 'knows the factor is present' from 'knows the posterior over its dynamics.' So part of the skill gap — possibly a large part — is a numbers gap, not an action gap.\n\nThe detection result itself is robust and interesting: all four models name 84–88% of stress episodes within half a week, and that finding does not depend on the oracle's parameter advantage. The skill-score divergence is striking, and the authors are unusually honest. They report that the grader audit covered only the original twenty-seed run, that later grading used the same configuration but was not freshly audited; they report single-run cost estimates and a severity confound on the knowing-doing rate; and a footnote discloses a previous grading-pipeline fault that was fixed. That is how benchmark papers should behave.\n\nThe remaining soft spots, in order: the parameter-knowledge confound is unquantified; the failure taxonomy for the two below-floor models rests on trace readings rather than a measured cost decomposition; and the seed pool is hand-filtered to favor separation, so below-floor rates are not population estimates. None of these are fatal. A bounded sensitivity analysis — giving the LLM the same parameter tables, or degrading the oracle's knowledge — would settle the central question. Without it, the paper is a strong benchmark contribution with an over-strong interpretation.\n\nThis is a paper for benchmark designers and anyone working on LLM agents in partially observable decision processes. It deserves a serious referee. My recommendation: send it to review, but require either the sensitivity analysis or a softened claim about control versus estimation before acceptance. The construction is worth publishing even if the strong conclusion does not survive.","headline":"A genuinely fair-oracle benchmark with a real, fixable confound: the skill score conflates missing world-model parameters with poor control, so the 'failures are in control, not perception' headline is not yet proven.","tokens_in":18332,"tokens_out":3628,"would_cite":true,"duration_ms":33166,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four frontier LLMs see hidden supply-chain stress almost perfectly and still fail to act on it.","keywords":["knowing-doing gap","LLM agents","POMDP","fair oracle","Bayes filter","supply-chain replenishment","state estimation","control"],"falsifier":"Give the LLM agents the same quantitative generative parameters the oracle uses (transition probabilities, regime means, and the full cost table). If skill scores rise toward 1, the control gap is substantially a knowledge gap about numbers; if the spread from 0.62 to −0.23 persists, the gap is genuinely in acting on correct beliefs.","tokens_in":17313,"feed_emoji":"📦","tokens_out":8219,"duration_ms":73097,"temperature":0.7,"pith_summary":"STOCKTAKE is a 26-week inventory-replenishment benchmark cast as a factored partially observable Markov decision process with six hidden factor processes, built so that a fair reference policy is computable. That reference — an exact Bayes filter per factor driving a rollout on the identical observation stream the LLM receives — lets the authors attribute the gap between an agent and the oracle to acting, not to privileged information. Across fifty seeds, four frontier LLMs all detect 84–88% of hidden stress episodes within about half a week, yet skill scores span 0.62 to −0.23, with two models ending below a symptom-blind base-stock policy on roughly half their seeds. The paper's claim: on this task, frontier-model failures are concentrated in control, not state estimation, and the knowing-doing gap runs in both directions — under-responding to persistent stress and over-responding with costly protective actions.","feed_headline":"Frontier LLMs detect hidden risks but fail to act on them","feed_subtitle":"A fair-oracle benchmark shows the bottleneck is control, not state estimation.","key_machinery":"The fair Bayes-filter oracle and the skill score. The oracle is one exact predict–correct filter per hidden factor, updated on the same weekly observation dictionary serialized into the LLM's prompt, then a rollout over about 200 sampled futures scores a menu of candidate actions (order quantities, routes, suppliers, and mitigation levers gated on posterior risk); it never reads hidden state and is executed through the same action API as the LLM. Because the hidden dynamics are factored and action-independent, the posterior splits into six small filters and the reference is computable. The skill score locates each run on a scale anchored by 0 (a symptom-blind base-stock policy) and 1 (the or","core_discovery":"On STOCKTAKE, detection is nearly uniform while costs diverge sharply: all four frontier LLMs name hidden-factor stress episodes 84–88% of the time, with mean detection lag under half a week, so the 'seeing' part of the task is close to solved. Yet skill scores — where 0 is a symptom-blind base-stock floor and 1 is the fair oracle's mean — range from 0.62 to −0.23, and two of four models finish below the floor on roughly half their seeds. The fair oracle is the load-bearing construction: exact Bayes filters per factor, conditioned on exactly the weekly observation dictionary the LLM sees, so shortfall cannot be blamed on withheld information (the one disclosed asymmetry is that the oracle us","pith_inferences":["A direct test of the disclosed asymmetry: give the LLM the same quantitative generative parameters (transition tables, regime means, exact costs) the oracle uses; if skill scores jump toward 1, part of the 'action gap' is really an epistemic gap about numbers, not purely control.","The construction generalizes beyond supply chains: any factored POMDP whose exact filter is computable can become a knowing-doing benchmark for other hidden-state decision problems, such as portfolio management or clinical logistics.","The severity confound in the knowing-doing rate suggests a matching protocol — pairing diagnosed and undiagnosed stress weeks on severity features — to estimate the causal effect of diagnosis on stockouts.","Each model–seed cell is a single run; repeating runs per cell would tighten the estimate of how much of the cross-model skill spread is systematic versus single-run noise."],"forward_implications":["If the knowing-doing gap dominates, interventions aimed at turning a correct diagnosis into a calibrated order — planning scaffolds, explicit contingency rules, action-menu support — should raise skill more than better inference tooling.","Benchmarks for partially observable LLM-agent tasks should report perception and control metrics separately, or they will misattribute failure causes.","A symptom-blind base-stock rule can beat a frontier LLM on a substantial share of tapes, a concrete deployment flag that default policies should sometimes be simple rules rather than capable models.","The knowing-doing rate is a severity-dependent diagnostic, not a model ranking; it should be read alongside severity-matched baselines."],"fun_headline_variants":["STOCKTAKE: LLMs detect 84-88% of risks yet two score below floor","Seeing vs doing: LLM agents detect risks, fail to act","Fair oracle shows LLM control lags perception","Frontier AI sees hidden failures but can't fix them","Knowing-doing gap measured in LLM agents with fair oracle"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The clean split between seeing and doing assumes the qualitative description in the prompt is enough for an LLM to reconstruct the exact quantitative cost structure; if it isn't, part of the measured 'action gap' is really a gap about exact numbers.","fun_headline_variants_meta":{"raw":{"variants":["STOCKTAKE: LLMs detect 84-88% of risks yet two score below floor","Seeing vs doing: LLM agents detect risks, fail to act","Fair oracle shows LLM control lags perception","Frontier AI sees hidden failures but can't fix them","Knowing-doing gap measured in LLM agents with fair oracle"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1480,"prompt_tokens":905,"completion_tokens":575,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":484}},"tokens_in":649,"tokens_out":575,"duration_ms":6803,"temperature":1.0,"reasoning_tokens":484,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T04:36:47.674811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the LLM agents the same quantitative generative parameters the oracle uses (transition probabilities, regime means, and the full cost table). If skill scores rise toward 1, the control gap is substantially a knowledge gap about numbers; if the spread from 0.62 to −0.23 persists, the gap is genuinely in acting on correct beliefs.","supporting_citations":[],"review_version":1}