{"id":"457f63e1-5c04-485d-948b-eaee7945fd76","arxiv_id":"2607.14169","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A synthesized game model can pass a 100%-accurate transition gate yet be systematically outplayed when the missed dynamics are rare under random play but pivotal under competent play.","lead":"This paper shows an AI-written game engine can pass a standard accuracy check with perfect scores and still lose at the game, because its mistakes sit exactly on rare, decisive rules. It provides a formula for when this blind spot appears and argues that planning systems should be tested by how they play, not by how accurately they predict sampled moves.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gate metric inconsistency: the material-at-cap rule is invisible to (s,a,s') tuples, yet Panel B's gate failure depends on seeing it; the exact danger-law factor needs the gate's actual scoring definition.","rationale":"I read the paper in good faith and the qualitative existence claim is well supported: Panel B shows an end-to-end synthesized incomplete CWM that loses at play with a seed-clustered CI excluding zero, and the Beacon witness is a clean formal counterexample on the inference axis. For that reason the reader's weakest assumption, the play-equivalence of the hand-written instrument to the synthesized CWM, does not block acceptance: the synthesized replication covers the existence direction, and the paper is appropriately careful that Panel A isolates a rule-specific causal magnitude while Panel B measures the full pipeline. The more load-bearing problem is internal rather than a matter of empirical transfer. The paper defines the gate as tuple-only transition accuracy, but the only difference between the complete and incomplete army5x5a instruments is the terminal return at the material-at-cap state. A tuple-only gate cannot see that difference, so Panel B's observation that gate-passing seeds are exactly the material-terminal-absent seeds is inexplicable under the stated metric. This inconsistency does not disprove the result, but it makes the central numerical claim and the exact (1-r)^N gate-miss factor uncheckable as written. The fix is concrete: inspect the evaluator, rerun the Panel B gate under both metric definitions, and restate the gate accordingly. If the gate actually scores returns or state-level contract outputs, the paper's headline should say 'contract-output accuracy' rather than 'transition accuracy,' and the state-accuracy figures should be recomputed consistently. This warrants a conditional acceptance pending that clarification, not a rejection.","tokens_in":36045,"tokens_out":15543,"duration_ms":172556,"concrete_test":"Inspect the gate evaluator and referee used by scripts/play_cost_synth_ci.py (and the refinement harness) and record exactly which contract fields are compared. Then re-run Panel B with the same pipeline but score the gate strictly on (s,a,s') tuples, ignoring returns and any winner/terminal-result field in the state dict. Count gate-passing seeds. If all 20 incomplete seeds pass, the observed gate failure is caused by an implicit returns check and the (1-r)^N law must be rescoped to a contract-output/state-accuracy gate. If many still fail, the material rule must be carried in the state dict; document that encoding and restate the gate definition. Independently recompute state accuracy on D_truth for gate-passing seeds under both metric definitions, since the '>=98% state-accurate' claim should be checked against whichever gate metric is actually used.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2.2 and Table 1 define the gate as 'transition accuracy = 1.0 on the sampled (state, action, next-state) tuples.' But the material-at-cap rule described in Section 3.3 changes only the terminal return: at the ply cap with both generals alive, the true game awards the win to the material leader, while the base game draws; legal actions, is_terminal, and apply_action need not differ. A strict tuple-only gate therefore cannot detect the rule, even when a material-at-cap terminal appears in the sample. Yet Panel B (Table 8 and footnote 1) reports that gate-passing coincides exactly with material-terminal absence (9/9) and that 0/10 material-terminal-present seeds reach gate 1.0. That is possible only if the gate implicitly scores returns, or if the rule is encoded in a non-semantic state-dict field such as a winner marker. The paper does not state which. This matters because Proposition 1's exact (1-r)^N gate-miss factor assumes that a play-through which triggers the rule reveals a discrepancy. Under the stated tuple-only metric, the miss probability is 1, not (1-r)^N; under a returns-inclusive or state-accuracy metric, the law can hold but the headline '100% transition accuracy' is really a contract-output/state gate claim. The central distinction between transition accuracy and play adequacy needs the gate definition corrected before the claimed exact law is evaluable. The synthesized-pipeline replication in Panel B is strong evidence that the qualitative gap exists; the issue is that the reported quantitative law and '100% transition accuracy' wording are not yet consistently defined.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether passing a random-trajectory transition-accuracy gate certifies that an LLM-synthesized Code World Model is adequate for planning. It argues no, and reports four main results: (1) a constructed army5x5a variant with a rare material-at-cap tiebreak where a rule-blind CWM passes a 100% transition-accuracy gate, is at least 98% state-accurate on the planner's search distribution, and yet loses at play with a measured play cost of 0.091 (seed-clustered CI [0.065,0.117]); (2) a quantitative danger law, danger = play_cost x (1 - rarity)^N, with the gate-miss factor proven exact; (3) a synthesis study showing that the omitted rule is translated when specified but not inferred from examples, across GPT-5.x models and several data regimes; and (4) an imperfect-information extension with a coverage bound for the inference gate, a hand-constructed Beacon witness that passes the inference gate yet loses every game, and two targeted checks that close this witnessed gap. The paper also includes an honest null result on small fully-specified games, explicit limitations, and a reproducible code release.","tokens_in":36311,"tokens_out":11414,"duration_ms":129551,"significance":"If the central claims hold, this is a significant and timely result for the CWM/LLM game-playing paradigm and for sampling-based verification of synthesized world models more broadly. The paper is unusually careful: headline play results are accompanied by pooled Wilson intervals and seed-clustered intervals; the Beacon play cost is proven exactly and checked mechanically; the authors report an honest null and explicitly separate instrumented from synthesized evidence; and they disclose and fix a feedback-channel confound. The danger law is a useful, falsifiable quantitative statement, and the distinction between verification-reach and deployment-reach distributions is conceptually important. The paper also ships reproducible scripts and records exact commands for the headline results.","major_comments":[{"comment":"The gate's scoring definition is internally inconsistent, and the exact danger law depends on which definition is meant. Section 2.2 and Table 1 define transition accuracy as agreement on (state, action, next-state) tuples and the gate criterion as transition accuracy = 1.0 on those tuples. The material-at-cap rule, however, changes only the terminal returns(state) vector (draw vs material-based win); it need not alter any (s,a,s') tuple or is_terminal. Under the stated tuple-only metric the gate could never detect a material-at-cap terminal, so the gate-miss probability would be 1, not (1-r)^N, and Proposition 1 would not apply. Yet Table 8 reports that 0/10 material-terminal-present seeds reach gate 1.0, and §5.3 shows refinement feedback printing returns: expected={1: 1.0, ...} got={1: 0.0, ...} on cap states with unequal material. This is only possible if the gate scores terminal ret","section":"§2.2/Table 1; §3.3/Table 8; §5.3"},{"comment":"The abstract and Contribution (1) state that the incomplete CWM is '≥98% state-accurate' on the planner's own search distribution, but the paper never reports a measured value of agree(D_cwm). Table 5 reports only gap_truth = 0.000, and the prose says the divergence region is 'less than 1% of visited states' without counts, seeds, or a precise definition of which states count (played trajectory vs MCTS search tree). Because the 98% number is one of the two accuracy metrics claimed to be blind to the omission, it needs a direct measurement with a confidence interval, or the claim should be replaced by a clearly derived bound if that is what supports it. This is a missing support for a headline empirical number.","section":"§3.3, Table 5"}],"minor_comments":[{"comment":"The danger-curve table uses a rounded play cost of 0.12, which is outside the reported seed-clustered 95% CI [0.065,0.117] for the headline play cost. The threshold shape is insensitive, as stated, but the numerical danger values should use the central estimate 0.091 or be presented as illustrative only.","section":"Table 9"},{"comment":"The proposition is marked 'self-proved (□)', but the symbol is not defined and the proof is a definitional argument. Consider attaching a short proof block or explicitly labeling it as a definitional observation rather than a theorem.","section":"§6.7, Proposition 5"},{"comment":"The x-axes are reversed in the two panels of Figure 4. The reversal is explained in the caption, but a reader may initially misread the direction of the threshold; consider adding an arrow or labeling 'rarer ->' on both panels.","section":"§4.2, Figure 4"},{"comment":"Footnote 1 is long and contains important caveats about seed 11. It would be easier to follow if the seed-11 exclusion and the one-directional coincidence were stated in the main text of Table 8's caption.","section":"§3.3, Panel B"}],"recommendation":"major_revision","confidential_remarks":"The gate-definition inconsistency is the main blocker. It is likely fixable by clarifying that the referee evaluates terminal returns/contract outputs and by renaming the metric accordingly, but as written it affects the exact law and the headline '100% transition accuracy' claim. The rest of the paper is unusually careful, and the qualitative verified-vs-correct gap is strongly supported by the end-to-end Panel B replication, the honest null, and the Beacon witness. If the authors resolve the metric issue and report the missing state-accuracy measurement, I would be willing to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely new and worth taking seriously: an LLM-synthesized world model can pass a random-sampling transition gate, look nearly perfect on the planner's own search distribution, and still lose systematically because the rare rule it misses is exactly the one competent play exploits. The paper demonstrates this cleanly with the material-at-cap instrument, and the separate instrumented (Panel A) and synthesized (Panel B) presentations are the right way to handle the causal claim without overstating the bridge. The honest null on small games, the DAgger/repair battery, and the explicit disclosure of the feedback-channel confound all suggest the authors are trying to do careful work, not just sell a story. The coverage bound and the Beacon witness are useful formal contributions, and the enumeration-free error-mass bound is a nice complement.\n\nThe soft spots are real, though. The stress-test note lands: the gate is defined as transition accuracy on (state, action, next-state) tuples, but the material-at-cap rule changes only the terminal returns. A strict tuple-only gate should be blind to that rule even when a material-terminal appears in the sample, which contradicts Panel B's reported 9/9 coincidence between gate-passing and material-terminal absence. Either the gate implicitly scores returns or state-level contract outputs, or the paper's own description of the mechanism is wrong. This ambiguity matters because Proposition 1's exact (1−r)^N factor depends on the gate observing a discrepancy when the rule fires. If the gate can't see the rule, the miss probability is 1, not (1−r)^N. The law may still hold under the intended scoring, but the paper needs to define precisely what the gate compares and how the terminal-state returns enter the accuracy calculation. This is fixable in revision, but it is load-bearing for the quantitative claim.\n\nThe other caveat — that the headline causal magnitude is measured with a hand-written instrument, with the bridge to synthesis assumed rather than proven — is acknowledged in the paper and mitigated by Panel B, so I'd treat it as a strength of framing rather than a fatal flaw. The \"round constant\" play cost of 0.12 used in the danger table is a minor wart; the threshold shape is insensitive to it, so it doesn't undermine the qualitative law.\n\nBottom line: the paper deserves a serious referee and a careful revision. The qualitative \"verify on the search distribution or play directly\" conclusion is well supported by the instrumented and end-to-end evidence; the exact law needs tightening. I'd send it to review with a request that the authors clarify the gate's scoring definition and rerun or re-express the relevant numbers under that clarified metric.","headline":"The verified-vs-correct gap is a real and important result, but the paper needs to clarify exactly what the gate scores before the quantitative law can be taken at face value.","tokens_in":36889,"tokens_out":2678,"would_cite":true,"duration_ms":34231,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Code World Model can pass a random-trajectory gate at 100% transition accuracy and still lose systematically at play, because the <1% error sits exactly where competent play goes.","keywords":["Code World Models","LLM synthesis","transition accuracy","sampling gate","rare rules","Monte Carlo Tree Search","imperfect-information games","belief inference"],"falsifier":"Run the full synthesis pipeline with many more gate-passing seeds and measure play cost: if the incomplete CWM's win rate approaches the fair baseline (≈0.495) instead of staying near 0.404, the verified-vs-correct gap does not transfer to actual synthesis. Alternatively, find one LLM synthesis that, given on-manifold transitions containing a rare rule, later encodes the rule and plays at parity—that would refute 'translation, not inference.' On the inference side, a game with a rare-but-pivotal belief error where a pure random gate still catches the error would falsify the coverage bound's pr","tokens_in":35843,"feed_emoji":"🎲","tokens_out":4293,"duration_ms":47105,"temperature":0.7,"pith_summary":"This paper argues that verifying an LLM-synthesized game world model by transition accuracy on random play-throughs is the wrong test for planning. It shows a model can score 100% on that gate, be 98% accurate on states the planner actually visits, and still be systematically outplayed, because the rare transition it gets wrong is the one that decides competent games. The paper quantifies the harm: danger = play cost × (1 − rarity)^N, with the gate-miss factor proven exact and the play cost empirically measured at 0.091 with separated 95% confidence intervals. It also shows that feeding more examples does not repair the gap: an omitted rule is statistically unidentifiable from a sample that misses it, and tested LLMs behave as rule translators rather than rule inferrers. The same mechanism is demonstrated for belief functions in imperfect-information games, where a hand-built witness passes every inference check yet loses every game.","feed_headline":"100% verified game model loses: rare rule costs 0.091 in play","feed_subtitle":"Sampling gates miss rare-but-pivotal rules; danger is play cost times miss probability. Check play, not prediction.","key_machinery":"The central mechanism is a reach-distribution shift between the verification policy (uniform-random play, which concentrates on shallow, common histories) and the deployment policy (Monte Carlo tree search, which concentrates on deep, strategically significant histories). This shift creates a rare-but-pivotal region that a random gate almost never samples and a planner reliably reaches. The quantitative carrier is the danger law danger = play cost × (1 − rarity)^N, whose gate-miss factor is proven exact by Bernoulli sampling and whose play-cost factor is bounded above by the planner's query-hit mass, certified below by measurement, and exact on the Beacon witness. On the inference side, a co","core_discovery":"The central discovery is that a Code World Model synthesized by an LLM can satisfy the standard acceptance test—transition accuracy 1.0 on a random trajectory sample—and even be ≥98% state-accurate on the distribution the planner actually visits, while being wrong in a rare-but-pivotal region that competent play reliably reaches. The paper calls this the verified-vs-correct gap, and measures its play cost at 0.091 (from a 0.404 win rate against a 0.495 fair baseline). The harm follows a quantitative law, danger = play cost × (1 − rarity)^N, where the gate-miss factor (1 − rarity)^N is proven exact under i.i.d. sampling and play cost is empirically bounded. The paper further proves that an om","pith_inferences":["A natural testable extension would apply the policy-guided mixture gate to other games beyond Beacon, predicting that a few reference-planner trajectories expose rare-region errors that pure random gates systematically miss.","The synthesis-residual channel—where the rule is present in the sample but the LLM still fails to encode it—means the danger law is a lower bound under an idealized synthesizer; real pipelines may suffer larger end-to-end harm than the law predicts, as the paper's synthesized replication suggests.","The reach-shift diagnosis is not specific to LLM-synthesized code: any learned world model validated on data from a different distribution than the deployment distribution could show the same verified-yet-inadequate pattern, including neural model-based RL and simulation-to-real pipelines.","The structural analogy to off-equilibrium-path beliefs points toward a design principle: verification should discipline the unconstrained region the way equilibrium refinements do in game theory, rather than relying purely on random sampling."],"forward_implications":["Adequacy for planning-oriented world models should be measured on the planner's search distribution or by direct play, not by transition accuracy on randomly sampled transitions.","Synthesis pipelines should include held-out play-throughs from a trusted reference planner; replacing one of 2000 random Beacon gates with one reference trajectory changed the verdict from accept to reject.","The specification must be complete before synthesis: example transitions do not reliably teach an omitted rule, and off-manifold examples actively corrupt the synthesized model.","The danger law gives a threshold test: a rule with random rarity r is safe only when (1 − r)^N is negligible; games with deep rare regions sit in the danger zone while shallow games are provably covered.","For imperfect-information games, belief functions must be gated separately from transitions, because the information partition is invisible to any transition accuracy gate."],"fun_headline_variants":["Verified yet wrong: 1% rare rule costs 0.091 in play","When 100% accuracy still loses: the verified-correct gap","Sampling can't see pivotal rules: play cost 0.091","Accuracy isn't adequacy: rare rules break verified models","Gate passes, play fails: danger = play cost × miss"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline 0.091 play cost rests on a hand-written rule-blind version of the game standing in for the LLM-synthesized incomplete model, and the paper states this bridge is empirical rather than proven—if the two are not play-equivalent outside the rare region, the instrument would not isolate the omitted-rule mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Verified yet wrong: 1% rare rule costs 0.091 in play","When 100% accuracy still loses: the verified-correct gap","Sampling can't see pivotal rules: play cost 0.091","Accuracy isn't adequacy: rare rules break verified models","Gate passes, play fails: danger = play cost × miss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1263,"prompt_tokens":953,"completion_tokens":310,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":697,"completion_tokens_details":{"reasoning_tokens":219}},"tokens_in":697,"tokens_out":310,"duration_ms":4609,"temperature":1.0,"reasoning_tokens":219,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T04:41:34.971019+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full synthesis pipeline with many more gate-passing seeds and measure play cost: if the incomplete CWM's win rate approaches the fair baseline (≈0.495) instead of staying near 0.404, the verified-vs-correct gap does not transfer to actual synthesis. Alternatively, find one LLM synthesis that, given on-manifold transitions containing a rare rule, later encodes the rule and plays at parity—that would refute 'translation, not inference.' On the inference side, a game with a rare-but-pivotal belief error where a pure random gate still catches the error would falsify the coverage bound's pr","supporting_citations":[],"review_version":1}