{"id":"57c0347f-7c04-4de8-ae83-39c1f4abeaff","arxiv_id":"2608.10145","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The reported LeWorldModel TwoRoom result is protocol-dependent: the released configuration's goal offset reproduces it on the authors' own weights, the appendix's does not, and four undocumented pipeline conventions are required to make the released configuration train a converging predictor.","lead":"An independent reproduction of LeWorldModel finds that the reported 87% success rate on TwoRoom is reproduced only under one of two published evaluation protocols, and only after four undocumented data-convention fixes. The same reproduction finds that one-step prediction accuracy does not predict long-horizon planning success, and that a batch-normalisation artifact can inflate validation loss by up to 300x.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No fatal objection: protocol-dependence claim is solid; the one real gap is the §5.3 context-length confound, which the paper itself discloses.","rationale":"The reader's ACCEPT verdict is well calibrated. The paper's central claim is narrow, concrete, and backed by differential testing on the authors' own released weights: the evaluation protocol determines the reported result. The evidence includes a matched-pair comparison on identical episodes (84.0% vs 8.0%, p < 1e-8) and the fact that only the repository configuration reproduces the reported figure. That claim is independent of the paper's own training-script choices, and it holds regardless of the context-length confound. The paper also supports its claims with committed evaluation reports, released checkpoints, source-level citations, and stated limitations.\n\nThe single most load-bearing concern is the paper's secondary generalisation in §5.3: one-step prediction accuracy does not predict long-horizon planning success. The three checkpoints compared differ in a second variable, context length, which the paper itself flags as a confound. The two more accurate checkpoints use three-frame context; the least accurate uses one-frame. Since the reversal at offset 100 could be caused by context length rather than prediction error, the paper's most transferable claim is not fully identified. This concern does not change the verdict because the paper names the confound explicitly and the central protocol-dependence claim does not depend on it. The recommended action is unchanged: ACCEPT, with the caveat that §5.3 should be read as a two-variable observation requiring matched checkpoints before it can be attributed to prediction accuracy alone.","tokens_in":23489,"tokens_out":2927,"duration_ms":22709,"concrete_test":"Train two additional checkpoints at the corrected pipeline settings with a one-frame context and a three-frame context, holding all else fixed, and evaluate both at goal offset 100 under the 150-step budget. If the three-frame checkpoint overshoots and the one-frame checkpoint plans cautiously, the §5.3 reversal is a context-length effect rather than a prediction-accuracy effect; if both checkpoints order by one-step error, the dissociation is independent of context length. Alternatively, evaluate the existing three checkpoints with a one-frame context by truncating their context inputs at planning time, which is cheaper but weaker because the checkpoints were trained with different contexts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that the evaluation protocol determines the reported result — is well supported by differential testing on the authors' own released weights. Under the repository configuration the released checkpoint reaches 84.0%, under the appendix protocol 14.0%, and changing only the goal construction on identical episodes moves the same checkpoint from 84.0% to 8.0% (matched-pair p < 1e-8). This evidence does not depend on the paper's own training runs, so the main claim is secure.\n\nThe load-bearing concern is the paper's secondary generalisation in §5.3: one-step prediction accuracy does not predict long-horizon planning success. The three checkpoints compared at offset 100 differ in prediction error (0.829, 0.410, 0.116), but also in context length: the two more accurate checkpoints use a three-frame context and the least accurate uses a one-frame context. The reversal at the longer horizon could be a context-length effect rather than a prediction-accuracy effect. The paper states this confound explicitly in §5.3 and §7, so the disagreement is with the strength of the generalisation, not with the paper's honesty. The confound also limits the practical implication: selecting a world model by one-step prediction error is not reliably shown to be worse than chance at long horizons by this comparison alone.\n\nA smaller gap is that the 'released configuration is insufficient' claim is based on a reimplementation rather than a rerun of the authors' training script. The paper acknowledges this (§2, §7) and the four undocumented conventions are corroborated by the released checkpoint's parameter count and prediction error, so this is a stated limitation rather than a hidden flaw.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript independently reproduces LeWorldModel on the TwoRoom environment. The authors reimplement the architecture from the released code and configuration files, train four checkpoints on rented GPUs, and evaluate on CPU. They report three main findings: (1) the position-probe result reproduces (Pearson r = 0.9988 versus a reported 0.996); (2) the planning result is protocol-dependent, with the authors' released checkpoint reaching 84.0% under the repository evaluation configuration (goal offset 25, step budget 50) and 14.0% under the protocol described in Appendix F.1 (offset 100, budget 150); and (3) the released configuration files alone are insufficient to produce a converging run unless four undocumented data-pipeline conventions are corrected: dense action gathering, programmatic action-encoder width, ImageNet pixel normalisation, and action z-scoring. Additional contributions include a demonstration that one-step prediction error does not order long-horizon planning success across three checkpoints, a batch-normalisation evaluation artifact that inflated validation losses, and a pre-registered mechanism-level effect that did not survive a change of checkpoint. The paper is explicit about its limitations, including single-seed training, the ten-epoch budget, the reimplementation rather than a rerun of the original training script, and the context-length confound in the planning-accuracy comparison.","tokens_in":23884,"tokens_out":8020,"duration_ms":82634,"significance":"If the central claim holds, this is a valuable reproducibility contribution: it shows that two published evaluation protocols in the released material disagree and that the choice determines whether the reported 87% figure is reproduced, with 84% versus 14% obtained on the authors' own released weights. The evidence for this claim is unusually strong: the evaluation environment is verified bit-level against the recorded data; the authors' released checkpoint is used as an independent reference; evaluations are deterministic and committed; a pre-registered experiment is reported in full including its failure; and a known defect in the evaluation reports is disclosed. These practices materially increase the credibility of the negative results. The secondary claim that one-step prediction accuracy fails to order long-horizon planning success is weaker because the three checkpoints differ in context length as well as prediction error; the authors disclose this confound, but it limits the breadth of the generalisation. Overall, the paper is a model of transparent negative reporting and should be of interest to the reproducibility and world-model communities.","major_comments":[],"minor_comments":[{"comment":"The abstract's phrase 'changing nothing but how the goal is constructed' is not literally accurate: the two paired arms in §4.2 also differ in step budget (50 versus 150). The paper discloses this difference and correctly notes that it runs against the observed effect, but the abstract wording should be revised to 'changing the goal construction and the associated step budget,' and the same-budget comparison (84% versus 12% at budget 50) should be cited as the cleanest evidence that goal construction drives the effect.","section":"Abstract and §4.2"},{"comment":"The abstract and contribution list state that one-step prediction accuracy 'does not predict long-horizon planning success' without the caveat stated in §5.3 and §7, namely that the three checkpoints differ in context length (three-frame versus one-frame) as well as prediction error. Because the confound is acknowledged in the body, I am not treating this as a correctness error, but the abstract-level wording should be qualified so that readers do not take the dissociation as established independently of context length.","section":"Abstract and §5.3"},{"comment":"The manuscript would benefit from a light copy edit: notation is inconsistent between 'R2', 'R^2', and 'r'; '~87%' and 'approximately 87%' are used interchangeably; and some table cells in Table 1 mix status values ('match', 'deviation', 'undocumented — corrected', 'conflict') without a single concise legend. These are presentation issues only.","section":"General"},{"comment":"The title's broad claim that 'the evaluation protocol determines the result' is supported for TwoRoom only; the subtitle already narrows the scope, but the first sentence of the abstract could state explicitly that the protocol-dependence claim is demonstrated on this one environment and at this model scale, which is the scope the authors themselves define in §1.","section":"Title and §1"}],"recommendation":"minor_revision","confidential_remarks":"To the editor: this is a strong and unusually honest reproduction paper. The central protocol-dependence result is solid, and the authors' practice of committing evaluation reports, pre-registering a confirmatory experiment, and disclosing a defect in their own reports sets a high standard. My reservation is limited to the wording of two abstract-level claims that outrun the evidence the paper itself provides: the 'changing nothing but the goal construction' phrase and the prediction-accuracy generalisation. Both are local and can be fixed by rewording; no new experiments are necessary. I support publication after a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the central finding holds: the released material publishes two disagreeing evaluation protocols, and on the authors' own released weights only the repository configuration's values (goal offset 25, budget 50) reproduce the reported ~87%. The evidence for this is differential testing on the released checkpoint, with identical episodes and committed reports. That is exactly the right way to validate an evaluation harness, and it makes the protocol-dependence claim secure. Second, the flashier §5.3 claim — that one-step prediction error does not predict long-horizon planning — is real but softer than the abstract suggests. The three checkpoints differ in context length as well as prediction error, so the reversal at offset 100 could be a context-length effect. The paper says this plainly in §5.3 and §7, so the weakness is in the strength of the generalization, not in the honesty.\n\nWhat is actually new: the four undocumented pipeline conventions (dense action gathering, programmatic action-encoder width, ImageNet normalization, action z-scoring) that determine whether the released configuration converges; the matched-pair result where changing only goal construction moves the authors' checkpoint from 84% to 8% (p < 1e-8); the BatchNorm eval-mode artifact with its mechanism and cheap check; and a pre-registered mechanism-level effect that died on a different checkpoint, reported in full with all three arms. That last one is rare and valuable. The paper ships code, checkpoints, evaluation reports, a fidelity audit against source, and even discloses a defect in its own committed reports. That is the behavior we want from reproduction work.\n\nSoft spots, in proportion. The context confound is the main one: the practical advice \"don't select world models by one-step error\" rests on three checkpoints, not a controlled sweep, and the paper's own qualifications should stay attached to any citation of that result. The claim that the released configuration is insufficient is based on a reimplementation rather than a rerun of the authors' training script; the paper acknowledges this, and the parameter count and prediction-error checks give it independent support, so I treat it as minor. Single seed and unpublished episode selection limit the like-for-like comparison to 94% vs 84% on identical episodes, not vs the reported 87%; both limits are stated explicitly.\n\nThis paper is for anyone doing world-model evaluation, reproducibility work, or checkpoint selection by loss. It deserves a serious referee. The main claim holds, the secondary claim is honestly caveated, and the artifacts are there for others to check. I would send it out.","headline":"A genuinely careful reproduction: the protocol-dependence claim is solid and the paper is honest about its own limits, but the §5.3 generalization is weaker than it looks because of the context-length confound the authors themselves flag.","tokens_in":24273,"tokens_out":1894,"would_cite":true,"duration_ms":20568,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two published protocols give the same LeWorldModel checkpoint 14% and 84%; only one reproduces the reported 87%.","keywords":["reproducibility","world model","planning","evaluation protocol","goal offset","batch normalization","prediction error","TwoRoom"],"falsifier":"Two experiments would settle the central claims. First, run the authors' released checkpoint on identical episodes under both published protocols: if the appendix reading (goal 100 frames ahead, 150-step budget) scores near 87% rather than 14%, the claim that only the repository configuration reproduces the reported figure fails. Second, train checkpoints matched on context length while varying prediction error: if the low-error one-frame-context model no longer plans best at offset 100, the prediction-accuracy dissociation collapses.","tokens_in":23294,"feed_emoji":"🧪","tokens_out":8530,"duration_ms":80403,"temperature":0.7,"pith_summary":"This paper tries to establish that the reported ~87% planning result for LeWorldModel on TwoRoom is a product of the evaluation protocol and data pipeline, not a stable property of the model. On the authors' own released weights, the two protocols published with the paper give 14.0% and 84.0% of goals reached; only the repository configuration's goal offset of 25 frames and 50-step budget reproduces the reported figure, and changing the goal construction alone turns 84.0% into 8.0%. Reaching a converging model also required four conventions that appear nowhere in the released configuration: dense action gathering across a frameskip block, a programmatically set action-encoder width, ImageNet pixel normalisation, and action z-scoring. With those corrections, the reproduction reaches 94.0% at the repository offset and reproduces the positional probe (Pearson r = 0.9988). The paper also argues that one-step prediction error does not predict long-horizon planning success, and that a batch-normalisation artifact can inflate reported validation losses by up to 300 times.","feed_headline":"One protocol change drops LeWorldModel from 84% to 14%","feed_subtitle":"The authors' own weights score 84% under one published protocol and 14% under the other; only one reproduces 87%.","key_machinery":"The load-bearing objects are the evaluation protocol and the data contract. Two protocol constants do the work: the goal offset (25 versus 100 frames) and the step budget (50 versus 150), together with the way the goal is constructed (an offset-defined future frame versus a recorded target). On the data side, the central identity is that displacement across a frameskip block equals speed times the summed actions of that block; under the sub-sampled action convention the supplied action was wrong by a median of 25.59 units against a typical block displacement of 13.3, which made the predictor's target unexplainable and explains the training plateau. The fourth mechanism is BatchNorm's stored running variance, of order $10^{-4}$ in the reproduced checkpoints, which amplifies stale statistics by a factor of 72 to 141 in evaluation mode and can inflate validation loss by up to 300 times.","core_discovery":"The central discovery, stated on the paper's own terms, is that the number one obtains from LeWorldModel on TwoRoom is set by how the goal is constructed and how actions and pixels are fed to the model. The authors' released checkpoint measures 84.0% under the evaluation configuration (goal 25 frames ahead, 50-step budget) and 14.0% under the appendix protocol (goal 100 frames ahead, 150-step budget); the reported ~87% reproduces only under the configuration's values. On fifty identical episodes, keeping weights, planner, and episodes fixed and changing only the goal definition moved the checkpoint from 84.0% to 8.0%, a matched-pair difference with p < $10^{-8}$. The appendix protocol cannot even be followed as written on the released dataset: episode lengths cap at 101 frames, so a goal 100 frames ahead admits exactly one legal start and selects the 6,056 episodes in which the data-collection policy timed out. The paper further reports that a more accurate one-step predictor can be a worse long-horizon planner, and that a BatchNorm layer with stale running statistics can make a converging training run look like a failure.","pith_inferences":["If the protocol sensitivity generalises beyond TwoRoom, published world-model comparisons that do not fix horizon, budget, and goal construction may be ranking evaluation setups rather than ranking models.","The most direct test the author could not afford is to train matched checkpoints that vary prediction accuracy while holding context length fixed; if a one-frame-context model with low prediction error no longer plans best at offset 100, the long-horizon dissociation is a context-length artifact rather than a prediction-accuracy effect.","The overshoot mechanism suggested in the paper, in which a sharper cost landscape pushes planners toward near-maximal actions under a terminal-cost objective, could be tested by measuring planned action magnitudes across checkpoints and could motivate cost-shaping or action-regularised planning.","The BatchNorm diagnostic, comparing each layer's running variance with activation scale and watching for a validation loss that oscillates while the training loss is flat, is cheap enough to become a standard checkpoint sanity check in world-model releases."],"forward_implications":["Planning numbers should always be reported with goal offset, step budget, and episode selection; without those, the same checkpoint can look like an 84% or a 14% method.","Held-out one-step prediction error is not a safe model-selection criterion for long-horizon planning; on this task it would have chosen the worst of three planners at the long horizon.","Checkpoints containing BatchNorm should have their running statistics verified against activation scale, because recalibration can change planning outcomes even when the reported loss moves by under 2%.","Reproduction should include differential testing against released weights and a physics-level invariant such as block displacement equals summed action, as a gate against silently broken pipelines.","The learned representation is not the bottleneck: position is linearly decodable at R^2 = 0.9977 even in runs where the predictor does not converge."],"supporting_citations":[{"why":"Supplies the original reported ~87% planning figure, the position probe claim (Pearson ~0.996), and the appendix protocol (goal offset 100, budget 150) that the reproduction finds non-reproducing.","marker":"Maes et al. 2026b"},{"why":"Supplies the TwoRoom environment and the data buffer whose dense action-gathering convention is one of the four undocumented corrections that determine convergence.","marker":"Maes et al. 2026a"},{"why":"Code release whose train.py and utils.py are the source of the programmatic action-encoder width, ImageNet pixel normalisation, and action z-scoring.","marker":"Maes 2026"},{"why":"Published critique of Euclidean-distance planning whose reachability prediction is tested by the pre-registered matched-pair experiment that did not survive a change of checkpoint.","marker":"Li et al. 2026"}],"fun_headline_variants":["LeWorldModel score swings 84% to 14% by evaluation protocol alone","Goal tweak flips LeWorldModel from 84% to 8%","LeWorldModel's 87% result hinges on unlisted evaluation settings","Prediction accuracy doesn't predict planning: LeWorldModel case"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's claim that one-step prediction error does not predict long-horizon planning assumes the three checkpoints differ only in prediction accuracy; they also differ in context length (three frames versus one), so the dissociation is not uniquely attributable to prediction error.","fun_headline_variants_meta":{"raw":{"variants":["LeWorldModel score swings 84% to 14% by evaluation protocol alone","Goal tweak flips LeWorldModel from 84% to 8%","LeWorldModel's 87% result hinges on unlisted evaluation settings","Prediction accuracy doesn't predict planning: LeWorldModel case"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1771,"prompt_tokens":1141,"completion_tokens":630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":757,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":757,"tokens_out":630,"duration_ms":5555,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:27.895195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Two experiments would settle the central claims. First, run the authors' released checkpoint on identical episodes under both published protocols: if the appendix reading (goal 100 frames ahead, 150-step budget) scores near 87% rather than 14%, the claim that only the repository configuration reproduces the reported figure fails. Second, train checkpoints matched on context length while varying prediction error: if the low-error one-frame-context model no longer plans best at offset 100, the prediction-accuracy dissociation collapses.","supporting_citations":[],"review_version":1}