Pith. sign in

REVIEW 4 minor 1 cited by

The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom

T0 review · 0 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two published protocols give the same LeWorldModel checkpoint 14% and 84%; only one reproduces the reported 87%.

desk verdict A genuinely careful reproduction: the protocol-dependence claim is solid and the paper is honest about its own limits, but the §5.3 generalization is weaker than it looks because of the context-length confound the authors themselves flag. read the letter →

arxiv 2608.10145 v1 pith:QNGEPJOC submitted 2026-08-10 cs.LG

Joyjeet Singh This is my paper · ORCID
classification cs.LG
keywords reproducibilityworldmodelplanningevaluationprotocolgoaloffsetbatchnormalizationpredictionerrorTwoRoom
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the reported ~87% planning result for LeWorldModel on TwoRoom is a product of the evaluation protocol and data pipeline, not a stable property of the model. On the authors' own released weights, the two protocols published with the paper give 14.0% and 84.0% of goals reached; only the repository configuration's goal offset of 25 frames and 50-step budget reproduces the reported figure, and changing the goal construction alone turns 84.0% into 8.0%. Reaching a converging model also required four conventions that appear nowhere in the released configuration: dense action gathering across a frameskip block, a programmatically set action-encoder width, ImageNet pixel normalisation, and action z-scoring. With those corrections, the reproduction reaches 94.0% at the repository offset and reproduces the positional probe (Pearson r = 0.9988). The paper also argues that one-step prediction error does not predict long-horizon planning success, and that a batch-normalisation artifact can inflate reported validation losses by up to 300 times.

What carries the argument

The load-bearing objects are the evaluation protocol and the data contract. Two protocol constants do the work: the goal offset (25 versus 100 frames) and the step budget (50 versus 150), together with the way the goal is constructed (an offset-defined future frame versus a recorded target). On the data side, the central identity is that displacement across a frameskip block equals speed times the summed actions of that block; under the sub-sampled action convention the supplied action was wrong by a median of 25.59 units against a typical block displacement of 13.3, which made the predictor's target unexplainable and explains the training plateau. The fourth mechanism is BatchNorm's stored running variance, of order $10^{-4}$ in the reproduced checkpoints, which amplifies stale statistics by a factor of 72 to 141 in evaluation mode and can inflate validation loss by up to 300 times.

What would settle it

Two experiments would settle the central claims. First, run the authors' released checkpoint on identical episodes under both published protocols: if the appendix reading (goal 100 frames ahead, 150-step budget) scores near 87% rather than 14%, the claim that only the repository configuration reproduces the reported figure fails. Second, train checkpoints matched on context length while varying prediction error: if the low-error one-frame-context model no longer plans best at offset 100, the prediction-accuracy dissociation collapses.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that the number one obtains from LeWorldModel on TwoRoom is set by how the goal is constructed and how actions and pixels are fed to the model. The authors' released checkpoint measures 84.0% under the evaluation configuration (goal 25 frames ahead, 50-step budget) and 14.0% under the appendix protocol (goal 100 frames ahead, 150-step budget); the reported ~87% reproduces only under the configuration's values. On fifty identical episodes, keeping weights, planner, and episodes fixed and changing only the goal definition moved the checkpoint from 84.0% to 8.0%, a matched-pair difference with p < $10^{-8}$. The appendix protocol cannot even be followed as written on the released dataset: episode lengths cap at 101 frames, so a goal 100 frames ahead admits exactly one legal start and selects the 6,056 episodes in which the data-collection policy timed out. The paper further reports that a more accurate one-step predictor can be a worse long-horizon planner, and that a BatchNorm layer with stale running statistics can make a converging training run look like a failure.

Load-bearing premise

The paper's claim that one-step prediction error does not predict long-horizon planning assumes the three checkpoints differ only in prediction accuracy; they also differ in context length (three frames versus one), so the dissociation is not uniquely attributable to prediction error.

Editorial extensions

If this is right

  • Planning numbers should always be reported with goal offset, step budget, and episode selection; without those, the same checkpoint can look like an 84% or a 14% method.
  • Held-out one-step prediction error is not a safe model-selection criterion for long-horizon planning; on this task it would have chosen the worst of three planners at the long horizon.
  • Checkpoints containing BatchNorm should have their running statistics verified against activation scale, because recalibration can change planning outcomes even when the reported loss moves by under 2%.
  • Reproduction should include differential testing against released weights and a physics-level invariant such as block displacement equals summed action, as a gate against silently broken pipelines.
  • The learned representation is not the bottleneck: position is linearly decodable at R^2 = 0.9977 even in runs where the predictor does not converge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the protocol sensitivity generalises beyond TwoRoom, published world-model comparisons that do not fix horizon, budget, and goal construction may be ranking evaluation setups rather than ranking models.
  • The most direct test the author could not afford is to train matched checkpoints that vary prediction accuracy while holding context length fixed; if a one-frame-context model with low prediction error no longer plans best at offset 100, the long-horizon dissociation is a context-length artifact rather than a prediction-accuracy effect.
  • The overshoot mechanism suggested in the paper, in which a sharper cost landscape pushes planners toward near-maximal actions under a terminal-cost objective, could be tested by measuring planned action magnitudes across checkpoints and could motivate cost-shaping or action-regularised planning.
  • The BatchNorm diagnostic, comparing each layer's running variance with activation scale and watching for a validation loss that oscillates while the training loss is flat, is cheap enough to become a standard checkpoint sanity check in world-model releases.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. This manuscript independently reproduces LeWorldModel on the TwoRoom environment. The authors reimplement the architecture from the released code and configuration files, train four checkpoints on rented GPUs, and evaluate on CPU. They report three main findings: (1) the position-probe result reproduces (Pearson r = 0.9988 versus a reported 0.996); (2) the planning result is protocol-dependent, with the authors' released checkpoint reaching 84.0% under the repository evaluation configuration (goal offset 25, step budget 50) and 14.0% under the protocol described in Appendix F.1 (offset 100, budget 150); and (3) the released configuration files alone are insufficient to produce a converging run unless four undocumented data-pipeline conventions are corrected: dense action gathering, programmatic action-encoder width, ImageNet pixel normalisation, and action z-scoring. Additional contributions include a demonstration that one-step prediction error does not order long-horizon planning success across three checkpoints, a batch-normalisation evaluation artifact that inflated validation losses, and a pre-registered mechanism-level effect that did not survive a change of checkpoint. The paper is explicit about its limitations, including single-seed training, the ten-epoch budget, the reimplementation rather than a rerun of the original training script, and the context-length confound in the planning-accuracy comparison.

Significance. If the central claim holds, this is a valuable reproducibility contribution: it shows that two published evaluation protocols in the released material disagree and that the choice determines whether the reported 87% figure is reproduced, with 84% versus 14% obtained on the authors' own released weights. The evidence for this claim is unusually strong: the evaluation environment is verified bit-level against the recorded data; the authors' released checkpoint is used as an independent reference; evaluations are deterministic and committed; a pre-registered experiment is reported in full including its failure; and a known defect in the evaluation reports is disclosed. These practices materially increase the credibility of the negative results. The secondary claim that one-step prediction accuracy fails to order long-horizon planning success is weaker because the three checkpoints differ in context length as well as prediction error; the authors disclose this confound, but it limits the breadth of the generalisation. Overall, the paper is a model of transparent negative reporting and should be of interest to the reproducibility and world-model communities.

minor comments (4)
  1. [Abstract and §4.2] The abstract's phrase 'changing nothing but how the goal is constructed' is not literally accurate: the two paired arms in §4.2 also differ in step budget (50 versus 150). The paper discloses this difference and correctly notes that it runs against the observed effect, but the abstract wording should be revised to 'changing the goal construction and the associated step budget,' and the same-budget comparison (84% versus 12% at budget 50) should be cited as the cleanest evidence that goal construction drives the effect.
  2. [Abstract and §5.3] The abstract and contribution list state that one-step prediction accuracy 'does not predict long-horizon planning success' without the caveat stated in §5.3 and §7, namely that the three checkpoints differ in context length (three-frame versus one-frame) as well as prediction error. Because the confound is acknowledged in the body, I am not treating this as a correctness error, but the abstract-level wording should be qualified so that readers do not take the dissociation as established independently of context length.
  3. [General] The manuscript would benefit from a light copy edit: notation is inconsistent between 'R2', 'R^2', and 'r'; '~87%' and 'approximately 87%' are used interchangeably; and some table cells in Table 1 mix status values ('match', 'deviation', 'undocumented — corrected', 'conflict') without a single concise legend. These are presentation issues only.
  4. [Title and §1] The title's broad claim that 'the evaluation protocol determines the result' is supported for TwoRoom only; the subtitle already narrows the scope, but the first sentence of the abstract could state explicitly that the protocol-dependence claim is demonstrated on this one environment and at this model scale, which is the scope the authors themselves define in §1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the protocol-dependence claim rests on differential testing of the authors' released weights, the four undocumented conventions are sourced from code and artifact checks rather than fitted to the target, and the disclosed §5.3 confound is a validity limitation, not an input-output identity.

full rationale

The paper's central claim is that the evaluation protocol determines the reported result. This is supported by running the original authors' released checkpoint through an independent harness under two published protocols, yielding 84.0% versus 14.0%, and by a matched-pair goal-construction change on identical episodes (84.0% to 8.0%, p < 1e-8). No parameter of this paper is fitted to reproduce the 87% figure, and the released checkpoint is used as an external reference artifact, not as an input to a derivation. The four undocumented conventions (dense action gathering, programmatic action-encoder width, ImageNet normalisation, action z-scoring) were identified by reading the reference source code and corroborated by properties of the released checkpoint, such as the ten-channel action encoder and a twelvefold one-step error difference; they are not chosen to match the reported planning number. The training-convergence claim is explicitly scoped to the paper's reimplementation, with a full deviation table and the pre-registered arm reported even when it failed. The one candidate concern, §5.3, is a three-checkpoint comparison in which one-step prediction error is confounded with context length; the paper itself states this confound and qualifies the conclusion accordingly. A confound is a limitation on inference, not a circular step in which a prediction is equivalent to its inputs by construction. Self-citations appear only as pointers to the paper's own repository, pre-registration, and gate outputs, and none is load-bearing for the main claims. The derivation chain is therefore self-contained against external benchmarks and exhibits no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions about the released artifact's representativeness and about the interpretability of three-checkpoint comparisons; none of these are fitted constants, and the paper verifies the environment and protocol claims where possible. The main unverified premise is the context-length confound in the planning generalisation, which the authors flag themselves.

assumptions (4)
  • domain assumption The released dataset and environment match the original training and evaluation conditions.
    The paper verifies pixel MAE 0.00 and replay error 0.000 (§3.3), but the equivalence of the released data to the original training distribution is ultimately an assumption because the original's data-collection details are not fully published.
  • domain assumption The repository evaluation configuration (goal offset 25, step budget 50) is a valid representative of the protocol used for the reported 87%.
    The paper shows only this protocol reproduces the reported figure on the authors' own weights (§4.2), but the authors did not confirm which protocol they used (§6.4).
  • domain assumption The three checkpoints compared for the planning dissociation can be attributed to prediction accuracy.
    The paper itself flags the confound: the two overshooting checkpoints use a three-frame context and the cautious one uses a single frame; three checkpoints cannot separate prediction accuracy from context length (§5.3, §7).
  • domain assumption The authors' released checkpoint is unaffected by the batch-normalisation eval-mode artifact.
    The paper measures the authors' evaluation-to-training gap at 1.09x and its projector running variance at 0.0172 versus 1e-4 in its own checkpoints (§4.3), but this is a claim about the authors' training state that cannot be fully verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom." pith.science (2026). https://pith.science/paper/QNGEPJOC

@misc{pith2026260810145,
  author       = {Pith},
  title        = {Pith review of: The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNGEPJOC}},
  note         = {Machine review of arXiv:2608.10145}
}
abstract

LeWorldModel trains a latent world model with a prediction loss and a single anti-collapse regulariser, and reports approximately 87% of goals reached on TwoRoom, its simplest diagnostic environment. We reproduce that result by independent reimplementation on roughly $25 of rented compute, with all evaluation on one laptop CPU. We reach 94.0% at the repository's evaluation goal offset, against 84.0% for the authors' own released checkpoint measured under our protocol on identical episodes, and we reproduce the reported representation result directly (position probe Pearson r = 0.9988 against a reported 0.996). Reaching that point required four conventions that determine the outcome and appear in no released configuration file: dense action gathering across a frameskip block, a programmatically-set action-encoder width, ImageNet pixel normalisation, and action z-scoring. A reproducer following the released configurations alone obtains a model whose predictor cannot converge. The evaluation protocol is itself contested by the released material. The paper's appendix and the repository's configuration specify different goal offsets and step budgets; on the authors' own weights these yield 14.0% and 84.0%, and only the configuration's values reproduce the reported figure. On fifty identical episodes, changing nothing but how the goal is constructed moves that checkpoint from 84.0% to 8.0%. Two findings generalise. One-step prediction accuracy does not predict long-horizon planning success: across three checkpoints spanning a sevenfold range in prediction error, including the authors' own, it orders short-horizon success monotonically and fails to order long-horizon success at all. And a batch normalisation layer inflated our reported validation loss by up to a factor of 300, concealing a training loss that was flat throughout.

Figures

Figures reproduced from arXiv: 2608.10145 by the authors.

Figure 1
Figure 1. The representation carries what the predictor needs. (a) [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Prediction accuracy orders short-horizon planning success and fails to order long [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Objective Is the Bottleneck: Latent World Models Encode What Their Planners Cannot Use

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Squared latent distance, the objective a latent-world-model planner minimizes, saturates and then inverts with true distance; replacing it with a reachability cost fixes long-horizon planning without retraining.

Reference graph

Works this paper leans on

6 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2511.08544 , eprint =

    LeJEPA: Provable and Scalable Self-Supervised Learning without the Heuristics , author =. arXiv preprint arXiv:2511.08544 , eprint =

    arXiv

  2. [2]

    Beyond Euclidean Proximity: Repairing Latent World Models with Horizon-Matched Trajectory Reachability Metrics

    Beyond Euclidean Proximity: Repairing Latent World Models with Horizon-Matched Trajectory Reachability Metrics , author =. arXiv preprint arXiv:2605.22164 , eprint =

    work page Pith review arXiv

  3. [3]

    arXiv preprint arXiv:2603.19312 , eprint =

    LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels , author =. arXiv preprint arXiv:2603.19312 , eprint =

    arXiv

  4. [4]

    arXiv preprint arXiv:2602.08968 , eprint =

    stable-worldmodel-v1: Reproducible World Modeling Research and Evaluation , author =. arXiv preprint arXiv:2602.08968 , eprint =

    arXiv

  5. [5]

    Maes, Lucas , howpublished =. The

    work page

  6. [6]

    tinylab: reimplementation, checkpoints, evaluation reports, fidelity audit, pre-registration and gate outputs , author =

    work page

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.