{"id":"c981e547-9b49-4ed1-9196-0ad1f7eecbb9","arxiv_id":"2608.11174","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VIScore, a product of veracity, influence, and sobriety factors, predicts planning success in latent world models better than existing single-component diagnostics.","lead":"This paper introduces VIScore, a diagnostic that combines rollout accuracy, action influence, and planner hallucination to score why latent world models succeed or fail at planning. On held-out models and datasets it correlates with planning success (Spearman over 0.75) more strongly than straightness, physical-state probing, or empowerment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 11's unverified horizon-consistency assumption for the noise floor could bias influence and undermine VIScore's cross-task comparability claims; direct test needed.","rationale":"The reader's weakest-assumption analysis identifies exactly the same load-bearing concern: Eq. 11's trace-matching of the one-step noise covariance to the H-step horizon assumes stable error eigendirections, and the paper itself labels this assumption unverified. I agree that this is the most critical soft spot in the argument. Influence is one of three multiplicative factors in VIScore and is specifically responsible for making the numerical scale comparable across tasks, so any task-dependent bias in \\(m_{\\text{emp}}\\) would directly corrupt the pooled Spearman correlation and calibration-error comparisons that form the central empirical claim. The concern is concrete and testable: one can empirically estimate the H-step residual covariance from the very rollouts already used for veracity and influence, compare its eigenspaces with the one-step covariance, and recompute the headline numbers with the empirical floor. The paper has notable strengths that prevent this concern from being disqualifying: a run-level train/test split that prevents leakage, cluster-bootstrap intervals over runs, robustness sweeps for the influence cap, and unusually honest limitations sections, including the explicit admission of the Eq. 11 assumption. The held-out results being better than development is also consistent with no leakage. However, because the bias from the assumed error-direction stability could be systematic and invisible in the reported experiments, the central claim should not be accepted as fully established until the direct test is run. The reader's CONDITIONAL verdict appropriately reflects this state; my stress-test does not move that verdict, so I mark the verdict as UNCHANGED while emphasizing that the proposed test is a necessary condition for upgrading to ACCEPT.","tokens_in":18793,"tokens_out":9345,"duration_ms":83284,"concrete_test":"Take a set of probe windows (e.g., K=64 anchors) and compute open-loop rollout terminal error vectors \\(e_i = \\text{rollout}(z_t, a_{t:t+H})_H - z_{t+H}\\) for H=5. Compute the empirical H-step residual covariance \\(\\hat{E}_H^{\\text{emp}} = \\operatorname{Cov}(e_i)\\). Compare its eigenspaces with the one-step \\(\\hat{E}\\): e.g., the Grassmann distance between top-k subspaces or the angle between principal eigenvectors. If the top eigenvectors rotate substantially (say subspace distance > 0.3), the trace-matching assumption is violated. Then recompute \\(m_{\\text{emp}}\\) and VIScore using \\(\\hat{E}_H^{\\text{emp}}\\) instead of the trace-matched \\(\\hat{E}_H\\), and re-run the held-out checkpoint pool evaluation (Table 6).","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 11 constructs the H-step noise floor as \\(\\hat{E}_H = (\\sigma_{\\text{roll}}^2 / \\operatorname{tr} \\hat{E}) \\hat{E}\\), rescaling the one-step teacher-forced residual covariance's magnitude to match the observed H-step rollout error while preserving its eigenbasis. This assumes the directions of predictor error are stable between one and H steps, with only the scale growing. The paper explicitly states (Appendix B.1.1): \"This makes an explicit assumption that we do not verify: that the directions in which the predictor errs are approximately stable between one step and H steps... If error directions rotate with the horizon, \\(\\hat{E}_H\\) misallocates the floor across eigendirections and \\(m_{\\text{emp}}\\) is biased.\" Influence = \\(\\min(m_{\\text{emp}}/\\tau, 1)\\) is one third of VIScore and is the component that makes the metric scale comparable across tasks. A task-dependent rotation of error directions would therefore bias \\(m_{\\text{emp}}\\) differently on different tasks, breaking the cross-task comparability that drives VIScore's pooled Spearman (+0.91 held-out) and calibration-error (7.0 vs 18.2) advantages. Because every checkpoint in every pool uses the same trace-matched \\(\\hat{E}_H\\), the bias could be systematic and undetectable in the reported experiments. This is the single most load-bearing assumption because it affects all pools and is directly admitted as unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper has two connected parts. First, it presents a controlled comparison of SIGReg and VISReg in the LeWorldModel latent planning stack, finding that VISReg's component-level flexibility does not improve planning on in-distribution tasks, while a closer match to the target isotropic Gaussian (obtained with larger batch sizes) improves out-of-domain PushObj performance. Second, it proposes VIScore, a product of veracity, influence, and sobriety, and evaluates this diagnostic against planning success rate using Spearman correlation and calibration error on run-level split development and held-out checkpoint pools, plus held-out methods, an unseen MAZE task, and six planner families. The central claim is that VIScore is the best available diagnostic of planning-relevant quality for predictor-based latent world models, with pooled correlations above 0.75 and calibration error below a constant reference in the Table 6 pools.","tokens_in":19057,"tokens_out":9392,"duration_ms":84391,"significance":"If the central claim is correct, VIScore is a practically useful diagnostic because it covers encoder, predictor, and planner rather than the latent alone. The paper's validation design is a genuine strength: checkpoints are split at the training-run level, all constants are fixed on a development pool before testing, cluster bootstraps over runs are used for intervals, held-out methods and an unseen task are evaluated under a frozen calibration map, and the failure modes (Cube's discrete mode, amortized planners, factor saturation) are explicitly scoped. The paper also reports computational cost and provides code. The main risk is not leakage or circularity but an unverified technical assumption in the influence estimator, plus overstatement of the planner-transfer and unseen-dataset evidence. These are fixable with additional analysis and more careful claims.","major_comments":[{"comment":"The construction of the H-step noise floor E_H = (sigma_roll^2 / tr E) E assumes that the eigendirections of the predictor error are stable between one-step teacher-forced rollouts and H-step open-loop rollouts, with only the magnitude changing. The manuscript explicitly states that this is an assumption it does not verify and that a violation would bias m_emp in a way the experiments cannot detect. Since influence is one third of VIScore and is the component that makes the score scale-comparable across tasks, an unverified bias in m_emp directly threatens the central pooled-correlation and calibration-error claims in Table 6. This is fixable within the scope of the paper: the authors have open-loop rollouts available for every checkpoint and can estimate the full H-step residual covariance directly instead of rescaling the one-step covariance by a scalar, or at least can compare the eigenbasis of the H-step residual covariance with that of E and report the overlap; a sensitivity analysis showing that pooled correlations are stable under random perturbations of E_H's eigenbasis would be the minimal version. As written, the central claim rests on an admitted unverified assumption.","section":"Appendix B.1.1, Eq. (11)"},{"comment":"The manuscript's claim that 'VIScore fits into any searching-based planner' and that VIScore 'consistently achieves the most stable and competitive results' overstates the evidence. For predictive sampling the pooled Spearman correlation is +0.51, and for single-start gradient descent it is +0.12 while raw empowerment reaches +0.82 in the same column. The appendix's explanation that task-selective planner weakness moves labels without moving scores is a reasonable post-hoc account, but it does not change the fact that the strong cross-task comparability claim fails for two of the six planner families shown. The abstract and Section 1 should either restrict the claim to the CEM-family and multi-start gradient planners, or present the planner-transfer result as a qualified extension with explicit failure modes.","section":"Section 4.3 and Appendix B.6, Table 10"},{"comment":"The claim that VIScore generalizes to 'both seen and unseen models and datasets' rests on a held-out method pool of 23 checkpoints and a held-out dataset pool of 20 checkpoints from only two independent MAZE runs. With two clusters, the cluster bootstrap has essentially no resolution, and the +0.75 held-out-method correlation is exactly at the stated threshold rather than above it. The paper should report bootstrap intervals or per-run correlations for these transfer pools, and the 'unseen dataset' evidence should be described as preliminary rather than as a fully established transfer result.","section":"Section 4.2, Table 6"}],"minor_comments":[{"comment":"The wording 'Spearman correlation over 0.75' should be reconciled with the held-out method pool value of +0.75, which is not strictly over 0.75 unless the unrounded value exceeds the threshold; please state the precision or rephrase to 'at least 0.75'.","section":"Abstract and Section 1"},{"comment":"The note describing calibration error is ambiguous about which pools use leave-one-task-out isotonic fits and which use the frozen development-pool map; a separate sentence for the held-out checkpoint pool versus the held-out methods and held-out dataset pools would remove confusion.","section":"Section 4.2, Table 6 note"},{"comment":"Several references in Table 2 are incomplete, such as 'FICReg (Seo et al.)' with no year and 'SD-JEPA (Thil et al., 2026)' with a truncated author list; please complete these citations.","section":"Section 3.2, Table 2"},{"comment":"The sentence about single-seed evaluation overconfidence cites the numbers '(1025/3014/5976)' without explaining what they are; please clarify whether these are episode counts or another quantity, and define them in the text.","section":"Section 3.3"},{"comment":"The expression for m_emp in Eq. (10) uses the form log det(I + E_H^{-1} S), while the derivation in Appendix B.1.1 uses the symmetric form log det(I + E_H^{-1/2} S E_H^{-1/2}); the paper should note that the two are equal under the standing assumptions or use a consistent notation.","section":"Section 4.1, Eq. (10)"},{"comment":"The main text says that 'three alternative monotone maps give rank-identical results' for veracity, but the product VIScore is not rank-invariant because it mixes factor values; the appendix correctly acknowledges this, so the main-text sentence should be qualified to avoid implying rank-invariance of the full score.","section":"Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within scope for a robotics/learning journal, and the experimental discipline is above average: run-level splits, cluster bootstraps, and frozen calibration maps are the right tools for this kind of claim. The main issue is that the abstract and Section 1 make stronger claims than the evidence in Appendix B.6 and the two-run MAZE pool supports, and the influence estimator in Eq. (11) is explicitly unverified at a load-bearing point. I would request the robustness analysis for Eq. (11), bootstrap intervals for the transfer pools, and a qualified statement of the planner-transfer and unseen-dataset results before publication. No concerns about citation practices beyond missing years in a few references."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VIScore is a real contribution: a three-part diagnostic (veracity, influence, sobriety) for latent world models that is the first to cover encoder, predictor, and planner together. The validation is more careful than usual for this subfield—run-level train/test split, cluster bootstrap over runs, held-out methods, tasks, and planners, and honest discussion of what is fitted and what is not. The SIGReg-VISReg comparison is also a useful negative result: self-supervised benefits do not automatically transfer to planning.\n\nThe central claim—VIScore correlates with planning success better than straightness, physical-state probing, or raw empowerment—holds up better than I expected. Raw empowerment is a strong within-task competitor, so the baseline is not weak; it fails on cross-task comparability, which is exactly where VIScore's task-scaled veracity and capped influence buy something. The held-out checkpoint pool is the right test, and the numbers there (+0.91 pooled, calibration 7.0 vs 18.2) are not what leakage would produce.\n\nThe soft spots are real but not disqualifying. The biggest is Eq. 11: the influence factor scales the one-step residual covariance to H steps by matching the trace, assuming error directions don't rotate with horizon. The paper explicitly says this is unverified. I agree with the stress-test note that this could bias m_emp in a task-dependent way and break cross-task comparability. Because influence saturates on many checkpoints, the bias may be muted in some pools, but on MAZE—where m_emp is far below the cap—it matters, and that's where the paper's claimed advantage relies on that factor. A direct test is straightforward: compute the eigenbasis of the one-step residual covariance and the H-step rollout error covariance on a handful of checkpoints and compare. Until that's done, the cross-task calibration numbers should be read with caution.\n\nAlso worth flagging: tau and the d_tol recipe are fitted on the development pool (the paper is transparent about this), Cube is excluded from pooled numbers for good reasons, and code and data are not shipped, which slows adoption and makes the fitted constants hard to audit. These are minor in proportion to the contribution.\n\nIf I worked on JEPA-style world models, I would use VIScore as a diagnostic and I would want the Eq. 11 test before trusting it as a standard. It deserves serious peer review and would benefit from revision.","headline":"A genuinely useful and carefully validated diagnostic for latent world model planning, but the unverified horizon-consistency assumption in the influence factor needs a direct test before I'd trust its cross-task numbers.","tokens_in":19617,"tokens_out":2455,"would_cite":true,"duration_ms":21605,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VIScore predicts planning success with a 0.91 Spearman correlation.","keywords":["VIScore","latent world models","planning diagnostics","veracity","influence","sobriety","JEPA","success rate correlation"],"falsifier":"Measure the full $H$-step residual covariance matrix directly from open-loop rollouts and compare its eigenbasis with the one-step teacher-forced residual covariance rescaled by trace; if the principal eigendirections rotate substantially on a task where VIScore currently correlates well, that task's VIScore–success association should weaken. Alternatively, recompute VIScore using the empirically measured $H$-step covariance instead of the trace-rescaled one-step covariance and check whether the pooled Spearman correlations and calibration errors change materially.","tokens_in":18562,"feed_emoji":"🎯","tokens_out":6974,"duration_ms":50826,"temperature":0.7,"pith_summary":"The paper proposes VIScore, a single diagnostic number for predictor-based latent world models that claims to measure the three components that decide planning success: whether open-loop rollouts stay within the task's success tolerance (veracity), whether actions can steer futures above the predictor's noise floor (influence), and whether search can hallucinate improvements over expert actions (sobriety). The authors argue that existing diagnostics—straightness, physical-state probing, and empowerment—look only at the encoded latent or at raw capacity, so they miss the predictor and planner that actually execute planning. Across held-out checkpoints, held-out training methods, an unseen task, and five alternative planners, VIScore is reported to achieve Spearman correlations with success rate above 0.75 and to be the only metric whose calibration error stays below a constant predictor in every tested setting. The stated aim is to give world-model developers a fast, interpretable diagnostic that says which component is failing, at about seven seconds per checkpoint on one GPU.","feed_headline":"VIScore predicts planning success with 0.91 Spearman correlation","feed_subtitle":"Veracity, influence, and sobriety beat straightness, probing, and empowerment across tasks and planners.","key_machinery":"The central object is the product identity that composes three bounded, task-scaled diagnostics: veracity $\\mathrm{erf}(d_{\\mathrm{tol}} / (2\\sqrt{2}\\,\\sigma_{\\mathrm{roll}}))$, influence $\\min(m_{\\mathrm{emp}}/\\tau, 1)$, and sobriety $1-\\hat{p}$. Veracity is a ratio, so it is invariant to latent rescaling; influence is capped empowerment with a shared cap $\\tau$ chosen by cross-validation, which lets one numerical scale mean the same thing across tasks; sobriety is a sign-based search-exploitation probe. The product forces all three conditions to hold simultaneously and is fixed before observing test pools, so it functions as a diagnostic rather than a fitted success predictor. A key technical assumption sits in the influence estimator: the one-step teacher-forced residual covariance is rescaled to horizon $H$ by matching trace, which presumes the error eigendirections do not rotate with horizon.","core_discovery":"VIScore is the product of three bounded factors, $\\mathrm{VIScore} = \\mathrm{erf}(d_{\\mathrm{tol}} / (2\\sqrt{2}\\,\\sigma_{\\mathrm{roll}})) \\cdot \\min(m_{\\mathrm{emp}}/\\tau, 1) \\cdot (1-\\hat{p})$. Veracity converts the ratio of the task's success tolerance to the RMS open-loop rollout error into a monotone map to $[0,1]$; influence takes latent empowerment—the channel capacity from action perturbations to terminal latent displacements—and clips it at a shared sufficiency threshold $\\tau$; sobriety measures the fraction of expert anchors for which a small internal search finds an imagined action with lower terminal cost than the recorded expert action. The paper claims that all three conditions must hold at once: a model can roll out accurately and have capacity yet still fail because the planner exploits localized predictor errors. Validation against planning success uses leave-one-task-out isotonic calibration and run-level splits, and reports pooled Spearman correlations of $+0.88$ on the development pool, $+0.91$ on held-out checkpoints, $+0.75$ on held-out methods, and $+0.87$ on an unseen task, with calibration error below the constant predictor in every pool.","pith_inferences":["If the three factors are genuinely the binding constraints, planning can be improved by targeted interventions: raise rollout fidelity when veracity is low, expand action-conditioned capacity when influence is below the cap, and regularize against search exploitation when sobriety is low.","The trace-matching assumption in the influence estimator is testable in isolation; if error directions rotate, a direction-aware horizon correction, such as a low-rank update of the noise floor, might preserve VIScore's cross-task scale.","The Cube outlier suggests a fourth factor—representation of discrete mode transitions—may be needed for tasks whose success hinges on a discontinuous event; extending VIScore with a mode-conditioned term is a natural next step.","A practical extension would be to use VIScore during training as an early-stopping or checkpoint-selection signal, since it costs far less than a full multi-seed planning evaluation."],"forward_implications":["Developers can run a roughly seven-second-per-checkpoint probe that reports which of the three components—reachability, capacity, or search-exploitable hallucination—is binding, instead of relying on training loss or latent-space statistics.","A shared influence cap makes VIScore numerically comparable across tasks, so a single threshold can rank checkpoints from different environments on the same scale, something raw empowerment cannot do.","Because VIScore transfers to unseen world-modeling methods and an unseen task with all constants frozen, it can be used to vet new architectures without retuning.","The SIGReg–VISReg intervention shows that self-supervised flexibility does not transfer to planning, while realized distribution-match accuracy does; this separates regularization flexibility from planning quality.","VIScore works across sampling-based and gradient-based searching planners when the planner is comparably effective on all tasks; task-selective planner weakness breaks cross-task comparability."],"supporting_citations":[{"why":"Supplies the LeWorldModel architecture and training setup whose checkpoints form the development and held-out pools.","marker":"(Maes et al., 2026)"},{"why":"Defines SIGReg, the regularization baseline replaced by VISReg, and the isotropic-Gaussian latent theory.","marker":"(Balestriero & LeCun, 2025)"},{"why":"Introduces VISReg, the regularizer used to test whether self-supervised flexibility transfers to planning.","marker":"(Wu et al., 2026)"},{"why":"Provides the straightness diagnostic that VIScore must beat.","marker":"(Wang et al., 2026)"},{"why":"Defines empowerment, which the influence factor reformulates into a capped sufficiency condition.","marker":"(Klyubin et al., 2005)"},{"why":"Supplies the calibration-error protocol used to compare diagnostic scores against a constant predictor.","marker":"(Zadrozny & Elkan, 2002)"},{"why":"Defines CEM, the default planner used in all main success-rate evaluations.","marker":"(Rubinstein, 1999)"}],"fun_headline_variants":["VIScore predicts planning success with 0.91 Spearman correlation","VIScore: 0.91 Spearman correlation to planning success","VIScore combines veracity, influence, sobriety; hits 0.91 Spearman","VIScore: veracity × influence × sobriety predicts planning success","VIScore: the only metric with calibration error below constant baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the directions in which the predictor makes errors stay roughly the same between one-step and $H$-step rollouts, so rescaling the one-step error covariance by matching its trace to the observed $H$-step rollout error gives an unbiased noise floor; if error directions rotate with the horizon, the influence factor misallocates the noise floor and VIScore's cross-task correlations and calibration would be biased.","fun_headline_variants_meta":{"raw":{"variants":["VIScore predicts planning success with 0.91 Spearman correlation","VIScore: 0.91 Spearman correlation to planning success","VIScore combines veracity, influence, sobriety; hits 0.91 Spearman","VIScore: veracity × influence × sobriety predicts planning success","VIScore: the only metric with calibration error below constant baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001451,"raw_usage":{"total_tokens":5926,"prompt_tokens":1114,"completion_tokens":4812,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":730,"completion_tokens_details":{"reasoning_tokens":4715}},"tokens_in":730,"tokens_out":4812,"duration_ms":32803,"temperature":1.0,"reasoning_tokens":4715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:44:20.069478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the full $H$-step residual covariance matrix directly from open-loop rollouts and compare its eigenbasis with the one-step teacher-forced residual covariance rescaled by trace; if the principal eigendirections rotate substantially on a task where VIScore currently correlates well, that task's VIScore–success association should weaken. Alternatively, recompute VIScore using the empirically measured $H$-step covariance instead of the trace-rescaled one-step covariance and check whether the pooled Spearman correlations and calibration errors change materially.","supporting_citations":[],"review_version":1}