Pith.

Computer Vision and Pattern Recognition · arXiv:2610.31668 v2 · submitted 22 Oct 2026

Reading, Writing, Sharing, and Interacting with the World through Video

Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr, Hao Fei

The paper says

“Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict.”

more of the abstract

Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 23 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.

A well-built survey whose four-operation framework and coherence-versus-consistency distinction are the real contribution; the abstract's trajectory claim is plausible but outruns the hand-curated evidence it rests on. This line is wrong

3 major objections · 3 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Abstract and Figure 1

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The abstract's shift claim [b5] is load-bearing, but the evidence is a hand-curated milestone chart and representative tables. Figure 1 places reading and writing milestones across 2017-2026, while the interacting line is populated almost entirely by 2023-2026 models. Because the operation categories are partly chosen by current research emphasis, selecting representative milestones within each category and then reading dates off the resulting timeline can manufacture the asserted shift.

more of objection 1

No denominator, sampling protocol, or inter-annotator consistency check is reported for the trajectory claim. Please either temper the claim to a scoped statement about current emphasis and open challenges, or add a year-stratified audit over the in-scope literature with explicit inclusion criteria.

The paper says, at Section 3.4

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The scope exclusions [b139][b140] (explicit 3D structure such as neural fields and 3D Gaussians, purely symbolic or text-based world models, and MBRL whose state is not learned from video) mean that [b5], worded as a field-level shift, is computed over a video-centered subset.

more of objection 2

If substantial progress on world consistency lives in those excluded lines, the trend claim would be overstated. The paper should either explicitly scope the shift claim to the in-scope video-centered set or provide evidence that the excluded lines do not change the direction of the trend.

The paper says, at Figure 2 and Tables 4-16

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The paper appropriately labels its work selection as representative rather than exhaustive, but the trend direction is read from that selection. Since the trajectory claim is the empirical conclusion of the survey, the selection needs a stated protocol and at least a basic robustness check. A year-stratified random sample from the maintained repository, or a sensitivity analysis that drops the most influential milestones, would show whether the shift survives changes in the representative set.

3 minor objections

The paper says, at Throughout

Pith says: minor objection 1 of 3

Typesetting artifacts appear in the TOC and section headings, e.g., '/ecange' and '/gaead' in place of 'Sharing' and 'Interacting', and '✏' near 'Writing'. These should be cleaned before publication.

The paper says, at Figure 1

Pith says: minor objection 2 of 3

Several entries are listed more than once (for example Gen4U, GenCeption, and V-JEPA 2.1), and the legend does not explain how horizontal position or bracket structure should be read. Adding a legend and removing duplicates would make the timeline easier to audit.

show 1 more

The paper says, at Definition 2.4

Pith says: minor objection 3 of 3

World consistency is defined existentially over admissible world trajectories, so the force of the definition depends entirely on how the constraint set C is specified. A sentence encouraging papers that use this definition to state their C explicitly would help readers apply it to generated and predicted traces.

What would settle it

Audit the trajectory claim computationally: (1) export all entries from the project's GitHub repository with publication year and operation tags; (2) label each entry for whether its primary emphasis is clip-level/perceptual fidelity versus persistent state, controllability, physical/causal validity, or interaction/downstream evaluation, using a fixed keyword rubric or two independent annotators; (3) compute the year-stratified share of world-consistency emphasis in the curated repository and compare it with a matched random sample of arXiv cs.CV video papers from the same years, using the same rubric. If the growth in world-consistency emphasis survives in the broader sample, the [b5] shift stands; if it appears only in the curated timeline, the trend is a curation artifact and should be downgraded to a conjecture.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 23 Oct 2026

  1. Reader holds up confidence moderate
  2. Skeptic agrees: says it holds up only with conditions 1 min 51 s · 53,288 tokens read, 10,618 written The shift claim [b5] is read off a hand-curated timeline; a year-stratified audit is needed to rule out selection artifact.
  3. Editor wrote the verdict above 38 s · 54,914 tokens read, 1,557 written
  4. Referee major revision: 3 major objections, 3 minor 34 s · 53,537 tokens read, 5,181 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The short version: the framework is the contribution, and it is a good one. Organizing roughly six hundred works around read, write, share, and interact holds up, and Observation 2.5 — temporal coherence versus world consistency — is a real analytical distinction, not a repackaged one. Definition 2.4 makes it precise: local frame smoothness is a property of the trace, while world consistency is a property of the world trajectory the trace implies. That distinction then does actual work in the evaluation sections (5.1.2 and 7.4), which is the right test for a survey instrument. Table 2's evidence ladder, from plausible output to decision utility, is similarly practical: it gives authors and referees a concrete bar for what counts as a world-modeling claim. The companion project page and GitHub curation add field value. The paper is also disciplined about evidence. Closed systems appear only where an official source documents the capability, Section 3.4 states scope exclusions plainly, and the survey repeatedly separates what a model's design supports from what its evaluation demonstrates. The 2 by 3 taxonomy for world models (prediction space by transition interface) is clean, and the explicit choice to count VAE-latent prediction as observation-space avoids a common category error. Now the soft spot, in proportion. The abstract's trajectory claim — the shift from clip-level recognition toward persistent state and physical/causal validity — is the one load-bearing empirical statement, and it rests on Figure 1 and the representative tables. The paper acknowledges the selection is non-exhaustive, and the 3.4 exclusions are explicit, so those are not hidden assumptions. What remains unresolved is the temporal confound the stress-test flags: the interacting line is populated almost entirely by 2023-2026 models, and since the categories themselves reflect current emphasis, reading dates off that timeline can manufacture the asserted shift. There is no denominator, sampling protocol, or inter-annotator check. The trend is plausible, and I think probably real, but the survey's own evidence does not establish it. The fix is cheap: soften the abstract or add a year-stratified audit. One caveat about my own read: the copy I saw truncated mid-Section 7.2.1, but nothing framework-level depends on the missing tail, and no reviewer can audit all six hundred citations. That is inherent to the genre. Who this is for: anyone writing or reviewing a paper that claims world-model behavior. The vocabulary and the evidence ladder will get cited. It deserves a serious referee; the framework should be accepted as the contribution, and the trajectory claim is the main revision target.

The strongest objection

The central thesis has two parts: a definitional framework (Sections 2.1–2.6) that stands on its own, and an empirical trajectory claim stated in the abstract: 'a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity...' [b5]. The load-bearing support for that claim is Figure 1 plus the representative-work tables (Figure 2, Tables 4–16). The paper acknowledges that the selection is representative, not exhaustive, and Section 3.4 states explicit scope exclusions, so this is not a hidden assumption. What remains unresolved is the temporal confound: Figure 1 places reading and writing milestones across 2017–2026, while the interacting line is populated almost entirely by 2023–2026 models. Because the operation categories were themselves chosen partly by current research emphasis, selecting representative milestones within each category and then reading dates off the resulting timeline can manufacture exactly the shift the abstract asserts, even if the proportion of all video papers emphasizing world consistency did not grow. No denominator, sampling protocol, or inter-annotator consistency check is reported for the trajectory claim, so the direction of the trend is plausible but not established. The Section 3.4 exclusions (explicit 3D structure, symbolic/text world models, and MBRL not learned from video) compound this: the claim [b5] is worded as a field-level shift, but the evidence population is a deliberately narrowed video-centered subset.

The framework's formal definitions are internally coherent and carefully hedged; I found no internal inconsistency in Definitions 2.1–2.6 or Observation 2.5 that would undermine the organizational contribution. The evidence ladder (Table 2) is explicitly an evaluation target rather than a metric, and the paper states that its benchmark lists are representative, not exhaustive. The weakest link is the abstract's empirical trajectory [b5], because that sentence is the load-bearing trend claim: it is what turns a taxonomy into a diagnosis of the field. The support is a hand-selected timeline (Figure 1) and representative work tables, and the authors acknowledge the selection is representative. The unresolved issue is that the trend could be an artifact of how the operation categories and timeline were constructed: the interacting category's milestones cluster after 2023, and the categories themselves were chosen to capture current emphasis, so a temporal reading of the curated timeline is not independent evidence of a shift. The proposed audit is feasible because the project ships a GitHub repository that can serve as the candidate list, and it would settle whether the trend survives a broader, year-stratified sample. If the audit fails, the survey's framework remains useful, but the abstract's shift should be framed as a conjecture or a within-scope observation; that is why I recommend CONDITIONAL rather than REJECT. The reader's weakest assumption identified the same representativeness premise, and my concern sharpens it to the temporal-confounding mechanism behind the curated timeline.

The referee's report

If the framework holds, it is a valuable organizational contribution: the definitions are stable across chapters, the coherence/consistency distinction gives evaluation discussions a clear target, and the evidence ladder and world-model taxonomy are actionable. The survey is unusually broad and includes useful tables of representative systems, benchmarks, and data, plus a public resource page. The main open risk is not internal coherence but the empirical support for the temporal-shift thesis, which is the one part of the abstract that goes beyond a definitional contribution.

Pith's reading of the paper (a machine's summary, not the authors')

This survey attempts to organize the entire landscape of video foundation models — understanding, generation, combined systems, and world models — under one relation: an observed video is a partial, lossy, temporally ordered projection of a latent, evolving world. From that relation it derives four operations — reading (infer world properties), writing (synthesize traces from conditions), sharing (couple the two directions), and interacting (roll out state under actions) — and claims these cover what every line of current research does. It then traces a shift in the field's own emphasis, from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions. A reader should care because the framework separates two properties that are routinely conflated — temporal coherence (locally smooth frames) and world consistency (frames compatible with some admissible world trajectory) — and because it grades world-modeling evidence on a four-level ladder, from plausible output up to decision utility. If the paper is right, fragmented literatures share one vocabulary and one evaluation bar.

What carries the argument. The load-bearing machinery is the formal video–world relation of Definitions 2.1–2.6, whose main pieces are the latent temporal world $W$, the observed trace $V$, the represented state $s_t \sim q_\theta(\cdot \mid x_{\le t}, a_{< t})$ that a model builds (in general $s_t \neq w_t$), and the explicit distinction between temporal coherence — local agreement of neighboring frames, $d(\hat{x}_t, \hat{x}_{t+1}) \le \delta$ — and world consistency — existence of at least one admissible trajectory $w_{1:T} \in C$ and camera path whose sensing could have produced the trace. On this base the paper builds three further instruments: the four operations (reading, writing, sharing, interacting); a $2 \times 3$ taxonomy of world models crossing prediction space (observation-space vs latent-state) with transition interface (passive, action-conditioned, interactive); and a four-level evidence ladder running from plausible output, through fidelity to evidence and conditions, to a consistent world state, and finally to decision utility. These instruments do the organizational work of the survey: every model is placed by which operation it performs, and every evaluation is read as evidence at one of the four levels.

Core claim. The paper's central claim is that the relationship between video and the world it depicts is the organizing principle of video AI. Formally, it defines a latent temporal world $W = (w_t)_{t\ge 1}$ as a stochastic process with dynamics $w_{t+1} \sim T(\cdot \mid w_t, a_t)$, and an observed video $V = (x_1, \ldots, x_T)$ as a trace produced by a sensing process $x_t \sim O(\cdot \mid w_t, \pi_t)$: partial, because only part of the world is visible under the camera configuration; lossy, because distinct world states can produce indistinguishable observations; and temporally ordered, because frame order carries evidence about the transition law. Around this the paper groups the literature into four operations: reading ($V \to W$), writing ($W \to V$), sharing ($V \leftrightarrow W$ through common tokens, parameters, representations, or objectives), and interacting (advancing and updating world state under actions and new observations). The load-bearing trend claim is that the field is moving from clip-level recognition and perceptual fidelity toward persistent represented state, controllable evolution, physical and causal validity, and evaluation that checks interaction and downstream decisions. The paper also asserts, as a matter of evaluation logic, that temporal coherence and world consistency are independent properties, so world-modeling evidence must test consistency directly rather than infer it from smooth frames.

The premise it rests on. The survey's trajectory claim is computed over its chosen corpus of video-centered models, so if the strongest evidence for persistent state, physical validity, or decision utility lives in the excluded lines — explicit 3D structure (neural fields, 3D Gaussians, occupancy grids), purely symbolic or text world models, or model-based RL world models whose state is not learned from video — the claimed shift would be overstated, and the 'representative, not exhaustive' selection of works must also be representative for the trend to hold as stated.

If the paper is right (Pith's inferences, not the authors')

World-modeling claims should be graded on the evidence ladder: plausible output alone no longer licenses statements about state, and claims of persistence, physics, or causality require consistency-level or decision-utility evidence (Table 2).

Because neither temporal coherence nor world consistency implies the other, benchmarks must probe consistency directly — object identity across occlusion, layout across revisits, effects across interventions — instead of inferring it from frame-smoothness metrics.

Video world models sort into six cells by prediction space (observation vs latent state) and transition interface (passive, action-conditioned, interactive), which separates what a model predicts from how it is conditioned and updated.

In unified models, shared tokens, parameters, or objectives demonstrate coupling, but synergy requires a controlled ablation of the cross-direction path under matched data, capacity, and compute.

Understanding and generation share a limit: what a model can do is bounded by the state it keeps and grounds, not by how many frames its context window accepts.

Editorial inference: the coherence–consistency distinction suggests a concrete, testable re-ranking exercise — score current video generators with standard metrics (FVD, VBench-style suites) and separately with consistency probes (identity persistence under occlusion, layout persistence on revisit, response to interventions) and check whether the two orderings diverge.

Editorial inference: if the paper's trajectory claim is correct, the next evaluation generation will be decision-grounded — whether a generated or predicted rollout selects the right action — and current perceptual metrics become necessary but not sufficient.

Editorial inference: the four-operation lens could be used diagnostically — a model's operation assignment (pure writer vs interactor, say) may predict which consistency failures it will exhibit, a hypothesis the paper does not itself test.

Editorial inference: the framework assumes a single latent world behind a trace; settings where one video supports several competing world interpretations, such as counterfactual editing or multi-agent scenes, may require extending the deterministic 'admissible trajectory' notion to a distribution over worlds.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-23

The authors' main claim, as the classifier picked it out 1.00

“Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.10 · human participants 0.04 · controlled comparison 0.02 · randomized assignment 0.01 · states quantitative result 0.02 · compares with prior results 0.09 · mentions code or data availability 0.02

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. background 1.00 0.04 0.04 asserted 1.00
Despite their different objectives, both concern the relationship between video observations and an evolving world. background 0.97 0.09 0.04 asserted 1.00
This survey adopts world modeling as a unifying lens for examining these developments. contribution 0.88 0.85 0.08 asserted 0.96
Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. contribution 0.75 0.63 0.19 asserted 0.99
This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. contribution 0.77 0.56 0.12 asserted 1.00
Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow. contribution 0.60 0.84 0.68 asserted 1.00

What kind of paper the classifier reports this is: review 1.00 · evidence from literature 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-23: 1 request, 4,986 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (17)
Figure 1 of the paper
Figure 1. Selected milestones in the parallel evolution of video understanding, generation, shared representations and workflows, and world prediction and control. 5
Figure 2 of the paper
Figure 2. Knowledge graph of the four research fields. Reading, writing, sharing, and interacting branch into technical streams and representative works. 8
Figure 3 of the paper
Figure 3. Four families of video representation learning. They differ in how supervision targets are constructed and how predictions are compared with or generated from those targets. 4.2.1 Contrastive Learning Contrastive learning pulls positive pairs together in embedding space and pushes mismatched pairs apart, commonly with an InfoNCE-style objective. In video-only learning, CVRL [380] and ρMoCo [103] draw positives from augmentations or clips of the same video, and the sampling distance sets a trade-off: views too close give little contrast, while views too far apart may show different…
Figure 4 of the paper
Figure 4. Latent representations for video generation. Continuous, discrete, hybrid, and semantically aligned designs differ in how tokenizers and VAEs shape reconstruction fidelity, compression, sequence modeling, and unification with understanding. Semantic alignment can be applied to either continuous or discrete representations. 5.2.2 Discrete Tokens From vector quantization to codebook-free schemes. VQ-VAE [359] replaces each encoder output with the index of a learned codebook entry, producing categorical symbols that next-token or masked-token models can predict. VQGAN [93] improves…
Figure 5 of the paper
Figure 5. Generative modeling paradigms. Diffusion and flow matching learn denoising or velocity targets over video latents. Autoregressive modeling factorizes an ordered sequence, while masked prediction fills selected positions from visible context through parallel conditional predictions. instead decode with a diffusion model. Divot learns continuous features for LLM-based understanding and reconstruction, and DiCoDe compresses video into deep tokens that an LLM predicts autoregressively. Semantic alignment adds a further requirement. A reconstruction-optimal latent may keep texture and…
Figure 6 of the paper
Figure 6. System-level organization for video generation. An LLM or MLLM plans for a visual generator through an explicit plan, such as shot lists, storyboards, or layouts, or through learned plan embeddings, and agentic systems close the loop with scripting, generation, evaluation, and revision under a controller. 5.3.3 Flow-Based Generation Flow matching [280] trains a time-dependent vector field that transports samples from a simple base distribution to the data along a prescribed probability path, and generation integrates the corresponding ordinary differential equation. It differs from…
Figure 7 of the paper
Figure 7. Control mechanisms by intervention site. Each route is identified by the component through which its principal control signal enters generation: the input, the attention maps, the latent state, block features, parameters and embeddings, or the sampling update. Current agents mainly decompose long-form creation or coordinate tools. MovieAgent [519] assigns screenplay, directing, storyboard, and scene construction to cooperating agents, and Anim-Director [263] has an MLLM write a storyline and script, produce consistent scene references, and select outputs from generative tools. Mora…
Figure 8 of the paper
Figure 8. Four architecture families for unified video understanding and generation, ordered by the abstraction of what is shared: discrete-token unification shares a token sequence and autoregressive backbone; continuous latent-space unification shares a continuous representation with direction-specific heads; decoupled and modular unification links separate experts through a semantic bridge; and agentic and tool-augmented unification shares a task state maintained by a controller. The families can be combined: an agent may call tools built on any of the other three
Figure 9 of the paper
Figure 9. Learning paradigms in unified video models. Isolated learning keeps independently pretrained experts; dual learning relates inverse tasks or reconstruction cycles; co-learning optimizes both directions jointly; and synergistic learning targets cross-direction benefit. The paradigms can coexist. appearance, geometry, or motion that the decoder needs. 6.1.4 Agentic and Tool-Augmented Unification Agentic systems leave perception, generation, editing, and evaluation to specialized models. A controller reads the request and media, plans operations, calls tools, inspects their outputs,…
Figure 10 of the paper
Figure 10. The 2 × 3 taxonomy of video world models. Rows give the prediction space: observations (frames, video tokens, or video-decoding latents) or latent states (video-derived features predicted without required rendering). Columns give the transition: passive prediction without action input, action-conditioned prediction with controls fixed before prediction, and interactive prediction that receives new controls after feedback. Each cell shows its prediction form and representative models; x denotes observations, z features, a actions, H the horizon, and C and Q the context and target…
Figure 11 of the paper
Figure 11. Hidden state evolves during occlusion; replaying an old observation can miss that change. One method direction is memory that updates a hidden state dur￾ing occlusion, trained to predict how unseen entities evolve, instead of replaying stored frames. Such memory complements rollout train￾ing: LIVE [187] targets error growth through forward–reverse cycle consis￾tency without any external memory, so the two address different failures and can be combined. Memory must also fit an interactive budget: un￾capped archives grow with the roll￾out, and retention budgets [1] turn this growth in
Figure 12 of the paper
Figure 12. Changing gravity alters the trajectory while unrelated scene prop￾erties remain fixed. Physical grounding can be tested at three increasingly demanding levels. Reading benchmarks [23, 395, 578] test whether a model recognizes persistence, stability, contact, and counterfactual outcomes in observed events. Gener￾ation benchmarks [16, 17, 137, 241, 341, 598] ask whether a synthesized outcome follows physical commonsense, a named law, or the direction of an irreversible process (Section 5.1.2). Action- and parameter-conditioned models face a stronger test: changing an action or…
Figure 13 of the paper
Figure 13. Two systems with matched resources differ only in whether information passes between understanding and generation. On the method side, each transfer path can be built as a separate component that can be switched off, such as a temporal￾analysis loss on the generator or generated intermediate states passed to a temporal reasoner. Round-trip consistency between inferred trajectories and rendered video can then serve as a training signal as well as a test. The test itself compares two systems with the same tasks, data, capacity, and com￾pute that differ only in whether the…
Figure 14 of the paper
Figure 14. Video and feature predictions need different probes, but both face the same decision test. World-model evaluation now covers control, mem￾ory, physical validity, and decision utility as well as visual realism, but the resulting numbers are rarely comparable. Suites for generated video, action re￾sponse, and broader world properties [7, 89, 96, 193, 267, 545, 546, 583, 645, 647] differ in prompts, control spaces, horizons, and judges (Section 7.4), so scores do not transfer across suites. For example, two predictors guiding the same robot may return video and features, respectively (
Figure 15 of the paper
Figure 15. Action effects mix with camera and background motion. Internet video covers objects, agents, and dynamics but usually lacks the action behind each transition. Latent-action methods [32, 61, 403, 574] infer actions as a learned vari￾able (Section 7.2.1), enabling the use of far more video than labeled datasets, but scale alone does not specify what the variable represents. For example, a gripper may push a block while the camera pans and a windmill turns (
Figure 16 of the paper
Figure 16. A held-out scene–body pair tests whether familiar factors compose while preserving action effects. Scene generalization changes appearance, layout, ob￾jects, or local dynamics for the same body, as when a robot enters a new room. Embodiment generalization changes the body, sensors, action frequency, degrees of freedom, or physical meaning of an action, as when skills move from human hands to a robot arm. Zero-shot claims therefore depend on which factor is held out. V-JEPA 2-AC [9] transfers to new scenes without target data, but on a robot type seen in post-training, not an unseen…
Figure 17 of the paper
Figure 17. Valid futures differ in frequency, and invalid and unreadable samples must be counted separately. One direction is to develop genera￾tors that assign calibrated outcome probabilities rather than merely pro￾duce diverse samples, using empiri￾cal event rates or justified simulator priors. Generated rare cases must be checked for physical feasibility, causal relevance, and downstream utility before entering training data, since prompting fixes which failures appear but not their frequencies. For a fixed history and action se￾quence, distributional evaluation as￾sesses whether samples…

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment