Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

A video world model can extract its own structure—flow, depth, segments—from counterfactual prompts, and then learn from those extractions to predict motion better.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 20:01 UTC pith:SASI5JVX

load-bearing objection Ambitions are real and the framework is genuinely novel, but the paper's central integration claim is undercut by a missing compute-matched baseline in Table 4. the 4 major comments →

arxiv 2509.09737 v1 pith:SASI5JVX submitted 2025-09-10 cs.CV cs.AIcs.LG

World Modeling with Probabilistic Structure Integration

Klemen Kotar , Wanhee Lee , Rahul Venkatesh , Honglin Chen , Daniel Bear , Jared Watrous , Simon Kim , Khai Loong Aw
show 8 more authors
Lilian Naing Chen Stefan Stojanov Kevin Feigelis Imran Thobani Alex Durango Khaled Jedoui Atlas Kazemian Dan Yamins
This is my paper
classification cs.CV cs.AIcs.LG
keywords world modelsvideo predictioncounterfactual promptingoptical flowself-supervised depthobject segmentationrandom-access autoregressive modelingtoken integration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes Probabilistic Structure Integration (PSI), a three-step loop for building self-improving world models from raw video. First, train a distributional autoregressive model that can condition any patch of a video on any other patch. Second, extract intermediate structures—optical flow, object segments, depth—zero-shot by comparing the model's predictions under factual versus counterfactual prompts. Third, tokenize those structures and mix them back into the training sequence, so the same model learns to predict and condition on them. The paper argues this loop improves base video prediction—flow integration cuts prediction error from 223 to 198 on DAVIS and 307 to 173 on YouTube VID—and creates new control handles, such as sparse-flow steering and probability-of-motion maps.

Core claim

PSI's central claim is that a probabilistic world model can bootstrap its own vocabulary: classical vision quantities are causal inferences in the model, and once re-tokenized, they become first-class tokens that improve the model itself. Concretely, the model is a random-access autoregressive transformer over pointer-content sequences, where each patch is an addressable variable. Optical flow is extracted by adding a tracer bump to frame 0 and locating the patch in frame 1 whose predicted distribution shifts most; segments come from motion hypotheticals that force coherent object movement; depth comes from camera-translation hypotheticals and measuring parallax. The extracted structures are

What carries the argument

The core mechanism is the Local Random-Access Sequence (LRAS) model: an autoregressive transformer trained on interleaved pointer tokens and content tokens, where each pointer specifies a patch location and each content token is a code from a hierarchical local quantizer. This lets any patch condition any other patch, turning the intractable problem of modeling a full probabilistic graphical model over video into a standard sequence-prediction task. Structure extraction then works by comparing predictions under factual versus counterfactual prompts—effectively applying do-operations on the learned graphical model—while integration works by inserting new token types into the same pointer-cont

Load-bearing premise

The entire extraction-and-integration loop assumes that when PSI is given synthetic, out-of-distribution conditioning—a tracer dot, a moved patch, a fake camera translation—the model's response is a faithful read on real physical causes rather than an artifact of its learned priors; the paper explicitly concedes that a tracer could instead be interpreted as schmutz on the camera lens.

What would settle it

Take a pair of frames with known object motion and camera motion, place a small Gaussian bump only on the moving object in frame 0, and compare the KL-traced displacement to ground-truth optical flow. If the bump's inferred correspondence lands on a stationary background location or is diluted across the frame—indicating the model reads it as lens dirt rather than a surface marking—then the causal interpretation of structure extraction fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The same 7B model, without task-specific heads, handles unconditional future prediction, sparse-patch completion, counterfactual edits, and camera-conditioned novel view synthesis.
  • Optical flow, object segments, and depth can be extracted from an RGB-only model without supervised labels, and these extractions rival or beat specialized and self-supervised baselines.
  • After flow integration, sparse flow vectors become a direct control surface: specifying a few flow patches on an object produces physically plausible motion, improving object manipulation and novel view synthesis.
  • Flow integration reduces single-frame video prediction error from 223 to 198 on DAVIS and from 307 to 173 on YouTube VID, counteracting the static-frame collapse seen in RGB-only predictors.
  • Once flow is a token type, second-order statistics such as probability-of-motion maps can be computed directly from the model, giving planning systems an anticipatory signal from a single static frame.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The PSI cycle reads as a general recipe for bootstrapping a 'prompting language' in any domain with pointer-structured data and identifiable counterfactual probes; video is the first worked example, but the same loop could apply to neural recordings, geospatial data, or other high-dimensional spatiotemporal domains.
  • The paper only closes the integration loop for flow, not for depth or segments—a direct test of the framework's own logic is whether integrating depth yields comparable gains in novel-view synthesis and prediction, or whether flow is uniquely suited because it is a stronger causal inference.
  • Because structure extraction currently relies on hand-crafted causal probes, the biggest open step is automating the discovery of useful probes; if that succeeds, PSI would move from human-guided structure selection to a scalable, data-driven discovery process.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Probabilistic Structure Integration (PSI), a three-step loop for building controllable world models from raw video. Step 1 trains a distributional random-access autoregressive model (LRAS) over spatiotemporal patch variables. Step 2 extracts intermediate structures—optical flow, object segments, depth—through counterfactual and hypothetical prompts treated as causal inferences. Step 3 tokenizes these structures and mixes them into the training sequence for continual training, creating a self-improving cycle. The paper reports benchmark gains in flow tracking (Table 1), novel view synthesis and object editing (Table 2), segmentation and depth (Table 3), and, centrally, improved base RGB prediction after flow integration (Table 4). The main empirical claim of the paper is that integrating self-extracted flow tokens reduces motion collapse and improves next-frame prediction.

Significance. If the PSI loop works as claimed, it is a significant step toward self-improving, controllable world models: it provides a unified prompting interface for visual prediction and structure extraction, and its posterior-as-data integration mechanism is conceptually novel. The paper is commendable for making the probabilistic formulation explicit, for the pointer-based LRAS design with strict local tokenization, and for using WSD to enable seamless continual integration. The counterfactual KL-tracing procedure for flow (Eq. 10) is a clean distributional extension of earlier CWM ideas. However, the load-bearing empirical evidence for the integration cycle is currently incomplete: the headline comparison in Table 4 is confounded by additional training compute, and several benchmark claims rely on single-run numbers without error bars and on companion papers for details. The theoretical framing is broad, but the demonstrated closed-loop integration is limited to flow, as the paper itself states in Section 6.4.

major comments (4)
  1. [§4.1, Table 4] The central claim that flow integration improves base prediction is not yet controlled. The Key Training Details paragraph in §4.1 states that the flow-integrated model was produced by continuing training for an additional 0.5T tokens, resuming from the last Stable-phase checkpoint. The RGB-only row in Table 4 appears to be the pre-continuation checkpoint. No RGB-only model trained for a matched additional 0.5T tokens is reported. The improvement (DAVIS 223→198, YouTube 307→173) could therefore be due to extra training compute/tokens rather than to the flow token type. Please add a compute-matched RGB-only continuation baseline, or otherwise separate the effect of continued training from the effect of flow-token integration.
  2. [§4.1 ('Key Training Details')] The provenance of the flow labels used for integration is ambiguous. The text says 'flows extracted from our video dataset' but does not specify whether these flows come from the model's own KL-tracing procedure (Eq. 10), from a companion flow extractor, from an off-the-shelf supervised method, or from a combination. It also does not describe how the sparse displacement estimates are completed into dense flow fields or how occlusions are handled. This matters because the posterior-as-data claim depends on the labels being generated by the model's own causal inference, and because the integration result cannot be reproduced or interpreted without this information. Please detail the flow-label generation pipeline and any filtering or postprocessing.
  3. [§3.3, §3.4, Tables 1 and 3] The 'zero-shot' and 'state-of-the-art' claims are stronger than the evidence presented. Segmentation (Section 3.3) and depth (Section 3.4) pipelines rely on SEA-RAFT, a supervised optical flow model, to compute flow between the generated hypothetical and frame 0. Thus the extraction is not fully free of task-specific supervised components. In addition, Table 1 reports TAP-Vid First against two-frame baselines, not against state-of-the-art long-range trackers, so the abstract's 'state-of-the-art optical flow' claim is not supported by this paper alone. The depth evaluation in Table 3 reports only δ1, and segmentation is reported only on SpelkeBench. Please either temper the claims or provide complete benchmark comparisons with error bars.
  4. [§6.4, §6.3] The paper correctly discloses two important limitations, but they should be elevated in the framing. Section 6.4 states that the closed-loop integration is demonstrated only for flow, not for depth or segments. Section 6.3 acknowledges that the tracer counterfactual can be interpreted as 'schmutz on the camera lens', meaning the extracted flow may not be causal flow. Since the integration loop and all downstream structures depend on this generalization property, the main claims would benefit from a direct quantitative analysis of such failure modes, rather than only a qualitative caveat.
minor comments (5)
  1. [Abstract] Typo: 'any others set of variables' should be 'any other set of variables'. Also, the abstract claims 'state-of-the-art optical flow, self-supervised depth and object segmentation'; the body of this paper does not fully substantiate all three claims.
  2. [Tables 1–4] No table reports error bars, confidence intervals, or number of evaluation seeds. Given the relatively small differences in some comparisons (e.g., PSI vs. SEA-RAFT on Kubric in Table 1), it is important to show whether these differences are stable.
  3. [§2.2, Figure 2D] The scaling law plot in Figure 2D reports a slope of 0.045, but the axes are not labeled and the fit details are not given. Please add axis labels, units, and a brief description of the fitting procedure.
  4. [§3.3] The notation 'i→i+Δp' for the patch-copy counterfactual is not defined precisely; please specify whether Δp is an integer patch offset and how boundary cases are treated.
  5. [§4.5, Figure 14] The qualitative comparison in Figure 14 is illustrative, but the caption does not state whether the shown samples are representative or cherry-picked. Please add a statement about selection criteria.

Circularity Check

0 steps flagged

No significant circularity: structure extraction is validated against external benchmarks, and the integration gain, while confounded by unmatched compute, is not equivalent to the model's own inputs by construction.

full rationale

The paper's structure-extraction claims (flow, depth, segments) are not circular because the extracted quantities are evaluated against external ground-truth benchmarks (TAP-Vid in Table 1, NYUD/BONN and SpelkeEntitySeg in Table 3), not against the model's own counterfactual responses. Although §3.1 defines structures as "structured prompts comparing factual and hypothetical/counterfactual predictions from Ψ," the independent benchmarks break the definitional loop. The integration step (§4.1) is explicitly a "posterior-as-data" self-training loop, which is self-referential by design; however, the central quantitative claim (Table 4) is a reduction in prediction error on held-out ground-truth RGB frames, so the claimed improvement is not equivalent to the training target by construction. Same-group citations ([13], [22], [24], [17]) are used for background and companion detail, but the key numbers supporting the load-bearing claims appear in this paper's own tables, so the citations are not load-bearing. The most important caveat is an experimental-control confound, not a circularity: §4.1 reports continuing training for an additional 0.5T tokens with flow tokens but does not report a compute-matched RGB-only continuation, so the Table 4 gain could reflect extra compute rather than flow integration. That is a correctness/control gap, not a reduction of the claim to its inputs. The paper also honestly flags its own limitations in §6.4 (only local flow integration demonstrated) and §6.3 (a tracer could be read as "schmutz on the camera lens"), further indicating the assumptions are acknowledged. Overall, the derivation chain is not circular; score 1 reflects minor self-referential self-training and same-group citations that are not, in themselves, load-bearing circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The central integration claim rests on: (a) a handful of hand-chosen perturbation and selection hyperparameters whose values are not reported, (b) the assumption that counterfactual prompts act as physical interventions, and (c) the assumption that training on the model's own outputs improves rather than entrenches its errors.

free parameters (6)
  • Perturbation amplitude delta for flow extraction = not reported
    Eq. 10 adds a 'small Gaussian bump' delta to a patch in f0; magnitude must balance distribution shift vs. semantics; no value or selection protocol given.
  • Revealed patch subset R for flow extraction = not reported
    KL tracing conditions on random subset R of f1 patches; size and coverage are unstated but affect correspondence identification.
  • Camera translation baseline b for depth extraction = not reported
    Depth z_j = b / d_j; b sets the scale and is chosen by the user; only unscaled depth is claimed.
  • Motion threshold epsilon for P_motion = not reported
    P_motion sums mass over flow vectors with ||v|| > epsilon; epsilon is hand-chosen.
  • Patch displacement Delta p for segment prompts = not reported
    Segments are extracted by copying a patch to a shifted position; the magnitude and direction of the shift affect which objects move coherently.
  • Continued-training budget for integration = 0.5T tokens
    Integration trained for an additional 0.5T tokens during the Stable phase; the improvement in Table 4 depends on this budget and the mixture of pure and mixed sequences.
axioms (5)
  • domain assumption Psi trained on random-order LRAS sequences approximates the full joint distribution over patch variables, so any subset of pointer-content pairs can serve as conditioning.
    Eq. 1 claims Psi supports conditionals for arbitrary subsets; the training objective (Section 2.2) only sees random serializations within 2-4 frame clips, so full PGM coverage is assumed.
  • domain assumption A small appearance perturbation acts as a do-operation that traces the same physical point into the next frame (tracer/common-fate assumption).
    Eq. 10 flow extraction; Section 6.3 acknowledges the model may instead treat the tracer as lens noise.
  • domain assumption Pixels that move together under hypothetical prompts belong to the same object (common fate), and near objects show larger parallax under camera translation.
    Sections 3.3 and 3.4 definitions of segment and depth extraction.
  • ad hoc to paper Self-generated pseudo-labels (posterior-as-data) provide a valid training signal that improves the marginal likelihood of real video.
    Sections 4.1 and 4.5: continued training on z~t from Psi's own responses; this could reinforce model biases rather than ground truth.
  • domain assumption The HLQ tokenizer's strict locality means the same vocabulary can be reused for RGB, flow, and depth tokens without cross-modality interference.
    Section 4.1 Step 1; needed for token integration without new embeddings.
invented entities (3)
  • Pointer tokens (random-access addressing) independent evidence
    purpose: Serialize 2D/3D data in arbitrary order so that any conditional can be queried
    Not a physical entity but a new token type; its function is demonstrated by random-access conditioning results (Figs. 3-5) and scaling (Fig. 2D).
  • Probability-of-motion (P_motion) maps no independent evidence
    purpose: Higher-order structure summarizing the model's belief that each patch will move before any motion occurs
    Defined in Section 4.4 as a sum over Psi's own flow predictions; only qualitative heatmaps are shown, no quantitative external validation (e.g., predicting actual future motion).
  • Intermediate structure tokens (flow/depth/segment token types) independent evidence
    purpose: New token types integrated into training to expand control and prediction
    Optical flow, depth, and segments are external, measurable scene properties with independent benchmarks (Tables 1-3), so the associated tokens have falsifiable handles.

pith-pipeline@v1.3.0-alltime-deepseek · 32822 in / 16460 out tokens · 168714 ms · 2026-08-04T20:01:23.164853+00:00 · methodology

0 comments
read the original abstract

We present Probabilistic Structure Integration (PSI), a system for learning richly controllable and flexibly promptable world models from data. PSI consists of a three-step cycle. The first step, Probabilistic prediction, involves building a probabilistic graphical model Psi of the data, in the form of a random-access autoregressive sequence model. Psi supports a complete set of learned conditional distributions describing the dependence of any variables in the data on any other set of variables. In step 2, Structure extraction, we show how to extract underlying low-dimensional properties in the data, corresponding to a diverse set of meaningful "intermediate structures", in a zero-shot fashion via causal inference on Psi. Step 3, Integration, completes the cycle by converting these structures into new token types that are then continually mixed back into the training diet as conditioning signals and prediction targets. Each such cycle augments the capabilities of Psi, both allowing it to model the underlying data better, and creating new control handles -- akin to an LLM-like universal prompting language. We train an instance of Psi on 1.4 trillion tokens of internet video data; we use it to perform a variety of useful video prediction and understanding inferences; we extract state-of-the-art optical flow, self-supervised depth and object segmentation; and we use these structures to support a full cycle of predictive improvements.

Figures

Figures reproduced from arXiv: 2509.09737 by Alex Durango, Atlas Kazemian, Daniel Bear, Dan Yamins, Honglin Chen, Imran Thobani, Jared Watrous, Kevin Feigelis, Khai Loong Aw, Khaled Jedoui, Klemen Kotar, Lilian Naing Chen, Rahul Venkatesh, Simon Kim, Stefan Stojanov, Wanhee Lee.

Figure 2
Figure 2. Figure 2: LRAS Architecture. (A): Hierarchical Local Quantizer (HLQ) encodes each local patch into a sequence of local codes. The first code decodes to a low-resolution version of the input, while the rest provide details to upsample the reconstruction to the original resolution. (B): HLQ codes are transformed into the Pointer-Content representation to allow for random access to the patches when encoding and decodin… view at source ↗
Figure 3
Figure 3. Figure 3: Unconditional Promptable Prediction. Using the same unified model Ψ, single-frame unconditional prompt￾ing produces diverse plausible futures that reflect learned motion priors. 2.3 Diverse Inference Pathways Confer Rich Controllability The LRAS architecture’s flexible conditioning mechanism enables a rich spectrum of inference pathways through different prompting strategies. By varying which patches are p… view at source ↗
Figure 4
Figure 4. Figure 4: Patch-Conditional Promptable Prediction. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Camera-Conditional Prediction (Novel View Synthesis). [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Progressive Uncertainty Reduction Through Sequential Conditioning. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Optical Flow via KL Tracing. Procedure (top): Step 1 — run patch-conditional parallel prediction to obtain per-patch logits; Step 2 — add a tiny dot to f0 to form a perturbed input and recompute logits; Step 3 — take the KL divergence between perturbed and unperturbed predicted distributions to locate the correspondence in f1, yielding a flow vector. Extractions (bottom): flow tracks across diverse scenes … view at source ↗
Figure 8
Figure 8. Figure 8: Object Segments from Motion Hypotheticals. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Depth from Viewpoint Hypotheticals [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Mixing New Tokens Into Sequences (Integration Step 2). [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Improved Generation Control. By allowing PSI (Ψ) to accept flow tokens as conditioning, we expose a powerful new control surface to guide our generations. By strictly constraining the motion we wish our model to simulate, we significantly constrain the space of plausible outputs. WildRGB-D: Novel View Synthesis Model PSNR ↑ SSIM ↑ LPIPS ↓ MotionCtrl 12.39 0.293 0.404 ZeroNVS 16.14 0.460 0.283 ViewCrafter … view at source ↗
Figure 12
Figure 12. Figure 12: Improved Extractions Through Direct Intermediate Prediction. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Probability of Motion. A parallel prediction is generated using Ψf low conditioned on the input image. Instead of sampling from the logits, the cumulative probability of all tokens corresponding to any amount of motion is reported. This yields a 2D heatmap of the objects in the image that are likely to move. Top row: static scenes where objects show higher motion probability than backgrounds and surfaces.… view at source ↗
Figure 14
Figure 14. Figure 14: Unconditional Prediction with Flow Intermediate. [PITH_FULL_IMAGE:figures/full_fig_p027_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Regression, Diffusion, and Distributional Prediction. [PITH_FULL_IMAGE:figures/full_fig_p032_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Masked Visual Actions for Unified World Modeling

    cs.CV 2026-07 conditional novelty 7.0

    A single video model finetuned on masked pixel trajectories acts as both forward and inverse robot world model, enabling policy evaluation, planning, and action extraction.

  2. A Deep Learning Model of Mental Rotation Informed by Interactive VR Experiments

    q-bio.NC 2025-12 conditional novelty 7.0

    A deep neural model with equivariant, symbolic, and agentic components reproduces human accuracy and action counts on Shepard-Metzler mental rotation, but not full reaction-time behavior.

  3. Zero-shot World Models Are Developmentally Efficient Learners

    cs.AI 2026-04 unverdicted novelty 6.0

    A zero-shot visual world model trained on one child's experience achieves broad competence on physical understanding benchmarks while matching developmental behavioral patterns.

  4. Perceptual 3D Simulation With Physical World Modeling

    cs.CV 2026-06 unverdicted novelty 5.0

    P3Sim integrates a probabilistic physical world model with geometric conditioning and persistent memory to simulate 3D scenes under partial observations and incomplete transforms.

Reference graph

Works this paper leans on

91 extracted references · 27 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Univ of California Press, 1999

    Isaac Newton, I Bernard Cohen, and Anne Whitman.The Principia: mathematical principles of natural philosophy. Univ of California Press, 1999

    1999

  2. [2]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. InInternational Conference on Learning Representations, 2020

    2020

  3. [4]

    Reichert, Neil Rabinowitz, Andr´ e Barreto, and Thomas Degris

    David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Arnold, David P. Reichert, Neil Rabinowitz, Andr´ e Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. In34th International Conference on Machine Learning (ICML), pages 3191–3199, 2017

    2017

  4. [5]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning, 2024

    2024

  5. [6]

    Learning interactive real-world simulators.arXiv preprint arXiv:2310.06114, 1(2):6, 2023

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators.arXiv preprint arXiv:2310.06114, 1(2):6, 2023

    Pith/arXiv arXiv 2023

  6. [7]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨ uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Pith/arXiv arXiv 2023

  7. [8]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  8. [9]

    Cosmos tokenizer: A suite of image and video neural tokenizers.arXiv preprint arXiv:2411.12223, 2024

    Cosmos-Tokenizer Team. Cosmos tokenizer: A suite of image and video neural tokenizers.arXiv preprint arXiv:2411.12223, 2024

    Pith/arXiv arXiv 2024

  9. [10]

    Determining optical flow.Artificial Intelligence, 17(1-3):185–203, 1981

    Berthold KP Horn and Brian G Schunck. Determining optical flow.Artificial Intelligence, 17(1-3):185–203, 1981

    1981

  10. [11]

    Make3d: Learning 3d scene structure from a single still image.IEEE transactions on pattern analysis and machine intelligence, 31(5):824–840, 2008

    Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image.IEEE transactions on pattern analysis and machine intelligence, 31(5):824–840, 2008

    2008

  11. [12]

    Normalized cuts and image segmentation.IEEE Transactions on pattern analysis and machine intelligence, 22(8):888–905, 2000

    Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation.IEEE Transactions on pattern analysis and machine intelligence, 22(8):888–905, 2000

    2000

  12. [14]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.CoRR, abs/2212.10560, 2022

    Pith/arXiv arXiv 2022

  13. [15]

    David Marr.Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman, 1982

    1982

  14. [16]

    MIT press, 2009

    Daphne Koller and Nir Friedman.Probabilistic graphical models: principles and techniques. MIT press, 2009

    2009

  15. [17]

    3d scene understanding through local random access sequence modeling.arXiv preprint arXiv:2504.03875, 2025

    Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh, Jared Watrous, Honglin Chen, Khai Loong Aw, and Daniel LK Yamins. 3d scene understanding through local random access sequence modeling.arXiv preprint arXiv:2504.03875, 2025

    Pith/arXiv arXiv 2025

  16. [18]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bj¨ orn Ommer. Taming transformers for high-resolution image synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873– 12883, 2021

    2021

  17. [19]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. InAdvances in neural information processing systems, volume 30, 2017

    2017

  18. [20]

    Wang, David Hall, Percy Liang, and Tengyu Ma

    Kaiyue Wen, Zhiyuan Li, Jason S. Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective.ArXiv, abs/2410.05192, 2024

    Pith/arXiv arXiv 2024

  19. [21]

    Cambridge University Press, 2009

    Judea Pearl.Causality: Models, Reasoning and Inference. Cambridge University Press, 2009

    2009

  20. [22]

    Taming generative video models for zero-shot optical flow extraction.arXiv preprint arXiv:2507.09082, 2025

    Seungwoo Kim, Khai Loong Aw, Klemen Kotar, Cristobal Eyzaguirre, Wanhee Lee, Yunong Liu, Jared Watrous, Stefan Stojanov, Juan Carlos Niebles, Jiajun Wu, et al. Taming generative video models for zero-shot optical flow extraction.arXiv preprint arXiv:2507.09082, 2025

    arXiv 2025

  21. [23]

    Self-supervised learning of motion concepts by optimizing counterfactuals.arXiv preprint arXiv:2503.19953, 2025

    Stefan Stojanov, David Wendt, Seungwoo Kim, Rahul Venkatesh, Kevin Feigelis, Jiajun Wu, and Daniel LK Yamins. Self-supervised learning of motion concepts by optimizing counterfactuals.arXiv preprint arXiv:2503.19953, 2025

    Pith/arXiv arXiv 2025

  22. [24]

    Discovering and using spelke segments, 2025

    Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen, Seungwoo Kim, Luca Thomas Wheeler, Jared Watrous, Ashley Xu, Gia Ancone, Wanhee Lee, Honglin Chen, Daniel Bear, Stefan Stojanov, and Daniel Yamins. Discovering and using spelke segments, 2025

    2025

  23. [25]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. InEuropean Conference on Computer Vision (ECCV), pages 402–419. Springer, 2020

    2020

  24. [26]

    Sea-raft: Simple, efficient, accurate raft for optical flow

    Yihan Wang, Lahav Lipson Shi, and Jia Deng. Sea-raft: Simple, efficient, accurate raft for optical flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  25. [27]

    Do- duo: Learning dense visual correspondence from unsupervised semantic-aware flow

    Zixuan Jiang, Yifan Gao, Junyu Shi, Yujie Wang, Yinlong Zheng, Hujun Bao, and Xiaowei Zhou. Do- duo: Learning dense visual correspondence from unsupervised semantic-aware flow. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    2023

  26. [28]

    Smurf: Self-teaching multi-frame unsupervised raft with full-image warping

    Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. Smurf: Self-teaching multi-frame unsupervised raft with full-image warping. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3887–3896, 2021

    2021

  27. [29]

    Unifying (machine) vision via counterfactual world modeling.arXiv preprint arXiv:2306.01828, 2023

    Daniel M Bear, Kevin Fan, Damian Kuo, Damian Mrowca, Abhishek Gupta, Daniel LK Yamins, and Li Fei-Fei. Unifying (machine) vision via counterfactual world modeling.arXiv preprint arXiv:2306.01828, 2023

    Pith/arXiv arXiv 2023

  28. [30]

    Coun- terfactual world modeling for physical dynamics.arXiv preprint arXiv:2306.01830, 2023

    Rahul G Venkatesh, Honglin Duan, Kevin Fan, Daniel M Bear, Daniel LK Yamins, and Li Fei-Fei. Coun- terfactual world modeling for physical dynamics.arXiv preprint arXiv:2306.01830, 2023

    Pith/arXiv arXiv 2023

  29. [31]

    Generative modelling in latent space, April 2025

    Sander Dieleman. Generative modelling in latent space, April 2025. Blog post

    2025

  30. [32]

    Sequential modeling enables scalable learning for large vision models.arXiv preprint arXiv:2312.00785, 2024

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models.arXiv preprint arXiv:2312.00785, 2024

    Pith/arXiv arXiv 2024

  31. [33]

    A benchmark dataset and evaluation methodology for video object segmentation

    Federico Perazzi, Jordi Pont-Tuset, Benjamin McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 39 Probabilistic Structure Integration Stanford NeuroAI Lab

    2016

  32. [34]

    Anand Bhattad, Konpat Preechakul, and Alexei A. Efros. Visual jenga: Discovering object dependencies via counterfactual inpainting, 2025

    2025

  33. [35]

    Deep learning.Nature, 521:436–444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.Nature, 521:436–444, 2015

    2015

  34. [36]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  35. [37]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. InCVPR, 2020

    2020

  36. [38]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational Conference on Machine Learning, pages 1597–1607, 2020

    2020

  37. [39]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. InNAACL, 2019

    2019

  38. [40]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InCVPR, 2022

    2022

  39. [41]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078– 10093, 2022

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078– 10093, 2022

    2022

  40. [42]

    beta-vae: Learning basic visual concepts with a constrained variational framework.ICLR, 2017

    Irina Higgins, Loic Matthey, Arka Pal, et al. beta-vae: Learning basic visual concepts with a constrained variational framework.ICLR, 2017

    2017

  41. [43]

    Hierarchical text-conditional image generation with CLIP latents.CoRR, abs/2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents.CoRR, abs/2204.06125, 2022

    Pith/arXiv arXiv 2022

  42. [44]

    Language models are few-shot learners.NeurIPS, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, et al. Language models are few-shot learners.NeurIPS, 2020

    2020

  43. [45]

    Neural grammatical error correction with finite state transducers

    Felix Stahlberg, Christopher Bryant, and Bill Byrne. Neural grammatical error correction with finite state transducers. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4033–4039, Minneapolis, MN, 2019

    2019

  44. [46]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, andet al.Chain-of-thought prompting elicits reasoning in large language models.CoRR, abs/2201.11903, 2022

    Pith/arXiv arXiv 2022

  45. [47]

    Generative language modeling for automated theorem proving.CoRR, abs/2009.03393, 2020

    Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving.CoRR, abs/2009.03393, 2020

    Pith/arXiv arXiv 2009

  46. [48]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior.CoRR, abs/2304.03442, 2023

    Pith/arXiv arXiv 2023

  47. [49]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Justin Fu, Bob McGrew, Sergey Levine, Karol Hausman, and Chelsea Finn. Do as i can, not as i say: Grounding language in robotic affordances. InRobotics: Science and Systems (RSS), 2022

    2022

  48. [50]

    RT-2: Vision–language–action models transfer web knowledge to robotic control.CoRR, abs/2307.15818, 2023

    Anthony Brohan, Noah Brown, Brian Ichter, Karol Hausman, Alex Irpan, Yevgen Chebotar, Sergey Levine, Vincent Vanhoucke, and Quan Vuong. RT-2: Vision–language–action models transfer web knowledge to robotic control.CoRR, abs/2307.15818, 2023

    Pith/arXiv arXiv 2023

  49. [51]

    Danny Driess, Fei Xia, Michael Janner, Sergey Levine, Karol Hausman, andet al.PaLM-E: An embodied multimodal language model.CoRR, abs/2303.03378, 2023

    Pith/arXiv arXiv 2023

  50. [52]

    Lee, and Sergey Levine

    Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. InConference on Robot Learning (CoRL), pages 37–46, 2018

    2018

  51. [53]

    R3M: A universal visual representation for robot manipulation.CoRR, abs/2203.12601, 2022

    Ashvin Nair, Vitchyr Pong, Micha¨ el Chang, and Sergey Levine. R3M: A universal visual representation for robot manipulation.CoRR, abs/2203.12601, 2022

    Pith/arXiv arXiv 2022

  52. [54]

    Great apes anticipate that other individuals will act according to false beliefs.Science, 354(6308):110–114, 2016

    Christopher Krupenye, Fumihiro Kano, Satoshi Hirata, Josep Call, and Michael Tomasello. Great apes anticipate that other individuals will act according to false beliefs.Science, 354(6308):110–114, 2016

    2016

  53. [55]

    Spelke and Katherine D

    Elizabeth S. Spelke and Katherine D. Kinzler. Core knowledge.Developmental Science, 10(1):89–96, 2007

    2007

  54. [56]

    Auto-encoding variational bayes

    Diederik Kingma and Max Welling. Auto-encoding variational bayes. InICLR, 2014

    2014

  55. [57]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. Generative adversarial nets. InNeurIPS, 2014

    2014

  56. [58]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. InNeurIPS, 2020. 40 Probabilistic Structure Integration Stanford NeuroAI Lab

    2020

  57. [59]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, 2022

    2022

  58. [60]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, et al. Generative pretraining from pixels. InICML, 2020

    2020

  59. [61]

    Sigma-gpts: A new approach to autoregressive models.arXiv preprint arXiv:2404.09562, 2024

    Arnaud Pannatier Ning, Soufiane Hayou, Aishwarya Ghosh, Stefanie Jegelka, Johannes Kirschner, and Nikolay Malkin. Sigma-gpts: A new approach to autoregressive models.arXiv preprint arXiv:2404.09562, 2024

    Pith/arXiv arXiv 2024

  60. [62]

    Autoregressive image generation without vector quantization.arXiv preprint arXiv:2406.11838, 2024

    Zichen Gao, Tianhong Wang, Lijun Fan, Zangwei Liu, Yujie Tang, Junmin Chen, Jun Zhu, and Bo Zhang. Autoregressive image generation without vector quantization.arXiv preprint arXiv:2406.11838, 2024

    Pith/arXiv arXiv 2024

  61. [63]

    Xl-vae: Scaling up variational autoencoders for image generation with extra layers.arXiv preprint arXiv:2024.xxxxx, 2024

    Xiaoran Han, Ye Zhang, Yizhe Chen, et al. Xl-vae: Scaling up variational autoencoders for image generation with extra layers.arXiv preprint arXiv:2024.xxxxx, 2024

    2024

  62. [64]

    Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. InProc. 7th Int. Conf. on Machine Learning, pages 216–224, 1990

    1990

  63. [65]

    Pilco: A model-based and data-efficient approach to policy search

    Marc Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient approach to policy search. InProc. 28th Int. Conf. on Machine Learning (ICML), pages 465–472, 2011

    2011

  64. [66]

    Garcia, David M

    Carlos E. Garcia, David M. Prett, and Manfred Morari. Model predictive control: Theory and practice—a survey.Automatica, 25(3):335–348, 1989

    1989

  65. [67]

    World models.arXiv:1803.10122, 2018

    David Ha and J¨ urgen Schmidhuber. World models.arXiv:1803.10122, 2018

    Pith/arXiv arXiv 2018

  66. [68]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Learning latent dynamics for planning from pixels. InICML, 2019

    2019

  67. [69]

    Deep hierarchical planning from pixels

    Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. CoRR, abs/2206.04114, 2022

    Pith/arXiv arXiv 2022

  68. [70]

    Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker, and Henryk Michalewski

    Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari.CoRR, abs/1903.00374, 2019

    Pith/arXiv arXiv 1903

  69. [71]

    Lillicrap, and David Silver

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model.Nature, 588(7839):604–609, 2020

    2020

  70. [72]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. InProc. 6th Conf. on Robot Learning (CoRL), pages 2226–2240, 2023

    2023

  71. [73]

    ENTL: Embodied navigation trajectory learner

    Klemen Kotar, Aaron Walsman, and Roozbeh Mottaghi. ENTL: Embodied navigation trajectory learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10863–10872, October 2023

    2023

  72. [74]

    Learning graphical models of images, videos and their spatial transfor- mations.arXiv preprint arXiv:1301.3854, 2013

    Brendan J Frey and Nebojsa Jojic. Learning graphical models of images, videos and their spatial transfor- mations.arXiv preprint arXiv:1301.3854, 2013

    Pith/arXiv arXiv 2013

  73. [75]

    A graphical model for audiovisual object tracking.IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7):828–836, 2003

    Matthew J Beal, Nebojsa Jojic, and Hagai Attias. A graphical model for audiovisual object tracking.IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7):828–836, 2003

    2003

  74. [76]

    A temporal bayesian model for classi- fying, detecting and localizing activities in video sequences

    Manavender R Malgireddy, Ifeoma Inwogu, and Venu Govindaraju. A temporal bayesian model for classi- fying, detecting and localizing activities in video sequences. In2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 43–48. IEEE, 2012

    2012

  75. [77]

    Probabilistic programming with programmable variational inference.Proceedings of the ACM on Programming Languages, 8(PLDI):2123–2147, 2024

    McCoy R Becker, Alexander K Lew, Xiaoyan Wang, Matin Ghavami, Mathieu Huot, Martin C Rinard, and Vikash K Mansinghka. Probabilistic programming with programmable variational inference.Proceedings of the ACM on Programming Languages, 8(PLDI):2123–2147, 2024

    2024

  76. [78]

    3dp3: 3d scene perception via probabilistic programming.Advances in Neural Information Processing Systems, 34:9600–9612, 2021

    Nishad Gothoskar, Marco Cusumano-Towner, Ben Zinberg, Matin Ghavamizadeh, Falk Pollok, Austin Gar- rett, Josh Tenenbaum, Dan Gutfreund, and Vikash Mansinghka. 3dp3: 3d scene perception via probabilistic programming.Advances in Neural Information Processing Systems, 34:9600–9612, 2021

    2021

  77. [79]

    Gen: a general- purpose probabilistic programming system with programmable inference

    Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. Gen: a general- purpose probabilistic programming system with programmable inference. InProceedings of the 40th acm sigplan conference on programming language design and implementation, pages 221–236, 2019. 41 Probabilistic Structure Integration Stanford NeuroAI Lab

    2019

  78. [80]

    From word models to world models: Translating from natural language to the probabilistic language of thought.arXiv preprint arXiv:2306.12672, 2023

    Lionel Wong, Gabriel Grand, Alexander K Lew, Noah D Goodman, Vikash K Mansinghka, Jacob Andreas, and Joshua B Tenenbaum. From word models to world models: Translating from natural language to the probabilistic language of thought.arXiv preprint arXiv:2306.12672, 2023

    Pith/arXiv arXiv 2023

  79. [81]

    Modeling open-world cognition as on-demand synthesis of probabilistic models.arXiv preprint arXiv:2507.12547, 2025

    Lionel Wong, Katherine M Collins, Lance Ying, Cedegao E Zhang, Adrian Weller, Tobias Gersternberg, Timothy O’Donnell, Alexander K Lew, Jacob D Andreas, Joshua B Tenenbaum, et al. Modeling open-world cognition as on-demand synthesis of probabilistic models.arXiv preprint arXiv:2507.12547, 2025

    Pith/arXiv arXiv 2025

  80. [82]

    Oxford University Press, 2007

    Tim Maudlin.The metaphysics within physics. Oxford University Press, 2007

    2007

Showing first 80 references.