Pith. sign in

REVIEW 3 major objections 5 minor 52 references

Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Predicting the action between two frames can be used to train a vision-language model to predict the next frame, a task it otherwise fails at.

desk verdict A useful, honest paper showing that VLMs can bootstrap forward dynamics from inverse dynamics, but the synthetic-data leg of the bootstrapping story needs tighter evidence. read the letter →

arxiv 2506.06006 v3 pith:YIKPG3DR submitted 2025-06-06 cs.CV cs.AIcs.CL

Yifu Qiu , Yftah Ziser , Anna Korhonen , Shay B. Cohen , Edoardo M. Ponti This is my paper
classification cs.CVcs.AIcs.CL
keywords worldmodelsforwarddynamicspredictioninversevision-languageaction-centricimageeditingsynthetictrajectoriestest-timeverificationmultimodalgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to give a general-purpose vision-language model the ability to predict the next image frame from a current frame and a language instruction, a task called forward dynamics prediction, at which off-the-shelf VLMs fail, showing near-chance preference for real transitions over manipulated ones. The authors' central claim is that the reverse direction, inverse dynamics prediction, which captions the action that turns one frame into the next, is far easier to learn, and that this easier skill can bootstrap the harder one. They demonstrate two bootstrap routes in a single VLM: using the fine-tuned inverse model to annotate about 45 hours of unlabeled video with actions, turning those pairs into weak supervision for the forward model, and using the same inverse model at inference time to score candidate next frames and pick the best. On the action-centric AURORA-BENCH suite, the resulting forward model, still a general-purpose VLM, is reported to improve on state-of-the-art specialized image-editing models by between 7% and 13% under a GPT-4o judge and to achieve the best average human evaluation. If the claim holds, it offers a practical recipe for building world models without large amounts of paired action-outcome data.

What carries the argument

The load-bearing mechanism is the inverse dynamics model (IDM), namely the same VLM fine-tuned to output $p_{\mathrm{IDM}}(a \mid o_s, o_t)$, which performs two jobs. In the synthetic-supervision strategy, it annotates optical-flow-selected key-frame pairs from unlabeled videos with action labels, and a stratified top-k sampling step keeps triplets with high IDM likelihood while preserving class diversity; the resulting triplets train the forward dynamics model (FDM) through a recognition-weighted loss that multiplies each next-frame image token's loss by the squared L2 distance between the pre-quantized source and target features. In the verification strategy, the same IDM assigns reward $p_{\mathrm{IDM}}(a \mid o_s, o_t^{(i)})$ to each of $N$ sampled candidates, and the maximum-reward candidate becomes the final prediction. The mechanism's power is that one easy-to-learn function supplies both the extra training signal and the scoring function that the hard function lacks.

What would settle it

Train the same IDM on data with a random subset of action labels shuffled, use it to generate the synthetic triplets, then train the FDM on those triplets and compare GPT-4o scores on AURORA-BENCH; if the gains persist with shuffled labels, the reported bootstrap effect is not due to accurate inverse-dynamics knowledge.

Watch

Extended reading notes

Core claim

The central discovery is an asymmetry in how well a unified vision-language model grounds the two directions of a transition. Given an observation pair $(o_s, o_t)$, predicting the intervening action $a$ (inverse dynamics) becomes reliable after fine-tuning on a modest amount of ground-truth trajectories, whereas predicting $o_t$ from $(o_s, a)$ (forward dynamics) barely improves and remains prone to copying the source. The paper claims this asymmetry is exploitable: the inverse dynamics model can act as both a teacher and a judge for the forward dynamics model. As a teacher, it labels key-frame pairs selected by optical flow from unlabeled real-world videos, and the synthetic triplets are used to fine-tune the forward model under a recognition-weighted loss that concentrates gradients on image tokens that actually change. As a judge, it scores $N$ candidate next observations sampled from the forward model, and the candidate with the highest inverse likelihood is selected. With both strategies, the bootstrapped forward model, still a general-purpose VLM, is reported to improve over fine-tuning on supervised data alone by 13% for Chameleon and 7% for Liquid in average GPT-4o-as-judge scores across AURORA-BENCH, and to achieve the best human-evaluation average.

Load-bearing premise

The load-bearing premise is that the inverse model's automatically produced action labels for the unlabeled video pairs are correct enough to serve as training targets for the forward model; if the inverse model systematically names the wrong action, the forward model learns wrong (source, action, target) mappings and the reported gains would be an artifact of label noise rather than world knowledge.

Editorial extensions

If this is right

  • The IDM's synthetic annotations alone raise FDM performance: removing the unlabelled-video triplets drops GPT-4o scores on most AURORA-BENCH subsets, especially Something-Something and Action-Genome.
  • Inference-time verification with the IDM improves forward dynamics prediction to roughly the same level as trajectory synthesis, offering a training-free way to scale prediction quality with more candidate samples.
  • The recognition-weighted loss produces its largest gains in high-motion scenes, where copying the source frame fails, while slightly improving instruction-following at the cost of occasional over-editing.
  • FDP training transfers beyond editing: the bootstrapped Chameleon and Liquid models improve accuracy on the SpatialMQA and EmbodiedSpatial-Bench spatial-reasoning benchmarks over their zero-shot baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same inverse-to-forward bootstrap could apply to other modalities or settings where the 'action' coupling two states is easier to verbalize than the state itself, such as audio-to-video synchronization or robot control policies.
  • Nothing in the method prevents closing the loop: the improved FDM could generate harder candidate futures for the IDM to score, forming a self-improving dynamics cycle, but the paper evaluates verification only for a fixed FDM and does not retrain the IDM on FDM outputs.
  • The recognition-weighted loss is a natural training-time analogue of the 'minimal editing' criterion used by the GPT-4o judge; making the weights depend on the action rather than only on pixel differences could reduce the over-editing the paper reports.
  • Because the IDM and FDM share the same backbone, a systematic IDM error (for example, misnaming a class of actions) would be invisible to the verifier; an independent verifier trained on a different backbone would provide a stronger test of the bootstrap claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether unified vision-language models (VLMs) can perform forward dynamics prediction (FDP): predicting the next image observation from a source observation and a textual action. It first shows, via a zero-shot preference test on AURORA-BENCH, that nine VLMs do not consistently prefer ground-truth (o_s, a, o_t) trajectories over adversarially manipulated ones (Section 2). It then proposes to bootstrap FDP from an inverse dynamics model (IDM) that predicts actions from observation pairs. Two strategies are presented: (i) weakly supervised training on synthetic trajectories, where the IDM annotates key-frame pairs from unlabelled videos with actions, combined with a recognition-weighted loss (Sections 3.2); and (ii) inference-time verification, where the IDM scores multiple candidate future frames generated by the FDP (Section 3.3). The resulting forward dynamics models (C-FDM on Chameleon-7B and L-FDM on Liquid-8B) are evaluated on AURORA-BENCH with GPT-4o-as-a-judge and human evaluation, as well as on two spatial reasoning benchmarks. The paper reports that the best model improves over state-of-the-art image editing models by 7%–13% according to GPT-4o, and achieves the best average human evaluation.

Significance. If the central claims hold, the paper offers a practical and timely recipe for equipping general-purpose VLMs with elementary forward-dynamics capabilities without new human annotation, by transferring the easier inverse-dynamics task to a weak-supervision signal. The evaluation is more extensive than is typical for a method paper: it includes ablations for synthetic data and loss weighting, a blind human evaluation with significance tests, a judge-reliability analysis including pairwise win rates, and a stratified analysis of camera motion. The release of code and models is an additional strength. The main quantitative claims and the causal interpretation of the synthetic-data ablation, however, rest on assumptions that are not fully verified in the manuscript: the correctness of the IDM-generated action labels, the disjointness of training and evaluation data, and the statistical robustness of the reported margins.

major comments (3)
  1. [Section 4.1, H.1–H.3] The manuscript never states whether the AURORA data used to fine-tune the IDM (Sections 3.1 and H.1) and the FDM baselines (Sections 3.2 and H.3) are disjoint from the AURORA-BENCH evaluation subsets used in Section 4.3. Section 2 reports that each AURORA-BENCH subset contains 50 trajectory triplets; if these same triplets are included in the fine-tuning set, the reported gains over the zero-shot and fine-tuned baselines would be inflated or invalid. Please specify the official train/test split, confirm that the evaluation triplets are excluded from all training data, and clarify whether the same 50 triplets are used for the preference test in Section 2 and the FDP evaluation in Section 4.3.
  2. [Section 3.2, Eq. (1), Appendix E, Table 4] The synthetic trajectory labels are produced by the IDM and filtered in Algorithm 1 by the IDM's own predicted likelihood, with no independent verification of their correctness. The ablation in Table 4 shows that adding synthetic data changes FDP scores, but it does not establish that the semantic content of the IDM's action labels is the operative ingredient; the improvement could in principle stem from the extra visual diversity of the unlabelled video frames alone. To support the claim that IDP bootstraps FDP, the paper should include a control in which the same synthetic image pairs are paired with random (or shuffled) action texts, or provide a human or strong-model evaluation of a sample of the IDM-generated action labels. This is load-bearing because Eq. (1) optimizes p_theta(o_t | a_hat, o_s) with a_hat sampled from p_IDM, so systematic label errors would train the FDM on incorrect action-effect mappings.
  3. [Tables 2 and 4, Figure 4] The main results are reported as point estimates without error bars or confidence intervals, although the Limitations section acknowledges high variance across runs for Chameleon. Figure 4 shows a wide shaded standard deviation for C-FT on the average GPT-4o score, and Table 4 reports averages of 3 runs without indicating the spread. The headline improvements of 13% for Chameleon (C-FT 3.26 vs. C-FDM 3.67 in Table 2) and 7% for Liquid (L-FT 4.04 vs. L-FDM 4.32) may not be statistically significant if the underlying variance is high. Please provide confidence intervals or significance tests for the comparisons that motivate the abstract's performance claims.
minor comments (5)
  1. [Section 2] The text reports that Qwen2.5-VL-7B prefers the ground-truth action in 60.08% and 67.34% of cases, while Figure 2 appears to show 60.1 and 67.3; the rounding is acceptable, but please ensure the numbers are reported consistently.
  2. [Section 4.1] The distinction between AURORA and AURORA-BENCH is not defined in the main text; please state explicitly that AURORA refers to the training corpus and AURORA-BENCH to the evaluation benchmark, including the official split used.
  3. [Appendix H.5] The model list in the implementation section includes VILA-U, but VILA-U is not used in any experiment in the paper; please remove it or indicate where it is used.
  4. [Appendix G] The caption of Figure 11 refers to the metric as "OE", while the text and Table 3 use "ME" for Minimal Editing; the abbreviation should be made consistent.
  5. [Abstract] The phrase "competitive with state-of-the-art image editing models" should be qualified, because the comparison is on an action-centric image editing benchmark (AURORA-BENCH) and not on general image editing; consider rephrasing to avoid overgeneralization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims rest on held-out benchmark and human evaluation, and the IDP-to-FDP bootstrapping is a self-training procedure rather than a derivation that reduces to its inputs.

full rationale

The paper's central derivation chain is empirical rather than definitional. The claimed asymmetry between IDP and FDP is measured on held-out AURORA-BENCH trajectories with multiple negative controls, and the FDP results are compared against state-of-the-art image editing baselines using both GPT-4o-as-a-judge and human evaluation. Using the IDM to pseudo-label unlabelled video frames and then training the FDM on those labels is a form of self-training, not circularity: the FDM objective in Equation 1 is still grounded in real target observations o_t from videos, and the IDM labels â are intermediate language labels rather than the predicted quantity being evaluated. The IDM's own likelihood filtering in Appendix E is a quality heuristic, not a correctness certificate, and the paper does not claim it as one; the concern that noisy labels could make the synthetic-data gains a data-scale artifact is a correctness risk, not a circularity of derivation. The inference-time verification strategy selects among FDM candidates using IDM scores, but this is an empirical reranking method validated by external human judgments and by comparison with baselines, not a prediction that equals its inputs by construction. No load-bearing self-citation chain or imported uniqueness theorem appears; the authors' prior works are cited only as general context for world models and do not carry the central claim. The skeptical concerns in the provided commentary are empirical weaknesses about pseudo-label quality and metric reliability, and the paper itself acknowledges the copying failure mode and reports robustness checks. Therefore, no step in the claimed derivation reduces to its own inputs, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on benchmark label validity, likelihood-based trajectory preferences, IDM likelihood as reward, and GPT-4o as judge. No new physical entities are introduced; the main hand-set choices are key-frame extraction and synthetic data scale.

free parameters (3)
  • Key-frame interval If = 20
    Hand-set in Section 3.2 for all three video datasets; controls which frame pairs are annotated and therefore the synthetic trajectory distribution.
  • Key-frames per clip Kf = 6
    Hand-set in Section 3.2; defines how many observation pairs per clip enter annotation.
  • Synthetic triplets per dataset = 7K per dataset (MIT, UCF-101, Kinetics700)
    Chosen to balance compute and data scale; 7K per dataset is used in Chameleon FDM training (Section H.3).
assumptions (4)
  • domain assumption AURORA-BENCH ground-truth action-observation triplets are reliable labels for physically plausible transitions.
    Evaluation and human study assume these trajectory labels are correct; used as reference in Tables 2 and 5 and Figure 2 (Section 4.1).
  • domain assumption VLM next-token log-likelihood is a meaningful preference measure for trajectories.
    Section 2 compares log-likelihoods of reference versus negative trajectories to conclude that VLMs lack a consistent preference.
  • domain assumption IDM-predicted likelihood approximates correctness of an action for a frame pair.
    Test-time verification ranks candidates by p_IDM(a|os,ot) (Section 3.3), and synthetic data selection uses the same scores (Section 3.2, Appendix E).
  • domain assumption GPT-4o-as-a-judge scores reflect human notions of editing success and minimal editing.
    Primary automatic metric; supported by human evaluation, but it remains an assumed proxy for physical plausibility (Section 4.3, Appendix F).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics." pith.science (2026). https://pith.science/paper/YIKPG3DR

@misc{pith2026250606006,
  author       = {Pith},
  title        = {Pith review of: Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YIKPG3DR}},
  note         = {Machine review of arXiv:2506.06006}
}
read the original abstract

Can unified vision-language models (VLMs) perform forward dynamics prediction (FDP), i.e., predicting the future state (in image form) given the previous observation and an action (in language form)? We find that VLMs struggle to generate physically plausible transitions between frames from instructions. Nevertheless, we identify a crucial asymmetry in multimodal grounding: fine-tuning a VLM to learn inverse dynamics prediction (IDP)-effectively captioning the action between frames-is significantly easier than learning FDP. In turn, IDP can be used to bootstrap FDP through two main strategies: 1) weakly supervised learning from synthetic data and 2) inference time verification. Firstly, IDP can annotate actions for unlabelled pairs of video frame observations to expand the training data scale for FDP. Secondly, IDP can assign rewards to multiple samples of FDP to score them, effectively guiding search at inference time. We evaluate the FDP resulting from both strategies through the task of action-centric image editing on Aurora-Bench with two families of VLMs. Despite remaining general-purpose, our best model achieves a performance competitive with state-of-the-art image editing models, improving on them by a margin between 7% and 13% according to GPT4o-as-judge, and achieving the best average human evaluation across all subsets of Aurora-Bench.

Figures

Figures reproduced from arXiv: 2506.06006 by the authors.

Figure 1
Figure 1. A high-level illustration of our two strategies [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Percentage of times 9 VLMs assign higher probability to observation–action–observation Reference [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Heatmap visualization of image token weights predicted by the recognition model on examples from AG, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: GPT-4o scores for test-time verification with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: A qualitative case of real-world next-observation prediction, demonstrating C-FDM’s ability to steer pre [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Qualitative examples of the predicted next observation from the state-of-the-art specialised image editing [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Comparison of predicted negative log-likelihoods (lower values indicate stronger model preference) for [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Comparison of negative log-likelihoods (lower values indicate stronger model preference) of the action [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Distributions of triplet log-likelihoods predicted by IDM on Movements-in-Time, UCF-101, and Kinetics 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Prompt template used for GPT-4o-as-a-judge evaluation. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: GPT4o scores’ distributions of editing success (ES) and minimal editing (OE) for C-FDM trained with [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Instructions given to participants and the interface developed for conducting the evaluation. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 30 canonical work pages

  1. [1]

    Recurrent world models facilitate policy evolution

    David Ha and J\" u rgen Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018

    work page 2018

  2. [2]

    Cosmos world foundation model platform for physical AI

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical AI . arXiv preprint arXiv:2501.03575 , 2025

    arXiv 2025

  3. [3]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning , 2024

    2024

  4. [4]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. 2024. URL https://openai. com/research/video-generation-models-as-world-simulators , 3:1, 2024

    work page 2024

  5. [5]

    WorldSimBench : Towards video generation models as world simulators

    Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, et al. WorldSimBench : Towards video generation models as world simulators. arXiv preprint arXiv:2410.18072 , 2024

    arXiv

  6. [6]

    Do as I can, not as I say: Grounding language in robotic affordances

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on robot learning , pages 287--318. PMLR, 2023

    work page 2023

  7. [7]

    Inner Monologue : Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner Monologue : Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning , 2022

    work page 2022

  8. [8]

    Imagine while reasoning in space: Multimodal visualization-of-thought

    Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli \'c , and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thought. arXiv preprint arXiv:2501.07542 , 2025

    arXiv 2025

Show all 52 references
  1. [9]

    A generalist agent

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022

    2022 arXiv

  2. [10]

    Learning interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114 , 1(2):6, 2023

    2023 arXiv

  3. [11]

    Mastering diverse control tasks through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature , 640(8059):647--653, 2025

    2025

  4. [12]

    World model on million-length video and language with blockwise RingAttention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise RingAttention . arXiv preprint arXiv:2402.08268 , 2024

    2024 arXiv

  5. [13]

    Do generative video models learn physical principles from watching videos? arXiv preprint arXiv:2501.09038 , 2025

    Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models learn physical principles from watching videos? arXiv preprint arXiv:2501.09038 , 2025

    2025 arXiv

  6. [14]

    Physically grounded vision-language models for robotic manipulation

    Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 12462--12469. IEEE, 2024

    2024

  7. [15]

    Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay B Cohen. Are large language model temporally grounded? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V...

    2024

  8. [16]

    Can language models encode perceptual structure without grounding? A case study in color

    Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders S gaard. Can language models encode perceptual structure without grounding? A case study in color. In Arianna Bisazza and Omri Abend, editors, Proceedings of the 25th Conference on Compu...

    2021

  9. [17]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 , 2024

    2024 arXiv

  10. [18]

    Learning Action and Reasoning-Centric Image Editing from Videos and Simulations

    Benno Krojer, Dheeraj Vattikonda, Luis Lara, Varun Jampani, Eva Portelance, Christopher Pal, and Siva Reddy. Learning Action and Reasoning-Centric Image Editing from Videos and Simulations . In NeurIPS , 2024. Spotlight Paper

    2024

  11. [19]

    Video PreTraining (VPT) : Learning to act by watching unlabeled online videos

    Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video PreTraining (VPT) : Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems , 35:24639--24654, 2022

    2022

  12. [20]

    Moments in time dataset: one million videos for event understanding

    Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE Transactions on Pattern Analysis and Machine Intel...

    2019

  13. [21]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017

    2017 arXiv

  14. [22]

    A short note on the kinetics-700 human action dataset

    Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987 , 2019

    1907 arXiv

  15. [23]

    UCF101 : A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101 : A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012

    2012 arXiv

  16. [24]

    VILA-U : a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. VILA-U : a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429 , 2024

    2024 arXiv

  17. [25]

    Scaling egocentric vision: The EPIC-KITCHENS dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The EPIC-KITCHENS dataset . In Proceedings of the European conference on compu...

    2018

  18. [26]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 , 2025

    2025 arXiv

  19. [27]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 , 2024

    2024 arXiv

  20. [28]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pattern reco...

    2022

  21. [29]

    InstructPix2Pix : Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix : Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 18392--18402, 2023

    2023

  22. [30]

    GoT : Unleashing reasoning capability of multimodal large language model for visual generation and editing

    Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li, Shilin Yan, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, et al. GoT : Unleashing reasoning capability of multimodal large language model for visual generation and editing. arXiv preprint arXiv:2503.10639 , 2025

    2025 arXiv

  23. [31]

    SmartEdit : Exploring complex instruction-based image editing with multimodal large language models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. SmartEdit : Exploring complex instruction-based image editing with multimodal large language models. In Proceedings of the IEEE/CVF Conferenc...

    2024

  24. [32]

    BERTScore : Evaluating text generation with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore : Evaluating text generation with bert. In International Conference on Learning Representations , 2020

    2020

  25. [33]

    ROUGE : A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE : A package for automatic evaluation of summaries. In Text summarization branches out , pages 74--81, 2004

    2004

  26. [34]

    BLEU : a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU : a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311--318, 2002

    2002

  27. [35]

    Variational best-of-n alignment

    Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. The Thirteenth International Conference on Learning Representations , 2025

    2025

  28. [36]

    Learning to predict by the methods of temporal differences

    Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3:9--44, 1988

    1988

  29. [37]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning , pages 2555--2565. PMLR, 2019

    2019

  30. [38]

    Transformers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and Fran c ois Fleuret. Transformers are sample-efficient world models. In Deep Reinforcement Learning Workshop NeurIPS 2022 , 2022

    2022

  31. [39]

    Transformer-based world models are happy with 100k interactions

    Jan Robine, Marc H \"o ftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models are happy with 100k interactions. arXiv preprint arXiv:2303.07109 , 2023

    2023 arXiv

  32. [40]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and Fran c ois Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems , 37:58757--58791, 2024

    2024

  33. [41]

    Video-LLaVA : Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA : Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 5971--5984, 2024

    2024

  34. [42]

    Janus-Pro : Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-Pro : Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 , 2025

    2025 arXiv

  35. [43]

    iVideoGPT : Interactive VideoGPTs are scalable world models

    Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. iVideoGPT : Interactive VideoGPTs are scalable world models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Pro...

    2024

  36. [44]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084--15097, 2021

    2021

  37. [45]

    Vision-language models provide promptable representations for reinforcement learning

    William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision-language models provide promptable representations for reinforcement learning. In Automated Reinforcement Learning: Exploring Meta-Learning, AutoML, and LLMs , 2024

    2024

  38. [46]

    3D-VLA: A 3D Vision-Language-Action Generative World Model

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3D-VLA: A 3D Vision-Language-Action Generative World Model . In International Conference on Machine Learning , pages 61229--61245. PMLR, 2024

    2024

  39. [47]

    Video as the new language for real-world decision making

    Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making. arXiv preprint arXiv:2402.17139 , 2024

    2024 arXiv

  40. [48]

    VideoAgent : Self-improving video generation

    Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, and Sherry Yang. VideoAgent : Self-improving video generation. arXiv preprint arXiv:2410.10076 , 2024

    2024 arXiv

  41. [49]

    ANOLE : An open, autoregressive, native large multimodal models for interleaved image-text generation

    Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. ANOLE : An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06135 , 2024

    2024 arXiv

  42. [50]

    LoRA : Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA : Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022

    2022

  43. [51]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  44. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.