VChain: Chain-of-Visual-Thought for Reasoning in Video Generation

Gordon Chen; Haonan Qiu; Ning Yu; Paul Debevec; Ziqi Huang; Ziwei Liu

arxiv: 2510.05094 · v2 · pith:22QOAOZJnew · submitted 2025-10-06 · 💻 cs.CV

VChain: Chain-of-Visual-Thought for Reasoning in Video Generation

Ziqi Huang , Ning Yu , Gordon Chen , Haonan Qiu , Paul Debevec , Ziwei Liu This is my paper

Pith reviewed 2026-05-22 13:15 UTC · model grok-4.3

classification 💻 cs.CV

keywords video generationvisual reasoningchain of thoughtkeyframe guidancemultimodal modelsinference-time adaptationstate transitions

0 comments

The pith

VChain uses sparse keyframes from multimodal models to guide pre-trained video generators toward coherent multi-step dynamics.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current video generators often fail to maintain logical chains of visual consequences across complex sequences. VChain bridges this by first asking a multimodal model to output a small number of critical keyframes that mark key states and transitions, then adapting the generator only at those sparse moments during inference. The method requires no retraining and little extra computation. If the keyframes reliably encode the necessary visual logic, generated videos should show more consistent outcomes without dense frame supervision.

Core claim

VChain is an inference-time chain-of-visual-thought framework that directs large multimodal models to produce a sparse set of critical keyframes as visual-state snapshots; these snapshots then steer the sparse visual-state adaptation of a pre-trained video generator exclusively at the identified key moments, yielding more coherent modeling of state transitions and consequences.

What carries the argument

The VChain pipeline that extracts sparse critical keyframes via multimodal reasoning and applies them for targeted inference-time visual-state adaptation.

If this is right

Video quality improves on complex multi-step scenarios without retraining the base generator.
The approach remains tuning-efficient and adds only minimal inference overhead.
Dense per-frame supervision is no longer required to enforce visual consistency.
State transitions and future visual outcomes become more reliably modeled.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same sparse-keyframe guidance pattern could be applied to other generative domains such as 3D animation or interactive simulation.
Allowing users to edit or supply the keyframes themselves would add direct control over narrative outcomes.
Adaptive selection of how many keyframes to request based on scene complexity could extend the method to longer clips.

Load-bearing premise

That the keyframes produced by multimodal models will accurately identify and represent the critical visual states and transitions required for coherent long-term video sequences.

What would settle it

A controlled test set of multi-step action videos where the generated output using VChain still shows incoherent state changes at moments between or beyond the supplied keyframes.

Figures

Figures reproduced from arXiv: 2510.05094 by Gordon Chen, Haonan Qiu, Ning Yu, Paul Debevec, Ziqi Huang, Ziwei Liu.

**Figure 1.** Figure 1: Overview of VChain. We introduce VChain, an inference-time tuning framework for reasoning in video generation. Given a user-provided prompt (e.g., “A rock and a feather are falling from the sky towards the ground.”), VChain leverages large multimodal models to generate a Chain of Visual Thoughts, which are a sparse set of causally important keyframes to guide the video generator via Sparse Inference-Time T… view at source ↗

**Figure 2.** Figure 2: VChain Framework. An overview of our three-stage inference-time pipeline for reasoning in video generation. (a) Visual Thought Reasoning: Given a user-provided text prompt, a large multimodal model (GPT-4o) infers a causal chain of events and generates a sequence of keyframes, termed the Chain of Visual Thoughts, via iterative reasoning and image synthesis. (b) Sparse Inference-Time Tuning: These visual th… view at source ↗

**Figure 3.** Figure 3: Qualitative Results - Baseline Comparison. T2V fails to capture the key causal interaction: the pins remain mostly static or jitter slightly, with no meaningful collision, revealing a lack of physical reasoning despite temporal coherence. T2V + Prompt Aug introduces relevant elements and motion, but the dynamics are erratic and implausible. Pins deform unnaturally, visual artifacts appear, and later frames… view at source ↗

**Figure 4.** Figure 4: Qualitative Results - Ablation Study. We compare VChain with two ablated variants. (1) Without Visual Thought: Although the model recognizes that the video should be in a first-person perspective based on the textual prompt, it fails to capture the correct visual pattern for a ball-catching viewpoint. In contrast, VChain leverages the reasoned Visual Thoughts to render step-by-step intermediate visual stat… view at source ↗

**Figure 5.** Figure 5: Example of Visual Thoughts. We show the reasoned Visual Thoughts of the input prompt: “Concentrated sulfuric acid is poured onto a wooden table”. The sequence illustrates our pipeline’s inferred causal progression across keyframes. Time [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗

**Figure 6.** Figure 6: GPT Keyframe Limitations. Qualitative examples showing the accumulated saturation and smoothness artifacts produced by gpt-image-1 during iterative keyframe generation. As each generated image is recursively used as part of the input for the next step, slight over-saturation and over-smoothing compound over time, leading to slight color shifts (e.g., yellow cast) and reduced photorealism in later frames… view at source ↗

**Figure 7.** Figure 7: First Frame System Message. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

**Figure 8.** Figure 8: Next Frame System Message. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

**Figure 9.** Figure 9: Reasoning Output Example. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

**Figure 10.** Figure 10: CSV Output Example [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗

**Figure 11.** Figure 11: More Qualitative Comparisons. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗

**Figure 12.** Figure 12: More Qualitative Comparisons. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗

**Figure 13.** Figure 13: Additional Qualitative Comparisons - Egg Fall. Baseline Comparisons Input prompt: "A ball is dropped onto a pillow." Without Visual Thought Without Sparse Tuning Input prompt: "A man falls off a pile of bricks." T2V T2V + Prompt Aug VChain (Ours) T2V T2V + Prompt Aug VChain (Ours) [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗

**Figure 14.** Figure 14: Additional Qualitative Comparisons - Pillow. Baseline Comparisons Without Visual Thought Without Sparse Tuning VChain (Ours) Input prompt: "A rock and a feather falling from the sky towards the ground" T2V T2V + Prompt Aug VChain (Ours) Input prompt: "An egg falling from the sky towards concrete ground" [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗

**Figure 15.** Figure 15: Additional Qualitative Comparisons - Rocket Feather. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗

**Figure 16.** Figure 16: Additional Qualitative Comparisons - Cup. Without Visual Thought Without Sparse Tuning T2V T2V + Prompt Aug VChain (Ours) Input Prompt: “An egg is dropped onto a pillow." [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗

**Figure 17.** Figure 17: Additional Qualitative Comparisons - Egg Pillow. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗

**Figure 18.** Figure 18: Additional Qualitative Comparisons - Oil Milk. Without Visual Thought Without Sparse Tuning T2V T2V + Prompt Aug VChain (Ours) Input Prompt: “A sliced orange is squeezed right above an empty glass cup.” [PITH_FULL_IMAGE:figures/full_fig_p022_18.png] view at source ↗

**Figure 19.** Figure 19: Additional Qualitative Comparisons - Orange. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗

read the original abstract

Recent video generation models can produce smooth and visually appealing clips, but they often struggle to synthesize complex dynamics with a coherent chain of consequences. Accurately modeling visual outcomes and state transitions over time remains a core challenge. In contrast, large language and multimodal models (e.g., GPT-4o) exhibit strong visual state reasoning and future prediction capabilities. To bridge these strengths, we introduce VChain, a novel inference-time chain-of-visual-thought framework that injects visual reasoning signals from multimodal models into video generation. Specifically, VChain contains a dedicated pipeline that leverages large multimodal models to generate a sparse set of critical keyframes as snapshots, which are then used to guide the sparse inference-time visual-state adaptation of a pre-trained video generator only at these key moments. Our approach is tuning-efficient, introduces minimal overhead and avoids dense supervision. Extensive experiments on complex, multi-step scenarios show that VChain significantly enhances the quality of generated videos.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

VChain's sparse MLLM keyframe guidance for video coherence is a reasonable inference-time idea but rests on thin evidence so far.

read the letter

The main thing to know is that this paper proposes VChain as a way to improve coherence in generated videos by using multimodal models to pick a few critical keyframes and then adapt a pre-trained video generator only at those sparse points during inference. It frames this as a chain-of-visual-thought approach to handle complex multi-step dynamics without retraining or dense conditioning. That direction makes sense as an extension of reasoning methods into generation, and the emphasis on keeping overhead low and avoiding full supervision is a practical strength. The paper does a decent job laying out the problem of state transitions and consequence chains where current video models fall short, and it positions the method as a bridge between strong visual prediction in MLLMs and the limitations of existing generators. The tuning-efficient angle is worth noting since retraining video models is expensive. On the soft spots, the central claims about significant quality gains come with almost no supporting numbers, baselines, or details on the actual adaptation mechanism in the abstract. Without those, it's hard to judge whether the sparse guidance really propagates intended outcomes across frames or just adds temporary constraints that get overridden. The concern about drift in long sequences without denser signals is a fair one to check, and if the full paper lacks clear ablations isolating the keyframe effect or comparisons to standard conditioning, that would weaken the case. This work is aimed at researchers in generative video and multimodal models who are looking for lightweight ways to add reasoning signals. A reader tracking inference-time tricks for better consistency could get some value if the experiments hold up. I would send it for peer review so the implementation and results can be properly examined.

Referee Report

2 major / 1 minor

Summary. The paper introduces VChain, an inference-time chain-of-visual-thought framework that leverages large multimodal models to generate a sparse set of critical keyframes. These keyframes guide sparse visual-state adaptation in a pre-trained video generator only at key moments, with the goal of improving coherence and consequence chains in complex multi-step video generation scenarios. The method is presented as tuning-efficient with minimal overhead, and the abstract claims that extensive experiments demonstrate significant quality enhancements.

Significance. If the empirical claims hold, the approach offers a practical way to inject reasoning signals from MLLMs into frozen video generators without retraining or dense supervision, which could help address drift in long-horizon dynamics. The emphasis on sparse guidance and efficiency is a potential strength for scalable applications in controllable video synthesis.

major comments (2)

[Abstract] Abstract: the claim that 'extensive experiments on complex, multi-step scenarios show that VChain significantly enhances the quality of generated videos' is unsupported by any reported metrics, baselines, implementation details, or error analysis, leaving the central empirical claim unverifiable from the provided text.
[Method] Method pipeline: the description of 'sparse inference-time visual-state adaptation' provides no concrete mechanism (e.g., latent injection, cross-attention at specific denoising steps, or state resetting), which is load-bearing for the claim that sparse keyframe guidance alone can produce coherent long-term state transitions without dense signals or retraining.

minor comments (1)

[Abstract] Abstract: the relation between 'chain-of-visual-thought' and prior chain-of-thought or visual reasoning literature could be briefly contextualized to clarify novelty.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our work. We address each major comment point by point below, clarifying aspects of the manuscript and outlining revisions to enhance verifiability and detail.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that 'extensive experiments on complex, multi-step scenarios show that VChain significantly enhances the quality of generated videos' is unsupported by any reported metrics, baselines, implementation details, or error analysis, leaving the central empirical claim unverifiable from the provided text.

Authors: We agree that the abstract, as a concise summary, does not embed specific quantitative metrics or implementation details. The full manuscript reports these in the Experiments section, including coherence metrics, baseline comparisons, and error analysis on multi-step scenarios. To directly address verifiability from the abstract, we will revise it to include a brief statement of key results (e.g., relative improvements in coherence and consequence consistency). revision: yes
Referee: [Method] Method pipeline: the description of 'sparse inference-time visual-state adaptation' provides no concrete mechanism (e.g., latent injection, cross-attention at specific denoising steps, or state resetting), which is load-bearing for the claim that sparse keyframe guidance alone can produce coherent long-term state transitions without dense signals or retraining.

Authors: We appreciate this observation on the need for explicit mechanism details. The current method section describes the overall pipeline and sparse guidance principle but does not fully specify the adaptation implementation. We will revise the manuscript by adding a dedicated subsection detailing the concrete mechanism, including latent-space injection of keyframe features at targeted denoising timesteps and state management to maintain transitions without retraining or dense signals. revision: yes

Circularity Check

0 steps flagged

No circularity: VChain proposes external-model-guided sparse adaptation without self-referential reduction

full rationale

The paper presents VChain as a new inference-time pipeline that calls pre-trained multimodal models to produce sparse keyframes and then applies those keyframes for sparse visual-state adaptation inside a frozen video generator. No equations, parameter-fitting steps, or self-citations are shown that would make the claimed quality gains equivalent to the method's own inputs by construction. The central contribution is an empirical technique that depends on independent external models and a novel guidance schedule; the derivation chain therefore remains self-contained and does not collapse into tautology.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The approach depends on the domain assumption that multimodal models can reliably identify critical visual states. No free parameters or invented entities are explicitly detailed in the abstract, though the number and selection of keyframes may involve unstated choices.

axioms (1)

domain assumption Large multimodal models exhibit strong visual state reasoning and future prediction capabilities suitable for generating critical keyframes.
Directly invoked in the abstract to justify the keyframe generation step.

pith-pipeline@v0.9.0 · 5702 in / 1286 out tokens · 51828 ms · 2026-05-22T13:15:57.025257+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

VChain contains a dedicated pipeline that leverages large multimodal models to generate a sparse set of critical keyframes as snapshots, which are then used to guide the sparse inference-time visual-state adaptation of a pre-trained video generator only at these key moments.
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We only use these keyframes as supervision, treating them as anchor points that encode important state changes... Lvchain(θ) = E ... ∥uθ(xt, t,c)−vt∥2

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models
cs.CV 2026-05 unverdicted novelty 7.0

CollabVR improves video reasoning performance by coupling vision-language models and video generation models in a closed-loop step-level collaboration that detects and repairs generation failures.
Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond
cs.AI 2026-04 unverdicted novelty 7.0

Proposes a levels x laws taxonomy for world models in AI agents, defining L1-L3 capabilities across physical, digital, social, and scientific regimes while reviewing over 400 works to outline a roadmap for advanced ag...
Video Models Can Reason with Verifiable Rewards
cs.CV 2026-05 unverdicted novelty 6.0

VideoRLVR uses SDE-GRPO optimization, dense decomposed rewards, and Early-Step Focus to train video diffusion models on verifiable reasoning tasks, outperforming supervised fine-tuning and other video generators on Ma...
DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models
cs.CV 2026-04 unverdicted novelty 6.0

DiffHDR converts LDR videos to HDR by formulating the task as generative radiance inpainting in a video diffusion model's latent space, using Log-Gamma encoding and synthesized training data to achieve better fidelity...
Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamics
cs.CV 2026-04 unverdicted novelty 5.0

Phantom generates visually realistic and physically consistent videos by jointly modeling visual content and latent physical dynamics via an abstract physics-aware representation.
Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamics
cs.CV 2026-04 unverdicted novelty 5.0

Phantom jointly models visual content and latent physical dynamics via a physics-aware video representation to generate physically consistent videos.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · cited by 5 Pith papers · 3 internal anchors

[1]

Accessed March 31, 2025 [Online] https://www.langchain.com

Langchain. Accessed March 31, 2025 [Online] https://www.langchain.com

work page 2025
[2]

Accessed August 31, 2024 [On- line]https://hailuoai.com/

Minmax team. Accessed August 31, 2024 [On- line]https://hailuoai.com/

work page 2024
[3]

Accessed June 17, 2024 [On- line] https://runwayml.com/research/ introducing-gen-3-alpha

Gen-3. Accessed June 17, 2024 [On- line] https://runwayml.com/research/ introducing-gen-3-alpha

work page 2024
[4]

Accessed December 9, 2024 [Online] https://klingai.kuaishou.com/

Kling. Accessed December 9, 2024 [Online] https://klingai.kuaishou.com/. Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, and 1 others

work page 2024
[5]

Cosmos World Foundation Model Platform for Physical AI

Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575. Andreas Blattmann, Tim Dockhorn, Sumith Ku- lal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, and 1 others. 2023. Stable video dif- fusion: Scaling latent video diffusion models to large datasets.arXiv preprint...

work page internal anchor Pith review Pith/arXiv arXiv 2023
[6]

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Cogview2: Faster and better text-to-image generation via hierarchical transformers. InNeurIPS. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, and 1 others. 2020. An image is worth 16x16 words: Transformers for image recognition at...

work page internal anchor Pith review Pith/arXiv arXiv 2020
[7]

Gemini: A Family of Highly Capable Multimodal Models

U-net: Convolutional networks for biomedical image segmentation. InInternational Conference on Medical image computing and computer-assisted intervention. Jascha Sohl-Dickstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namics. InICML. Jiaming Song, Chenlin Meng, and Stefano Er...

work page internal anchor Pith review Pith/arXiv arXiv 2015
[8]

Context Frame:A richly detailed prompt used to generate the first frame in theChain of Vi- sual Thoughts

work page
[9]

Concise Prompt:A concise version of the Context Frame prompt (the full version is too long, so the first image is paired with this concise prompt during sparse inference-time tuning)

work page
[10]

The Context Frame is passed to GPT’s gpt-image-1 API to produce the corresponding image

Consequences:A sequence of inferred physi- cal outcomes that define the expected trajec- tory of the generated video. The Context Frame is passed to GPT’s gpt-image-1 API to produce the corresponding image. To generate subsequent keyframes in theChain of Visual Thoughts, we concatenate all previously generated images in our chain into a single compos- ite...

work page
[11]

Concentrated sulfuric acid is poured onto a wooden table

an image-editing instruction and 2) a boolean flag indicating whether a terminal state has been reached. We pass the same inputs as before along with the editing instruction to the gpt-image-1 API to generate the next keyframe. We repeat this process iteratively, where we predict the next key moment and generate the corresponding image, un- til the boolea...

work page 2025
[12]

Infer the objects/people/elements present in the scene, the perspective of the camera, the spatial relationship between the objects in the scene as well as details not explicitly mentioned in the text prompt

work page
[13]

A man throws a ball

Create a detailed, movie-like description of the scene that evokes visuals with strong detail and composition cues. This is the Context Frame. It should clearly describe the objects/people/elements present in the scene, the perspective of the camera, and the spatial relationships between the objects in it as well as the details not explicitly mentioned in...

work page
[14]

This should be a short, one-sentence description of the context frame

Create a concise version of the context frame. This should be a short, one-sentence description of the context frame

work page
[15]

A cat pushes a glass of water off a table

Infer a sequence of consequences/changes from the text prompt, even if it is not explicitly mentioned. Use assertive languange to clearly describe the changes in appearance, shape, color, size, and position that may occur as a result. Example: ————– Input Prompt: "A cat pushes a glass of water off a table." Thoughts: In order for the cat to tip the glass ...

work page

[1] [1]

Accessed March 31, 2025 [Online] https://www.langchain.com

Langchain. Accessed March 31, 2025 [Online] https://www.langchain.com

work page 2025

[2] [2]

Accessed August 31, 2024 [On- line]https://hailuoai.com/

Minmax team. Accessed August 31, 2024 [On- line]https://hailuoai.com/

work page 2024

[3] [3]

Accessed June 17, 2024 [On- line] https://runwayml.com/research/ introducing-gen-3-alpha

Gen-3. Accessed June 17, 2024 [On- line] https://runwayml.com/research/ introducing-gen-3-alpha

work page 2024

[4] [4]

Accessed December 9, 2024 [Online] https://klingai.kuaishou.com/

Kling. Accessed December 9, 2024 [Online] https://klingai.kuaishou.com/. Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, and 1 others

work page 2024

[5] [5]

Cosmos World Foundation Model Platform for Physical AI

Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575. Andreas Blattmann, Tim Dockhorn, Sumith Ku- lal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, and 1 others. 2023. Stable video dif- fusion: Scaling latent video diffusion models to large datasets.arXiv preprint...

work page internal anchor Pith review Pith/arXiv arXiv 2023

[6] [6]

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Cogview2: Faster and better text-to-image generation via hierarchical transformers. InNeurIPS. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, and 1 others. 2020. An image is worth 16x16 words: Transformers for image recognition at...

work page internal anchor Pith review Pith/arXiv arXiv 2020

[7] [7]

Gemini: A Family of Highly Capable Multimodal Models

U-net: Convolutional networks for biomedical image segmentation. InInternational Conference on Medical image computing and computer-assisted intervention. Jascha Sohl-Dickstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namics. InICML. Jiaming Song, Chenlin Meng, and Stefano Er...

work page internal anchor Pith review Pith/arXiv arXiv 2015

[8] [8]

Context Frame:A richly detailed prompt used to generate the first frame in theChain of Vi- sual Thoughts

work page

[9] [9]

Concise Prompt:A concise version of the Context Frame prompt (the full version is too long, so the first image is paired with this concise prompt during sparse inference-time tuning)

work page

[10] [10]

The Context Frame is passed to GPT’s gpt-image-1 API to produce the corresponding image

Consequences:A sequence of inferred physi- cal outcomes that define the expected trajec- tory of the generated video. The Context Frame is passed to GPT’s gpt-image-1 API to produce the corresponding image. To generate subsequent keyframes in theChain of Visual Thoughts, we concatenate all previously generated images in our chain into a single compos- ite...

work page

[11] [11]

Concentrated sulfuric acid is poured onto a wooden table

an image-editing instruction and 2) a boolean flag indicating whether a terminal state has been reached. We pass the same inputs as before along with the editing instruction to the gpt-image-1 API to generate the next keyframe. We repeat this process iteratively, where we predict the next key moment and generate the corresponding image, un- til the boolea...

work page 2025

[12] [12]

Infer the objects/people/elements present in the scene, the perspective of the camera, the spatial relationship between the objects in the scene as well as details not explicitly mentioned in the text prompt

work page

[13] [13]

A man throws a ball

Create a detailed, movie-like description of the scene that evokes visuals with strong detail and composition cues. This is the Context Frame. It should clearly describe the objects/people/elements present in the scene, the perspective of the camera, and the spatial relationships between the objects in it as well as the details not explicitly mentioned in...

work page

[14] [14]

This should be a short, one-sentence description of the context frame

Create a concise version of the context frame. This should be a short, one-sentence description of the context frame

work page

[15] [15]

A cat pushes a glass of water off a table

Infer a sequence of consequences/changes from the text prompt, even if it is not explicitly mentioned. Use assertive languange to clearly describe the changes in appearance, shape, color, size, and position that may occur as a result. Example: ————– Input Prompt: "A cat pushes a glass of water off a table." Thoughts: In order for the cat to tip the glass ...

work page