pith. sign in

arxiv: 2509.20360 · v3 · submitted 2025-09-24 · 💻 cs.CV

EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning

Xuan Ju , Tianyu Wang , Yuqian Zhou , He Zhang , Qing Liu , Nanxuan Zhao , Zhifei Zhang , Yijun Li
show 6 more authors
Yuanhao Cai Shaoteng Liu Daniil Pakhomov Zhe Lin Soo Ye Kim Qiang Xu
This is my paper

Pith reviewed 2026-05-18 13:41 UTC · model grok-4.3

classification 💻 cs.CV
keywords unified modelimage editingvideo editingin-context learningtoken sequencecross-modal transfervideo generationediting benchmark
0
0 comments X

The pith

A single model unifies image and video editing and generation by converting all inputs to one token sequence that supports in-context learning across modalities.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents EditVerse as a framework that treats text, images, and videos as parts of the same sequence of tokens fed into one model. Self-attention over this sequence lets the model pick up editing instructions from context examples and move knowledge between still-image tasks and video tasks without separate components for each. To make this possible for video, the authors built a pipeline that produced 232,000 video editing examples and trained the model jointly with large image and video collections. They also released EditVerseBench, a new test set for instruction-based video editing. The central claim is that this unified setup delivers stronger results than current separate systems and shows new abilities that emerge when modalities share the same learning process.

Core claim

By representing text, images, and videos as a single unified token sequence and applying self-attention, EditVerse enables robust in-context learning and natural cross-modal knowledge transfer inside one model. The approach supports flexible inputs and outputs of arbitrary resolutions and durations. Training combines a newly curated set of 232K video editing samples with large-scale image and video data. Experiments show the resulting model reaches state-of-the-art performance on both image and video editing and generation tasks while displaying emergent cross-modal capabilities.

What carries the argument

Unified token sequence representation of text, images, and video combined with self-attention to support in-context learning and cross-modal transfer.

If this is right

  • The same model can accept and produce outputs at any resolution or duration without architectural changes.
  • Editing instructions given in context transfer naturally from image examples to video outputs and vice versa.
  • Joint training on image and video data improves performance on both modalities beyond what separate training achieves.
  • A single trained system can replace multiple specialized tools for image generation, video generation, and their editing variants.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Developers could build applications that let users edit both photos and clips with the same interface and model weights.
  • The approach opens a path to test whether longer video sequences or mixed image-video prompts produce even stronger emergent behaviors.
  • If the token unification scales, future models might handle additional modalities such as audio or 3D content under the same mechanism.

Load-bearing premise

That turning every modality into tokens in one shared sequence and letting self-attention handle the rest will produce reliable in-context learning and cross-modal transfer without needing separate architectures or running into data problems.

What would settle it

A head-to-head test in which the unified model shows no improvement over separately trained image-only and video-only models on video editing accuracy or instruction following would disprove the benefit of the shared token sequence.

Figures

Figures reproduced from arXiv: 2509.20360 by Daniil Pakhomov, He Zhang, Nanxuan Zhao, Qiang Xu, Qing Liu, Shaoteng Liu, Soo Ye Kim, Tianyu Wang, Xuan Ju, Yijun Li, Yuanhao Cai, Yuqian Zhou, Zhe Lin, Zhifei Zhang.

Figure 1
Figure 1. Figure 1: The strong video editing performance of EditVerse [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of EditVerse. We design a unified framework for image and video editing and generation, which processes text and vision inputs into a unified sequence. The right part of the figure shows our positional embedding design. This framework leverages full self-attention to facilitate robust in-context learning and effective knowledge transfer among modalities. 3.1 INTERLEAVED TEXT AND VISION INPUT Follo… view at source ↗
Figure 3
Figure 3. Figure 3: Examples for the interleaved text and vision pattern. EditVerse is capable of processing image and video inputs and outputs of arbitrary resolution, duration, and sequential positions. 4 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Examples from the proposed EditVerseBench. EditVerseBench includes 200 editing pairs, evenly distributed across 20 editing categories as well as horizontal and vertical orientations. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: User study on EditVerseBench. Comparison on EditVerseBench. Since InsV2V (Cheng et al., 2023) and Lucy Edit (Team, 2025) are the only open-source instruction-based video edit￾ing method that exactly matches our setting, we se￾lected two well-known training-free methods, To￾kenFlow (Qu et al., 2025) and STDF (Yatim et al., 2024), as well as a first-frame propagation method, Senorita-2M (Zi et al., 2025), fo… view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of EditVerse and other video editing methods. EditVerse shows stronger context preservation and edit faithfulness. Complete comparisons are in the Appendix. Method ViCLIPdir ↑ ViCLIPout ↑ Tune-A-Video (Wu et al., 2023a) 0.131 0.242 TokenFlow (Qu et al., 2025) 0.128 0.237 STDF (Yatim et al., 2024) 0.093 0.227 Fairy (Wu et al., 2024) 0.140 0.197 InsV2V (Cheng et al., 2023) 0.174 0.236 SDEdit (M… view at source ↗
Figure 7
Figure 7. Figure 7: Compare EditVerse generated results with ground truth. Results show EditVerse can surpass ground-truth data quality by extracting knowledge from image and video generation data. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of ablation on training data. Image data plays a critical role. Training Datasets VLM evaluation Video Quality Text Alignment Temporal Consistency Image Video Gen Video Edit Editing Quality Frame Video Pick Score CLIP DINO ✓ ✓ ✗ 3.62 18.64 22.31 20.44 93.48 90.27 ✗ ✗ ✓ 5.76 19.41 25.22 22.37 98.26 97.83 ✓ ✗ ✓ 6.52 19.81 25.78 22.63 98.24 97.97 ✗ ✓ ✓ 6.40 19.72 25.37 22.51 98.77 98.60 ✓ ✓ ✓ 6.… view at source ↗
Figure 9
Figure 9. Figure 9: Failure case examples of EditVerse. (a) The model fails to add object (treasure chest) at the correct position (at the man’s feet). (b) Generation of blurry artifacts within the edited region. Computational Cost. Our reliance on a full self-attention mechanism across a unified one￾dimensional token sequence, while powerful for in-context learning, leads to significant compu￾tational overhead. The concatena… view at source ↗
read the original abstract

Recent advances in foundation models highlight a clear trend toward unification and scaling, showing emergent capabilities across diverse domains. While image generation and editing have rapidly transitioned from task-specific to unified frameworks, video generation and editing remain fragmented due to architectural limitations and data scarcity. In this work, we introduce EditVerse, a unified framework for image and video generation and editing within a single model. By representing all modalities, i.e., text, image, and video, as a unified token sequence, EditVerse leverages self-attention to achieve robust in-context learning, natural cross-modal knowledge transfer, and flexible handling of inputs and outputs with arbitrary resolutions and durations. To address the lack of video editing training data, we design a scalable data pipeline that curates 232K video editing samples and combines them with large-scale image and video datasets for joint training. Furthermore, we present EditVerseBench, the first benchmark for instruction-based video editing covering diverse tasks and resolutions. Extensive experiments and user studies demonstrate that EditVerse achieves state-of-the-art performance, surpassing existing open-source and commercial models, while exhibiting emergent editing and generation abilities across modalities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces EditVerse, a unified framework for image and video generation and editing. All modalities (text, image, video) are represented as a single token sequence processed by self-attention, enabling in-context learning and cross-modal transfer. A scalable pipeline curates 232K video-editing samples for joint training with image/video data; EditVerseBench is introduced as the first instruction-based video editing benchmark. The central claims are SOTA performance over open-source and commercial models plus emergent cross-modal editing/generation abilities.

Significance. If the empirical results and cross-modal transfer are robustly verified, the work would advance unification of vision foundation models by showing that a single self-attention transformer over mixed tokens can handle arbitrary-resolution image and video tasks without modality-specific architectures, while addressing video data scarcity through curated training data.

major comments (2)
  1. Abstract: the SOTA and emergent-ability claims are stated without any quantitative metrics, ablation details, error bars, or data-exclusion criteria, which are load-bearing for verifying that the model surpasses existing open-source and commercial baselines.
  2. Data curation and joint-training description (around the 232K video samples): the claim of natural cross-modal knowledge transfer via unified tokens and self-attention lacks supporting controls such as modality-ablated runs or attention-map analysis; without these it remains unclear whether video performance gains arise from genuine transfer or simply from extra capacity and the larger image corpus.
minor comments (1)
  1. Clarify the exact tokenization scheme and positional encoding used for variable-duration videos and arbitrary resolutions to ensure reproducibility of the unified sequence handling.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive review. We address each major comment below and indicate planned revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: Abstract: the SOTA and emergent-ability claims are stated without any quantitative metrics, ablation details, error bars, or data-exclusion criteria, which are load-bearing for verifying that the model surpasses existing open-source and commercial baselines.

    Authors: We agree that the abstract would be strengthened by including representative quantitative results. In the revised manuscript we will update the abstract to report key metrics from EditVerseBench (e.g., average gains over open-source and commercial baselines) together with explicit pointers to the full tables, ablations, error bars, and data-exclusion criteria already present in Sections 4 and 5. revision: yes

  2. Referee: Data curation and joint-training description (around the 232K video samples): the claim of natural cross-modal knowledge transfer via unified tokens and self-attention lacks supporting controls such as modality-ablated runs or attention-map analysis; without these it remains unclear whether video performance gains arise from genuine transfer or simply from extra capacity and the larger image corpus.

    Authors: We acknowledge that additional controls would provide stronger isolation of cross-modal transfer effects. Our current evidence rests on the observed emergent cross-modal editing/generation capabilities and the performance lift on video tasks when the model is trained jointly versus video-only. In the revision we will add attention-map visualizations to illustrate cross-modal attention patterns and expand the discussion of the unified token/self-attention design. Full modality-ablated training runs are not feasible given compute limits; we will therefore add an explicit limitations paragraph noting this constraint while clarifying why the architecture and in-context learning setup support transfer. revision: partial

Circularity Check

0 steps flagged

No circularity; claims rest on empirical training, data curation, and external benchmarks

full rationale

The paper presents EditVerse as an architectural choice (unified token sequence + self-attention over mixed modalities) together with an independent data-curation pipeline that produces 232K video-editing samples. These are then used for joint training whose outputs are measured on the newly introduced EditVerseBench and via user studies against external open-source and commercial baselines. No derivation chain, equation, or fitted parameter is shown to reduce by construction to its own inputs; the central performance and emergence claims are therefore falsifiable against held-out data and do not rely on self-citation load-bearing or self-definitional loops.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The framework rests on the standard transformer assumption that self-attention over mixed-modality tokens suffices for cross-modal transfer; the main added element is the engineering effort to curate video editing data rather than new mathematical axioms or invented physical entities.

free parameters (1)
  • tokenization and resolution handling parameters
    Arbitrary resolutions and durations are handled via the unified sequence; exact tokenization rules and padding strategies are not detailed in the abstract.
axioms (1)
  • domain assumption Self-attention on a unified token sequence enables robust in-context learning and natural cross-modal knowledge transfer.
    Stated directly in the abstract as the mechanism for unification.

pith-pipeline@v0.9.0 · 5773 in / 1311 out tokens · 39391 ms · 2026-05-18T13:41:27.678856+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. What Semantics Survive the Connector? Diagnosing VLM-to-DiT Alignment in Video Editing

    cs.CV 2026-05 unverdicted novelty 7.0

    VLM-to-DiT alignment in video editing models acts as a semantic bottleneck that degrades fine-grained structural semantics, demonstrated via a new diagnostic dataset and protocol on relation-based edits.

  2. Aurora: Unified Video Editing with a Tool-Using Agent

    cs.CV 2026-05 unverdicted novelty 7.0

    Aurora introduces a VLM-based agent that converts raw user video edit requests into structured conditioning inputs for a unified diffusion transformer, improving performance on underspecified tasks via a new benchmark.

  3. TrajectoryMover: Generative Movement of Object Trajectories in Videos

    cs.CV 2026-03 unverdicted novelty 7.0

    TrajectoryMover enables moving object trajectories in videos by training on large-scale synthetic paired data generated via the new TrajectoryAtlas pipeline.

  4. TrajectoryMover: Generative Movement of Object Trajectories in Videos

    cs.CV 2026-03 unverdicted novelty 7.0

    A synthetic data pipeline and fine-tuned video model enable generative editing to move object 3D trajectories in videos while keeping relative motion.

  5. VideoCoF: Unified Video Editing with Temporal Reasoner

    cs.CV 2025-12 unverdicted novelty 7.0

    VideoCoF adds an explicit reasoning step using edit-region latents in video diffusion models to enable precise mask-free editing and motion alignment with only 50k training pairs.

  6. Lance: Unified Multimodal Modeling by Multi-Task Synergy

    cs.CV 2026-05 unverdicted novelty 6.0

    Lance presents a dual-stream mixture-of-experts model with modality-aware positional encoding and staged multi-task training that outperforms prior open-source unified models on image and video generation while keepin...

  7. InsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptation

    cs.CV 2026-04 unverdicted novelty 6.0

    InsEdit adapts a video diffusion backbone for text-instruction video editing via Mutual Context Attention, achieving SOTA open-source results with O(100K) data while also supporting image editing.

  8. Bernini: Latent Semantic Planning for Video Diffusion

    cs.CV 2026-05 unverdicted novelty 5.0

    Bernini is a framework that uses an MLLM planner to output semantic representations for a DiT renderer to generate or edit videos, reporting SOTA benchmark performance.

  9. Lance: Unified Multimodal Modeling by Multi-Task Synergy

    cs.CV 2026-05 unverdicted novelty 5.0

    Lance introduces a dual-stream MoE model with modality-aware rotary positional encoding and staged multi-task training that outperforms open-source unified models on image and video generation while retaining understa...

Reference graph

Works this paper leans on

42 extracted references · 42 canonical work pages · cited by 7 Pith papers · 19 internal anchors

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    ReCamMaster: Camera-Controlled Generative Rendering from A Single Video, March 2025

    Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from a single video.arXiv preprint arXiv:2503.11647,

    work page arXiv

  3. [3]

    HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer

    Qi Cai, Jingwen Chen, Yang Chen, Yehao Li, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Yiheng Zhang, Fengbin Gao, Peihan Xu, et al. Hidream-i1: A high-efficient image generative foundation model with sparse diffusion transformer.arXiv preprint arXiv:2505.22705, 2025a. Yuanhao Cai, He Zhang, Xi Chen, Jinbo Xing, Yiwei Hu, Yuqian Zhou, Kai Zhang, Zhifei Zhang, S...

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025a. Junying Chen, Zhenyang Cai, Pengcheng Chen, Shunian Chen, Ke Ji, Xidong Wang, Yunjin Yang, ...

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Scaling Instruction-Finetuned Language Models

    URLhttps://arxiv.org/abs/2210.11416. Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Seed-data-edit technical report: A hybrid dataset for in- structional image editing.arXiv preprint arXiv:2405.04007,

    Yuying Ge, Sijie Zhao, Chen Li, Yixiao Ge, and Ying Shan. Seed-data-edit technical report: A hybrid dataset for instructional image editing.arXiv preprint arXiv:2405.04007,

    work page arXiv

  7. [7]

    Prompt-to-Prompt Image Editing with Cross Attention Control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control.arXiv preprint arXiv:2208.01626,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Vivid-10m: A dataset and baseline for versatile and interactive video local editing

    Jiahao Hu, Tianxiong Zhong, Xuebo Wang, Boyuan Jiang, Xingye Tian, Fei Yang, Pengfei Wan, and Di Zhang. Vivid-10m: A dataset and baseline for versatile and interactive video local editing. arXiv preprint arXiv:2411.15260,

    work page arXiv

  9. [9]

    Hq-edit: A high-quality dataset for instruction-based image editing

    Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction-based image editing.arXiv preprint arXiv:2404.09990,

    work page arXiv

  10. [10]

    Rtmpose: Real-time multi-person pose estimation based on mmpose,

    URLhttps:// arxiv.org/abs/2303.07399. Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing.arXiv preprint arXiv:2503.07598,

    work page arXiv

  11. [11]

    Direct inversion: Boosting diffusion-based edit- ing with 3 lines of code.arXiv preprint arXiv:2310.01506,

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting diffusion-based editing with 3 lines of code.arXiv preprint arXiv:2310.01506, 2023a. Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: A native skeleton-guided diffusion model for human image generation. InProceedings of the IEEE/CVF...

    work page arXiv

  12. [12]

    Auto-Encoding Variational Bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    HunyuanVideo: A Systematic Framework For Large Video Generative Models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Nohumansrequired: Autonomous high-quality image editing triplet mining

    Maksim Kuprashevich, Grigorii Alekseenko, Irina Tolstykh, Georgii Fedorov, Bulat Suleimanov, Vladimir Dokholyan, and Aleksandr Gordeev. Nohumansrequired: Autonomous high-quality image editing triplet mining.arXiv preprint arXiv:2507.14119,

    work page arXiv

  15. [15]

    FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space

    Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas M¨uller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 kontext: Flow matching for in-context image...

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Diffueraser: A diffusion model for video inpainting.arXiv preprint arXiv:2501.10018, 2025

    12 Xiaowen Li, Haolan Xue, Peiran Ren, and Liefeng Bo. Diffueraser: A diffusion model for video inpainting.arXiv preprint arXiv:2501.10018,

    work page arXiv

  17. [17]

    Flow Matching for Generative Modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Step1X-Edit: A Practical Framework for General Image Editing

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URLhttps:// llava-vl.github.io/blog/2024-01-30-llava-next/. Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. InProceedings of ...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations.arXiv preprint arXiv:2108.01073,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    YaRN: Efficient Context Window Extension of Large Language Models

    URLhttps://openai.com/index/ hello-gpt-4o/. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models.arXiv preprint arXiv:2309.00071,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [22]

    Movie Gen: A Cast of Media Foundation Models

    URLhttps://arxiv.org/abs/2410.13720. 13 Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. InProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15932–15942,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [23]

    SAM 2: Segment Anything in Images and Videos

    URLhttps://arxiv. org/abs/2408.00714. Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [24]

    Diffusion model-based video editing: A survey,

    Uriel Singer, Amit Zohar, Yuval Kirstain, Shelly Sheynin, Adam Polyak, Devi Parikh, and Yaniv Taigman. Video editing via factorized diffusion distillation. InEuropean Conference on Computer Vision, pp. 450–466. Springer, 2024a. Uriel Singer, Amit Zohar, Yuval Kirstain, Shelly Sheynin, Adam Polyak, Devi Parikh, and Yaniv Taigman. Video editing via factoriz...

    work page arXiv

  24. [25]

    Wan: Open and Advanced Large-Scale Video Generative Models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [26]

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [27]

    Zero-shot video editing using off-the-shelf image diffusion models.arXiv preprint arXiv:2303.17599, 2023a

    Wen Wang, Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chun- hua Shen. Zero-shot video editing using off-the-shelf image diffusion models.arXiv preprint arXiv:2303.17599, 2023a. Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-tex...

    work page arXiv

  27. [28]

    arXiv preprint arXiv:2310.16003 (2023)

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. InProceedings of the IEEE/CVF international conference on computer vision, pp. 7623–7633, 2023a. Jay Zhangjie Wu, Xiuyu Li, Difei Gao, ...

    work page arXiv 2023

  28. [29]

    Depth Anything V2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2.arXiv:2406.09414, 2024a. Ling Yang, Bohan Zeng, Jiaming Liu, Hong Li, Minghao Xu, Wentao Zhang, and Shuicheng Yan. Editworld: Simulating world dynamics for instruction-following image editing.arXiv preprint arXiv:2405.14785, 2024b. Zhuoyi Yang...

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [30]

    ImgEdit: A Unified Image Editing Dataset and Benchmark

    Yang Ye, Xianyi He, Zongjian Li, Bin Lin, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, and Li Yuan. Imgedit: A unified image editing dataset and benchmark.arXiv preprint arXiv:2505.20275, 2025a. Zixuan Ye, Xuanhua He, Quande Liu, Qiulin Wang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Qifeng Chen, and Wenhan Luo. Unic: Unified in-context video editing.arXiv p...

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [31]

    Anyedit: Mastering unified high-quality image editing for any idea, 2025

    Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea.arXiv preprint arXiv:2411.15738,

    work page arXiv

  31. [32]

    arXiv preprint arXiv:2412.09645 , year =

    Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, and Ziwei Liu. Evaluation agent: Efficient and promptable evaluation framework for visual generative models.arXiv preprint arXiv:2412.09645,

    work page arXiv

  32. [33]

    Knapformer: An online load balancer for efficient diffusion transformers training.arXiv preprint arXiv:2508.06001, 2025

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing.Advances in Neural Information Processing Systems, 36:31428–31449, 2023a. 15 Kai Zhang, Peng Wang, Sai Bi, Jianming Zhang, and Yuanjun Xiong. Knapformer: An online load balancer for efficient diffusion transformers training....

    work page arXiv

  33. [34]

    Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039,

    work page internal anchor Pith review Pith/arXiv arXiv

  34. [35]

    A golden retrieverwearing a red harness is walking slowly with its nose close to the dry, leaf-covered ground in a fenced yard next to a bush and a road in the background

    Bojia Zi, Penghui Ruan, Marco Chen, Xianbiao Qi, Shaozhe Hao, Shihao Zhao, Youze Huang, Bin Liang, Rong Xiao, and Kam-Fai Wong. Se\˜ norita-2m: A high-quality instruction-based dataset for general video editing by video specialists.arXiv preprint arXiv:2502.06734,

    work page arXiv

  35. [36]

    Comparison images in Figure 1 are from ImgEdit- Bench (Ye et al., 2025a)

    16 A APPENDIX A.1 IMAGE ANDVIDEOCOPYRIGHTS Figure 1 videos are frompixabay(Pixabay, 2025), stockbusters – stock.adobe.com (the first video on the top), andreybiling – stock.adobe.com (the second video on the top), and Mara Zemgaliete – stock.adobe.com (the third video on the top). Comparison images in Figure 1 are from ImgEdit- Bench (Ye et al., 2025a). E...

    work page 2025

  36. [37]

    More Examples

    and black- boxguild – stock.adobe.com (the first video in “More Examples”). Example videos in Figure 4, 6, and 8 are frompixabay(Pixabay, 2025). Adobe Stock (Adobe Inc.,

    work page 2025

  37. [38]

    videos are officially licensed from the website. A.2 EVALUATIONDETAILS Automatic Evaluation.To provide a comprehensive and robust evaluation of instruction-based video editing models on EditVerseBench, we employ a suite of six metrics spanning four aspects: overall editing quality evaluated by a Vision-Language Model (VLM), video quality, text alignment, ...

    work page 2024

  38. [39]

    Result 1

    to extract features of each frame in the edited video. The consistency score is calculated as the average cosine similarity between the features of all adjacent frames. Frame-wise DINO Consistency: To capture more fine-grained structural and textural con- sistency, we repeat the same procedure using features extracted from a pre-trained DINOv2 model (Caro...

    work page 2021

  39. [40]

    This highlights the effectiveness of our method

    The results demonstrate that EditVerse achieves highly competitive performance, surpassing a wide range of existing ap- proaches (Deng et al., 2025; Liu et al., 2025b). This highlights the effectiveness of our method. Method Add Adjust Extract Replace Remove Background Style Hybrid ActionOverall↑ MagicBrush 2.84 1.58 1.51 1.97 1.58 1.75 2.38 1.62 1.22 1.8...

    work page 2025

  40. [41]

    As shown, EditVerse achieves highly competitive performance compared with a wide range of both open-source and commercial models. Notably, 18 even though EditVerse is trained on diverse tasks beyond video generation and is built with a rela- tively small model size, it can still match or surpass the performance of several larger-scale systems. Models # Pa...

    work page 2024

  41. [42]

    Our method achieves state-of-the-art performance when compared against a wide range of both open-source and commercial systems, highlighting better semantically aligned generation

    shown in Table 8, which is designed to comprehensively assess text- to-image models across multiple aspects of visual reasoning and compositional fidelity. Our method achieves state-of-the-art performance when compared against a wide range of both open-source and commercial systems, highlighting better semantically aligned generation. Method Single Obj. T...

    work page 2024

  42. [43]

    I want to [edit prompt]. Detect the region that needs to be edited

    Noted that all V2VBench videos are square, whereas our training data does not include any square video editing samples. Our method achieves the best or competitive results across most metrics. A.4 DETAILEDTRAININGDATA Table 10 provides a detailed statistics overview of the whole training datasets that are used in our work, along with their respective rati...

    work page 2024