Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An MLLM agent can synthesize 2D, 3D, and 4D world data that, when used to fine-tune existing generative models, improves their quality and camera-accuracy metrics.

desk verdict Plausible MLLM-driven data synthesis pipeline with an honest limitations section, but the central 'significant boosts' claim is under-evidenced: 2D/3D numbers are missing and the multi-view optimizer is its own judge. read the letter →

arxiv 2508.05580 v1 pith:ZKCLXZ3D submitted 2025-08-07 cs.CV

Kunyu Feng , Yue Ma , Xinhua Zhang , Boshi Liu , Yikuang Yuluo , Yinhan Zhang , Runtao Liu , Hongyu Liu
show 4 more authors
Zhiyuan Qin Shanhui Mo Qifeng Chen Zeyu Wang
This is my paper
classification cs.CV
keywords MLLMagentsyntheticdata3Dscenegeneration4Dvideomulti-viewoptimizationvision-languagemodelfeedbackengineAIGC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Follow-Your-Instruction is a pipeline that turns text, images, or reference objects into rendered 2D, 3D, and 4D training data without manual scene construction. An MLLM collects assets and builds a 3D layout; a VLM checks the layout from multiple viewpoints and nudges object positions until the average confidence score passes a threshold; an MLLM planner then produces temporally coherent video frames. The paper's central claim is that this synthetic data transfers: fine-tuning three existing baselines (RoRem for 2D object removal, MV-Adapter for 3D reconstruction, ReCamMaster for 4D camera-controlled video) improves their task metrics, e.g. ReCamMaster FID 62.48 to 60.32 and rotation error 1.45 to 1.35. If true, the same pipeline can generate task-specific supervision, such as masks, depth, and camera trajectories, that generic web datasets lack, at much lower cost than manual scene design. This matters because data, not architecture, is often the bottleneck in specialized AIGC tasks.

What carries the argument

The load-bearing mechanism is the multi-view VLM feedback loop: render the current layout from $n$ views, ask a VLM whether each spatial relation holds, average the confidence scores into $S_{VLM}=\frac{1}{n}\sum_i s_i$, and if the score is below threshold $t$, ask the MLLM-Locator to move the offending object. This score is simultaneously the optimizer's objective and the paper's own measure of optimization success. Supporting machinery: MLLM-Collector maps multimodal input to an asset list; MLLM-Generator creates bounding boxes and the global 3D layout; MLLM-Planner converts scene understanding into executable frame-by-frame actions, with a VLM-guided frame prediction module that detects a

What would settle it

Replace the VLM grader with ground-truth geometric contact checks, such as rendered depth or ray casts, on the same scenes and then fine-tune the same baselines: if the VLM's success rate stays near 0.9987 while geometric placement errors do not fall, or if the fine-tuning gains vanish when VLM-judged scenes are filtered by geometry, the optimization loop was rewarding 2D plausibility rather than 3D correctness.

Watch

Extended reading notes

Core claim

The paper introduces Follow-Your-Instruction, a single MLLM-agent pipeline that takes text, images, or reference objects, retrieves matching assets, builds a 3D layout, and refines the scene through multi-view VLM feedback before planning temporally coherent frames. Its central claim is that the resulting synthetic data transfers to real generative models: fine-tuning the 2D object-removal model RoRem, the 3D reconstruction model MV-Adapter, and the 4D camera-controlled generator ReCamMaster on this data improves their results. For ReCamMaster, the reported gains are consistent across FID (62.48 to 60.32), FVD (160.72 to 155.71), rotation error (1.45 to 1.35), translation error (5.22 to 4.69

Load-bearing premise

The pipeline assumes the vision-language model's multi-view confidence score is a truthful measure of physical correctness, so an object that 'looks placed' from two views is treated as correctly placed; the reported success rate is measured by the same VLM that does the optimizing.

Editorial extensions

If this is right

  • Fine-tuning on Follow-Your-Instruction data improves 2D object removal, 3D reconstruction, and 4D camera-controlled video generation, meaning task-specific synthetic data can substitute for manually curated datasets in these settings.
  • With two views, multi-view optimization reaches essentially the same success rate as three views (0.9987 vs 0.9994) at lower time cost, so cheap multi-view verification is enough for the layout loop.
  • VLM-guided frame prediction raises temporal consistency from 0.6524 to 0.9128 and CLIP similarity from 50.76 to 68.75, indicating that MLLM planners need a visual critic to stay temporally coherent.
  • The quality of synthetic scenes tracks the underlying MLLM's strength: GPT-4o leads on all scene metrics, so improvements in general MLLMs should directly improve the synthesized data.
  • The pipeline outputs annotations (masks, depth, camera trajectories) alongside the rendered scenes, which is what makes it usable for fine-grained downstream tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the VLM confidence score is calibrated against ground truth, this multi-view loop is effectively an automated scene validator and could be reused as a general consistency check in other synthetic-data pipelines, not just as an optimizer.
  • The transfer claim would be stronger with a held-out real-world benchmark; because the paper's Limitations section says real-world generalization is not validated, the reported boost currently means gains on the three chosen baselines and their evaluation suites.
  • Because the same VLM grades the optimizer, the reported success rate of 0.9987 may be optimistic; a simple test is to swap in a different judge VLM or human raters and see whether the success rate and downstream gains survive.
  • A natural extension is to turn the discrete reposition-and-recheck loop into a differentiable objective: if the VLM score can be treated as a loss, object placements could be refined by gradient descent rather than by querying the MLLM-Locator.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Follow-Your-Instruction, an MLLM-driven pipeline that synthesizes 2D, 3D, and 4D training data from multimodal inputs (text, images, actions). The pipeline consists of four modules: MLLM-Collector for asset retrieval, MLLM-Generator for 3D layout construction, MLLM-Optimizer for multi-view VLM-based scene refinement, and MLLM-Planner for temporal video planning. The authors evaluate the quality of generated scenes across several proprietary and open-source MLLMs, then fine-tune three downstream baselines (RoRem for 2D object removal, MV-Adapter for 3D reconstruction, ReCamMaster for 4D video generation) on the synthetic data. The central claim is that this synthetic data significantly boosts downstream model performance. The paper includes qualitative comparisons and one quantitative 4D table, with additional 2D/3D results deferred to a supplementary file that is not present in the arXiv submission.

Significance. If fully substantiated, the framework would be a meaningful step toward scalable, automatic generation of task-specific training data across 2D, 3D, and 4D generative tasks, reducing reliance on manually curated real-world datasets. The multi-view optimization idea is timely, and the external downstream evaluation (ReCamMaster, MV-Adapter, RoRem) provides a useful grounding that goes beyond self-reported synthetic-data quality. However, the evidence as submitted is incomplete: the 2D/3D quantitative results are absent, the one quantitative table has no error bars or significance testing, and the main optimization metric is self-referential. These gaps currently prevent verification of the paper's headline claim, though they are addressable in a revision.

major comments (4)
  1. [Sec. 4.2, Quantitative Results] The abstract and Section 4.2 claim that synthetic data 'significantly boosts' performance in 2D, 3D, and 4D tasks, but the only quantitative evidence in the manuscript is Table 2 for the 4D ReCamMaster task. The 2D object removal and 3D reconstruction results are explicitly deferred to 'the supplementary material,' which is not included in the arXiv submission. This is load-bearing: without those numbers the central cross-domain claim is not verifiable. Either include the 2D/3D tables and their evaluation protocols in the main text/appendix, or revise the claim to be specific to the 4D setting.
  2. [Table 2, Sec. 4.2] Table 2 reports modest improvements (e.g., FID 62.48→60.32, FVD 160.72→155.71, RotErr 1.45→1.35, TransErr 5.22→4.69) with no error bars, no number of evaluation trials, and no significance tests. The word 'significantly' in the abstract and the phrase 'can be improved' in Sec. 4.2 are therefore asserted rather than demonstrated. Please report standard deviations across runs or bootstrap confidence intervals, run paired significance tests (e.g., paired bootstrap or Wilcoxon) on the relevant metrics, and state how many videos/clips were evaluated.
  3. [Eq. (7), Sec. 3.2, Table 3] The multi-view optimization is driven by a VLM confidence score S_VLM (Eq. 7), and the ablation in Table 3 reports a 'Success Rate' that appears to be the fraction of scenes for which this same VLM score exceeds threshold t. This creates a circularity: the optimizer moves objects to maximize the VLM's verdict, and the reported success rate is then the VLM agreeing with itself. There is no independent validation of the VLM judge (e.g., against human annotations, geometric collision checks, or physical plausibility metrics). The paper should validate Eq. (7) as a proxy for spatial/physical correctness, for instance by comparing VLM confidence against ground-truth object contacts or human ratings on a held-out set, and report the correlation or agreement rate.
  4. [Sec. 4.3, Table 3] Table 3 omits the threshold t used in Eq. (7), and the success-rate values (0.2415 for 1 view, 0.9987 for 2 views) are presented without explaining how the binary success is derived from a continuous score, how many scenes were tested, or which MLLM/VLM provided the judgments. Moreover, CLIP similarity jumps from 53.24 to 68.75 between 1 and 2 views; such a large jump warrants a concrete explanation (e.g., is it the same metric as in Table 1, and on how many generated scenes?). Without these details, the choice of n=2 views as 'optimal' is not reproducible.
minor comments (4)
  1. [Throughout] There are multiple typos: 'Follow-Y our-Instruction' in the title/header, 'Univerisity' in the affiliation, and 'Noteably' in Sec. 4.1. Please copyedit.
  2. [Sec. 4.1, Table 1] Table 1 reports results on '50 videos' but gives no information about prompt diversity, asset distribution, or the number of scenes per MLLM. Also, the 'user study is provided in the supplementary materials' is mentioned but no supplementary is included.
  3. [Sec. 3.1, Eq. (1)] Eq. (1) writes '(d1, A1), ...,(dk, Ak) ← MLLM-Collector(I)' but the notation for the asset list and the descriptions is not clearly defined in the text. Please define d_i, A_i, and k explicitly, and clarify how visual inputs bypass the retrieval process.
  4. [Sec. 4.2, Baselines] The fine-tuning protocols for RoRem, MV-Adapter, and ReCamMaster are not described (learning rates, number of steps, data amount, compute). Without these, the downstream results are hard to reproduce.

Circularity Check

1 steps flagged · score 3.0 of 10

Central claim is grounded in external downstream benchmarks; localized self-referential success metric in multi-view optimization ablation.

  1. self definitional [Sec. 3.2 Multi-View Optimization, Eq. (7); Sec. 4.3 Tab. 3]
    "our agent renders the current scene L from multiple views N = {v1, v2, . . . , vn}, and interacts with the VLM (e.g. "Is the pink cup placed on the table?") to verify the spatial relations for each view: SVLM = 1/n ... if the scores exceed the threshold t, our agent determines that the current scene optimization is successful."

    The optimizer's objective is exactly SVLM: it iteratively relocates objects until the VLM's confidence score exceeds threshold t. Table 3's Success Rate is then the fraction of scenes for which this same VLM score exceeds t. The ablation therefore reports how often the method satisfies its own optimization target, not an independent check of geometric/physical correctness. Improving this rate is partly forced: the optimization loop is directly maximizing the quantity being measured. The paper provides no human or geometric validation of the VLM verdicts, so the success-rate improvement from 1 to 2 views is self-referential. This does not infect the central downstream claim, which is measured on external benchmarks (ReCamMaster in Table 2; RoRem/MV-Adapter in supplementary), but it is a gen

full rationale

The paper's central claim—that synthetic data boosts existing baselines—is evaluated by fine-tuning external models (RoRem, MV-Adapter, ReCamMaster) and measuring standard metrics (FID, FVD, CLIP-T/F, rotation/translation error, view synchronization). Those evaluations are independent of the method's own MLLM components, so the main derivation chain is not circular. The one load-bearing circular step is the multi-view optimization success metric: Eq. (7) defines scene success as the VLM's average confidence, the MLLM-Optimizer optimizes exactly that score, and Table 3 reports 'Success Rate' as the fraction of scenes passing that same threshold. Thus the ablation's success-rate numbers are self-referential. I also note the limitations the paper itself flags: item (2) concedes no validation on other real-world benchmarks, and Section 4.2 defers all 2D and 3D quantitative results to a supplementary file absent from the arXiv version. These are evidentiary gaps, not circular reductions; they do not raise the circularity score further. No fitted-parameter-as-prediction, no load-bearing self-citation chain, and no uniqueness theorem imported from the authors' prior work were found.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework rests on several external resources and modeling assumptions: a pre-existing asset library with sufficient coverage, VLM judgments as proxies for physical correctness, and reliable MLLM output for bounding boxes. The number of optimization views was tuned to the success-rate metric, and the VLM success threshold is unspecified.

free parameters (4)
  • VLM success threshold t = not reported
    Equation (7) uses a confidence threshold t to decide whether scene optimization succeeds; the value is not stated and is a hand-set design choice that directly determines reported success rates.
  • Number of optimization views n = 2
    Chosen via ablation in Table 3 as the best efficiency/accuracy trade-off; it directly controls the reported success-rate jump.
  • Asset retrieval top-k = not reported
    Section 3.1 applies a top-k retrieval strategy over the asset repository; k is not specified and affects which assets compose each scene.
  • MLLM prompt templates = not disclosed
    Prompt templates for MLLM-Collector, Generator, Optimizer, and Planner are not provided; results are sensitive to their wording.
assumptions (5)
  • domain assumption Pre-existing asset libraries contain assets semantically matching arbitrary user inputs, and MLLM retrieval can select them reliably.
    The pipeline depends on the asset repository; the authors acknowledge in Limitations that scalability is constrained by reliance on pre-existing asset libraries.
  • domain assumption VLM multi-view confidence scores (Eq. 7) are a valid proxy for physical/spatial correctness of the scene.
    The optimizer uses VLM feedback as ground truth; no human verification or external geometric check is used for the success-rate metric in Section 4.3.
  • domain assumption MLLMs can reliably output 3D bounding boxes and placements consistent with semantics and physical plausibility.
    MLLM-Creator and MLLM-Locator outputs are taken as correct; the paper does not measure their failure rate independently.
  • standard math The pinhole camera projection model with calibrated intrinsic and extrinsic matrices K and E accurately renders the scene for downstream training.
    Equation (6) assumes a calibrated pinhole model; this is standard but unstated in terms of calibration accuracy.
  • domain assumption Rendering the composed 3D scene produces realistic enough images/videos for fine-tuning downstream generative models.
    The entire approach rests on synthetic renderings transferring to real-world downstream tasks; the authors note lack of validation on real-world benchmarks as a limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis." pith.science (2026). https://pith.science/paper/ZKCLXZ3D

@misc{pith2026250805580,
  author       = {Pith},
  title        = {Pith review of: Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKCLXZ3D}},
  note         = {Machine review of arXiv:2508.05580}
}
read the original abstract

With the growing demands of AI-generated content (AIGC), the need for high-quality, diverse, and scalable data has become increasingly crucial. However, collecting large-scale real-world data remains costly and time-consuming, hindering the development of downstream applications. While some works attempt to collect task-specific data via a rendering process, most approaches still rely on manual scene construction, limiting their scalability and accuracy. To address these challenges, we propose Follow-Your-Instruction, a Multimodal Large Language Model (MLLM)-driven framework for automatically synthesizing high-quality 2D, 3D, and 4D data. Our \textbf{Follow-Your-Instruction} first collects assets and their associated descriptions through multimodal inputs using the MLLM-Collector. Then it constructs 3D layouts, and leverages Vision-Language Models (VLMs) for semantic refinement through multi-view scenes with the MLLM-Generator and MLLM-Optimizer, respectively. Finally, it uses MLLM-Planner to generate temporally coherent future frames. We evaluate the quality of the generated data through comprehensive experiments on the 2D, 3D, and 4D generative tasks. The results show that our synthetic data significantly boosts the performance of existing baseline models, demonstrating Follow-Your-Instruction's potential as a scalable and effective data engine for generative intelligence.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    ProtoAda uses format-aware prototypes for better task routing and geometry-aware consolidation to reduce interference in multimodal continual instruction tuning.

  2. SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    SAME stabilizes MoE-based continual instruction tuning of vision-language models by constraining router updates to task-relevant subspaces and scaling expert updates with historical input covariance, achieving 66.82% ...

  3. CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    CRAM uses adaptive MoE with centroid routing and orthogonality constraints to enable parameter-efficient multimodal continual instruction tuning while mitigating forgetting.

Reference graph

Works this paper leans on

71 extracted references · 46 canonical work pages · cited by 3 Pith papers

  1. [1]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. 1, 5, 6

    work page 2024

  2. [2]

    Claude 4 sonnet, 2025

    Anthropic. Claude 4 sonnet, 2025. 5, 6

    work page 2025

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 5, 6

    arXiv 2023

  4. [4]

    Recammaster: Camera-controlled generative rendering from a single video

    Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lian- rui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from a single video. arXiv preprint arXiv:2503.11647, 2025. 3, 6, 7

    arXiv 2025

  5. [5]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 5, 6

    arXiv 2025

  6. [6]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021. 1

    work page 2021

  7. [7]

    Restoreagent: Autonomous image restoration agent via multimodal large language models

    Haoyu Chen, Wenbo Li, Jinjin Gu, Jingjing Ren, Sixi- ang Chen, Tian Ye, Renjing Pei, Kaiwen Zhou, Fenglong Song, and Lei Zhu. Restoreagent: Autonomous image restoration agent via multimodal large language models. Advances in Neural Information Processing Systems , 37: 110643–110666, 2024. 2

    work page 2024

  8. [8]

    Follow-your-canvas: Higher-resolution video outpainting with extensive content generation.arXiv preprint arXiv:2409.01055, 2024

    Qihua Chen, Yue Ma, Hongfa Wang, Junkun Yuan, Wenzhe Zhao, Qi Tian, Hongmei Wang, Shaobo Min, Qifeng Chen, and Wei Liu. Follow-your-canvas: Higher-resolution video outpainting with extensive content generation.arXiv preprint arXiv:2409.01055, 2024. 1

    arXiv 2024

Show all 71 references
  1. [9]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 5, 6

    2024 arXiv

  2. [10]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capab...

    2025 arXiv

  3. [11]

    Introducing gemini 2.0: our new ai model for the agentic era, 2024

    Google DeepMind. Introducing gemini 2.0: our new ai model for the agentic era, 2024. 5, 6

    2024

  4. [12]

    Dit4edit: Dif- fusion transformer for image editing

    Kunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi, Haozhe Chen, Qifeng Chen, and Zeyu Wang. Dit4edit: Dif- fusion transformer for image editing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2969– 2977, 2025. 1

    2025

  5. [13]

    Blender Foundation. Blender. https://www.blender.org/,

  6. [14]

    Text2world: Benchmarking large language models for symbolic world model generation

    Mengkang Hu, Tianxing Chen, Yude Zou, Yuheng Lei, Qiguang Chen, Ming Li, Qiwei Liang, Yao Mu, Hongyuan Zhang, Wenqi Shao, et al. Text2world: Benchmarking large language models for symbolic world model generation. In ICLR 2025 Workshop on World Models: Understanding, Modelling ...

    2025

  7. [15]

    Scenecraft: An LLM agent for synthesizing 3d scenes as blender code

    Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An LLM agent for synthesizing 3d scenes as blender code. In Forty-first International Conference on Ma- chine Learning, 2024. 3, 4

    2024

  8. [16]

    Mv-adapter: Multi-view consistent image generation made easy

    Zehuan Huang, Yuan-Chen Guo, Haoran Wang, Ran Yi, Lizhuang Ma, Yan-Pei Cao, and Lu Sheng. Mv-adapter: Multi-view consistent image generation made easy. arXiv preprint arXiv:2412.03632, 2024. 3, 6

    2024 arXiv

  9. [17]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  10. [18]

    Smarteraser: Remove anything from images using masked-region guidance

    Longtao Jiang, Zhendong Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Lei Shi, Dong Chen, and Houqiang Li. Smarteraser: Remove anything from images using masked-region guidance. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 24452– 24462, 2025. 3

    2025

  11. [19]

    Black Forest Labs. Flux. https://github.com/black-forest- labs/flux, 2024. 1

    2024

  12. [20]

    Recap: Better gaussian relighting with cross-environment captures

    Jingzhi Li, Zongwei Wu, Eduard Zamfir, and Radu Timofte. Recap: Better gaussian relighting with cross-environment captures. In Proceedings of the Computer Vision and Pat- tern Recognition Conference, pages 21307–21316, 2025. 3

    2025

  13. [21]

    Rorem: Training a robust object remover with human-in-the-loop

    Ruibin Li, Tao Yang, Song Guo, and Lei Zhang. Rorem: Training a robust object remover with human-in-the-loop. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14024–14035, 2025. 6

    2025

  14. [22]

    Freehand sketch generation from mechanical components

    Zhichao Liao, Fengyuan Piao, Di Huang, Xinghui Li, Yue Ma, Pingfa Feng, Heming Fang, and Long Zeng. Freehand sketch generation from mechanical components. InProceed- ings of the 32nd ACM international conference on multime- dia, pages 6755–6764, 2024. 1

    2024

  15. [23]

    Evaluating text-to-image diffu- sion models for texturing synthetic data

    Thomas Lips et al. Evaluating text-to-image diffu- sion models for texturing synthetic data. arXiv preprint arXiv:2411.10164, 2024. 1

    2024 arXiv

  16. [24]

    Video-p2p: Video editing with cross-attention control

    Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8599–8608, 2024. 1

    2024

  17. [25]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representa- tions, . 1

  18. [26]

    Visualagentbench: Towards large multi- modal models as visual foundation agents

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Song XiXuan, Yifan Xu, Shudan Zhang, Hanyu Lai, Jiadai Sun, Xinyue Yang, et al. Visualagentbench: Towards large multi- modal models as visual foundation agents. In The Thirteenth International Conference on Learning Representations, . 3

  19. [27]

    Ovis: Structural embed- ding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embed- ding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024. 5, 6

    2024 arXiv

  20. [28]

    Towards effective and sparse adversarial attack on spiking neural networks via breaking invisible surrogate gradients

    Li Lun, Kunyu Feng, Qinglong Ni, Ling Liang, Yuan Wang, Ying Li, Dunshan Yu, and Xiaoxin Cui. Towards effective and sparse adversarial attack on spiking neural networks via breaking invisible surrogate gradients. In Proceedings of the Computer Vision and Pattern Recognition Co...

    2025

  21. [29]

    Visual knowledge graph for human action rea- soning in videos

    Yue Ma, Yali Wang, Yue Wu, Ziyu Lyu, Siran Chen, Xiu Li, and Yu Qiao. Visual knowledge graph for human action rea- soning in videos. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia , pages 4132–4141, 2022. 1

    2022

  22. [30]

    Follow your pose: Pose- guided text-to-video generation using pose-free videos

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose- guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 4117–4125, 2024. 1

    2024

  23. [31]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. In SIGGRAPH Asia 2024 Conference Papers, pages 1–12, 2024. 1

    2024

  24. [32]

    Magic- stick: Controllable video editing via control handle transfor- mations

    Yue Ma, Xiaodong Cun, Sen Liang, Jinbo Xing, Yingqing He, Chenyang Qi, Siran Chen, and Qifeng Chen. Magic- stick: Controllable video editing via control handle transfor- mations. In 2025 IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pages 9385–9395. IEEE,

    2025

  25. [33]

    Controllable video generation: A survey

    Yue Ma, Kunyu Feng, Zhongyuan Hu, Xinyu Wang, Yucheng Wang, Mingzhe Zheng, Xuanhua He, Chenyang Zhu, Hongyu Liu, Yingqing He, et al. Controllable video generation: A survey. arXiv preprint arXiv:2507.16869 ,

  26. [34]

    Follow-your-creation: Empowering 4d creation through video inpainting

    Yue Ma, Kunyu Feng, Xinhua Zhang, Hongyu Liu, David Junhao Zhang, Jinbo Xing, Yinhan Zhang, Ayden Yang, Zeyu Wang, and Qifeng Chen. Follow-your-creation: Empowering 4d creation through video inpainting. arXiv preprint arXiv:2506.04590, 2025. 3

    2025 arXiv

  27. [35]

    Follow-your-click: Open-domain regional image animation via motion prompts

    Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Leqi Shen, Chenyang Qi, Jixuan Ying, Chengfei Cai, Zhifeng Li, Heung-Yeung Shum, et al. Follow-your-click: Open-domain regional image animation via motion prompts. In Proceed- ings of the AAAI Conference on Artificial Intelligence...

    2025

  28. [36]

    Follow-your-motion: Video motion transfer via efficient spatial-temporal decoupled finetuning

    Yue Ma, Yulong Liu, Qiyuan Zhu, Ayden Yang, Kunyu Feng, Xinhua Zhang, Zhifeng Li, Sirui Han, Chenyang Qi, and Qifeng Chen. Follow-your-motion: Video motion transfer via efficient spatial-temporal decoupled finetuning. arXiv preprint arXiv:2506.05207, 2025. 1

    2025

  29. [37]

    Lux post facto: Learning portrait performance relighting with conditional video diffusion and a hybrid dataset

    Yiqun Mei, Mingming He, Li Ma, Julien Philip, Wenqi Xian, David M George, Xueming Yu, Gabriel Dedic, Ahmet Lev- ent Tas ¸el, Ning Yu, et al. Lux post facto: Learning portrait performance relighting with conditional video diffusion and a hybrid dataset. In Proceedings of the Co...

    2025

  30. [38]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024

    Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024. 5, 6

    2024

  31. [39]

    A survey of synthetic data augmentation methods in machine vision

    Alhassan Mumuni, Fuseini Mumuni, and Nana Kobina Ger- rar. A survey of synthetic data augmentation methods in machine vision. Machine Intelligence Research, 21(5):831– 869, 2024. 1

    2024

  32. [40]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. 1, 3, 5, 6

    2024

  33. [41]

    Gpt-4o mini: advancing cost-efficient intelligence,

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence,

  34. [42]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,

  35. [43]

    Tokenflow: Unified image tokenizer for multi- modal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xin- glong Wu. Tokenflow: Unified image tokenizer for multi- modal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pa...

    2025

  36. [44]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  37. [45]

    Towards realistic scene generation with lidar diffusion models

    Haoxi Ran, Vitor Guizilini, and Yue Wang. Towards realistic scene generation with lidar diffusion models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14738–14748, 2024. 3

    2024

  38. [46]

    Syn- thetic data generation for bridging sim2real gap in a produc- tion environment

    Parth Rawal, Mrunal Sompura, and Wolfgang Hintze. Syn- thetic data generation for bridging sim2real gap in a produc- tion environment. arXiv preprint arXiv:2311.11039 , 2023. 1

    2023 arXiv

  39. [47]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  40. [48]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 1

    2021 arXiv

  41. [49]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- f...

    2022

  42. [50]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Ta- tiana Matejovicova, Alexandre Ram ´e, Morgane Rivi `ere, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. 5, 6

    2025 arXiv

  43. [51]

    Wan: Open and advanced large-scale video generative models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. 1

    2025 arXiv

  44. [52]

    Taming rectified flow for inversion and editing

    Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, and Ying Shan. Taming rectified flow for inversion and editing. In Forty-second In- ternational Conference on Machine Learning. 1

  45. [53]

    Cove: Unleashing the diffusion fea- ture correspondence for consistent video editing

    Jiangshan Wang, Yue Ma, Jiayi Guo, Yicheng Xiao, Gao Huang, and Xiu Li. Cove: Unleashing the diffusion fea- ture correspondence for consistent video editing. Advances in Neural Information Processing Systems, 37:96541–96565,

  46. [54]

    Clarity chatgpt: An interactive and adaptive processing sys- tem for image restoration and enhancement

    Yanyan Wei, Zhao Zhang, Jiahuan Ren, Xiaogang Xu, Richang Hong, Yi Yang, Shuicheng Yan, and Meng Wang. Clarity chatgpt: An interactive and adaptive processing sys- tem for image restoration and enhancement. arXiv preprint arXiv:2311.11695, 2023. 2

    2023 arXiv

  47. [55]

    Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence

    Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence. arXiv preprint arXiv:2505.23747, 2025. 3

    2025 arXiv

  48. [56]

    Bridging the gap: A unified video comprehension framework for mo- ment retrieval and highlight detection

    Yicheng Xiao, Zhuoyan Luo, Yong Liu, Yue Ma, Heng- wei Bian, Yatai Ji, Yujiu Yang, and Xiu Li. Bridging the gap: A unified video comprehension framework for mo- ment retrieval and highlight detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern rec...

    2024

  49. [57]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In The Thirteenth International Conference on ...

  50. [58]

    Towards multiple char- acter image animation through enhancing implicit decou- pling

    Jingyun Xue, Hongfa Wang, Qi Tian, Yue Ma, Andong Wang, Zhiyuan Zhao, Shaobo Min, Wenzhe Zhao, Kaihao Zhang, Heung-Yeung Shum, et al. Towards multiple char- acter image animation through enhancing implicit decou- pling. In The Thirteenth International Conference on Learn- ing ...

  51. [59]

    Follow-your-pose v2: Multiple-condition guided character image animation for stable pose control

    Jingyun Xue, Hongfa Wang, Qi Tian, Yue Ma, Andong Wang, Zhiyuan Zhao, Shaobo Min, Wenzhe Zhao, Kai- hao Zhang, Heung-Yeung Shum, et al. Follow-your-pose v2: Multiple-condition guided character image animation for stable pose control. arXiv e-prints, pages arXiv–2406, 2024. 1

    2024

  52. [60]

    Eedit: Rethinking the spatial and temporal redundancy for efficient image editing

    Zexuan Yan, Yue Ma, Chang Zou, Wenteng Chen, Qifeng Chen, and Linfeng Zhang. Eedit: Rethinking the spatial and temporal redundancy for efficient image editing. arXiv preprint arXiv:2503.10270, 2025. 1

    2025

  53. [61]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jia- long Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, J...

    2025

  54. [62]

    Thinking in space: How mul- timodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025. 3

    2025

  55. [63]

    Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents

    Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. In Forty-second In...

  56. [64]

    Exploring limits of diffusion- synthetic training with weakly supervised semantic segmen- tation

    Ryota Yoshihashi, Yuya Otsuka, Tomohiro Tanaka, Hi- rokatsu Kataoka, et al. Exploring limits of diffusion- synthetic training with weakly supervised semantic segmen- tation. In Proceedings of the Asian Conference on Computer Vision, pages 2300–2318, 2024. 1

    2024

  57. [65]

    Craft- ing a toolchain for image restoration by deep reinforce- ment learning

    Ke Yu, Chao Dong, Liang Lin, and Chen Change Loy. Craft- ing a toolchain for image restoration by deep reinforce- ment learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2443–2452,

  58. [66]

    Tra- jectorycrafter: Redirecting camera trajectory for monoc- ular videos via diffusion models

    Mark YU, Wenbo Hu, Jinbo Xing, and Ying Shan. Tra- jectorycrafter: Redirecting camera trajectory for monoc- ular videos via diffusion models. arXiv preprint arXiv:2503.05638, 2025. 3

    2025 arXiv

  59. [67]

    Mllm as retriever: Interactively learning multimodal re- trieval for embodied agents

    Junpeng Yue, Xinrun Xu, B ¨orje F Karlsson, and Zongqing Lu. Mllm as retriever: Interactively learning multimodal re- trieval for embodied agents. In The Thirteenth International Conference on Learning Representations. 1

  60. [68]

    Magiccolor: Multi-instance sketch colorization

    Yinhan Zhang, Yue Ma, Bingyuan Wang, Qifeng Chen, and Zeyu Wang. Magiccolor: Multi-instance sketch colorization. arXiv preprint arXiv:2503.16948, 2025. 1

    2025 arXiv

  61. [69]

    Instantswap: Fast customized concept swapping across sharp shape differ- ences

    Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, and Xiu Li. Instantswap: Fast customized concept swapping across sharp shape differ- ences. In The Thirteenth International Conference on Learn- ing Representations. 1

  62. [70]

    Multibooth: Towards generating all your concepts in an im- age from text

    Chenyang Zhu, Kai Li, Yue Ma, Chunming He, and Xiu Li. Multibooth: Towards generating all your concepts in an im- age from text. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10923–10931, 2025. 1

    2025

  63. [71]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 5, 6

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.