Pith. sign in

REVIEW 4 major objections 6 minor 50 references

SI-Edit: Toward Sketch-Instruction Guided Local Image Editing with Pixel-Level Precision

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that local image editing becomes pixel-precise only when a sketch and a text instruction are conditioned jointly, and it introduces both the dataset and the model to demonstrate it.

desk verdict A solid dataset-plus-method paper; the dataset is the real contribution, the flow-based sketch proxy and one-sided evaluation are the main soft spots. read the letter →

arxiv 2608.09097 v1 pith:G2JIOX56 submitted 2026-08-10 cs.CV

Weixin Ye , Wei Wang , Hongguang Zhu , Xuecheng Nie This is my paper · ORCID
classification cs.CV
keywords sketch-guidedimageeditinglocalgeometricdeformationdiffusionmodelsinstructionfollowingdatasetconstructionopticalflowsketchespixel-levelspatial-semanticconditioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that text-only and sketch-only editing fail for different, complementary reasons—text lacks precise spatial anchors, and sketches lack semantic intent—and that combining the two in one model can achieve pixel-level local geometric changes. To make that combination learnable, it introduces SI-Data, a 6.5K-sample dataset of (source, target, sketch, instruction) quadruplets built automatically from an MLLM-generated instruction, a generator-synthesized target, and an optical-flow-derived sketch. It then proposes SI-Edit, which fine-tunes a latent diffusion transformer with same-position encoding between the source and the sketch overlay and with a learnable task-trigger token. The paper reports that SI-Edit outperforms all compared sketch-based baselines on geometric alignment, semantic fidelity, and background preservation, with a localization error of 4.292 versus 7.434 for the best baseline. A sympathetic reader would take the central claim to be that spatial-semantic collaboration, not a single modality, is what unlocks precise local editing.

What carries the argument

The load-bearing object is the quadruplet of source image, target image, local sketch, and instruction, and the mechanism that makes it work is Same Position Encoding: the source image and the sketch-overlay image are patchified into token sequences that share identical temporal and spatial rotary positional coordinates, so the model cannot treat the sketch as an independent drifting condition. The second mechanism is the learnable task-trigger token, initialized from the mean embedding of the phrase 'follow the sketch lines' and placed at the beginning of the instruction, which switches the diffusion transformer into sketch-guided editing mode. These two devices, trained with low-rank adapters on a rectified-flow matching objective, are what carry the alignment between semantic intent and geometric constraint.

What would settle it

Take a set of source images, apply a camera-only pan or global color change with no local geometric deformation, and run the SI-Data sketch-extraction pipeline; if it yields nonzero sketches and the localized chamfer distance reports finite alignment error, the optical-flow proxy is measuring displacement rather than user intent, and both the dataset and geometric metrics need re-grounding.

Watch

Extended reading notes

Core claim

The paper's core discovery is that the bottleneck in sketch-based local editing is not generative capacity but the joint conditioning signal: when a sketch and a text instruction are aligned at the token level, the model can execute fine geometric deformations while preserving object identity and background. Technically, this is achieved by overlaying the binary sketch on the source image, giving source and overlay tokens identical 3D rotary positional coordinates, and prefixing the instruction with a learnable task-trigger token that anchors the editing mode. Evaluated on 159 test quadruplets, the method reports the lowest localized chamfer distance, the highest mask intersection-over-union, the largest improvement in CLIP-based semantic alignment, and the highest background-preservation score among the compared sketch-based baselines, and a pairwise user study confirms the trend. The same recipe is shown to transfer to human-pose movement editing using hand-drawn motion sketches.

Load-bearing premise

The load-bearing premise is that the dominant apparent displacement between source and target images, captured as optical-flow motion boundaries thresholded at 0.06, equals the geometric deformation a user would draw; if hand strokes do not coincide with those boundaries, the dataset's sketch signal and geometric metrics are mislabeled.

Editorial extensions

If this is right

  • If the central claim holds, the practical interface for local geometric editing should give users both a sketch and a short text instruction, since neither alone disambiguates the intended edit.
  • SI-Data's public quadruplet format gives other research groups a benchmark for training and comparing sketch-instruction editors, closing the data gap the paper identifies.
  • The combined set of localized chamfer distance, mask IoU, CLIP-score improvement, and background-preservation score can be reused as a standard evaluation protocol for local geometric editing, separating geometric adherence from semantic adherence and background preservation.
  • The same same-position-encoding and task-trigger recipe should transfer to other diffusion backbones and to pose or motion editing, because the mechanism is a conditioning-level device rather than a task-specific module.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors' ground-truth targets are themselves outputs of a generative model following an MLLM-written instruction, so the reported numbers certify alignment with a synthetic interpretation of intent; a human-drawn-sketch test with independently verified targets would test the claim more directly.
  • Thresholding motion-boundary gradients at 0.06 likely captures large smooth deformations better than small articulated ones, so the paper's 'pixel-level precision' is demonstrated on the dataset's deformation distribution, and an extension would build a test set of small-scale edits.
  • A directly testable consequence of the same-position-encoding mechanism is that alignment should degrade gracefully as the sketch overlay is eroded, occluded, or blurred; measuring localized chamfer distance under those perturbations would isolate whether the shared positional encoding carries the alignment.
  • The conditioning design invites an interactive editing loop, because the model accepts new sketches at inference time without retraining, so a deployable system could let users draw, generate, inspect, and redraw iteratively.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SI-Data, a dataset of 6.5K quadruplets (source image, target image, local sketch, instruction) for fine-grained geometric image editing, constructed by using Qwen3-VL to generate spatially explicit instructions, Nano Banana Pro to synthesize target images, and SEA-RAFT optical flow to extract binary sketches from motion boundaries. It also proposes SI-Edit, which fine-tunes FLUX.1 Kontext with a learnable <sk> task-trigger token and a same-position encoding (SPE) strategy, and evaluates on a 159-sample test set with four metrics: LCD, mIoU, ΔCS, and BPS. The paper reports that SI-Edit outperforms SketchEdit, MagicQuill, and FramePainter on all metrics, and includes ablations on the token configuration and qualitative ablations for SPE.

Significance. If the flow-derived sketch proxy is accepted as representative of user sketches, SI-Data is the first public quadruplet dataset for sketch-instruction guided local editing, and SI-Edit demonstrates a plausible way to combine semantic and geometric control through a learnable task token and shared positional encoding. The paper releases data and code, and the ablations on token placement and learnability support several design choices. However, the quantitative evidence for the central claim of pixel-level alignment with user intent is weakened by the issues below: the sketch proxy and the evaluation metrics are derived from the same pipeline, the baselines are not trained on the benchmark, and the reported numbers lack variance estimates.

major comments (4)
  1. [Sec. 3.2, step (4); Sec. 3.3] The binary sketch S_k is defined as optical-flow motion boundaries (Sobel gradients thresholded at 0.06), not as the target-contour strokes a user would draw for an instruction like 'bend the stem into an S-shape'. The same flow-derived sketches are then used as ground truth for LCD and mIoU in Section 3.3, and the test set of 159 samples is drawn from SI-Data. Consequently, the quantitative results in Table 2 measure alignment to flow-derived contours, not to genuinely hand-drawn input, which is the stated use case. The only hand-drawn evaluation is qualitative (Sec. E.4, Fig. 9 and Fig. 6 col. 5). This is load-bearing for the claim of pixel-level precision aligned with user intent. A concrete remedy: collect hand-drawn sketches on a subset of test images and report LCD/mIoU against those hand-drawn sketches, or compare the distribution of flow-derived sketches with hand-drawn sketches to validate the proxy.
  2. [Sec. 5.2.2, Table 2] SI-Edit is fine-tuned on SI-Data, while the three baselines (SketchEdit, MagicQuill, FramePainter) are applied zero-shot without any adaptation to the quadruplet format or fine-tuning on the training split. The claim of 'best performance among all competing approaches across all evaluation metrics' is therefore not a controlled comparison: the margins may reflect access to the SI-Data training pairs rather than the proposed architecture. The paper should either fine-tune the baselines on SI-Data (where the model formulation permits it), or evaluate SI-Edit in a zero-shot setting, and report both conditions so the dataset contribution and the method contribution can be separated.
  3. [Sec. 5.2.2, Table 2; Table 3] All metrics are reported as single point estimates on a 159-sample test set, with no error bars, confidence intervals, or significance tests. The LCD difference between SI-Edit (4.292) and MagicQuill (7.434) is large, but the ablation differences in Table 3 (e.g., ΔCS 0.0149 vs 0.0174, LCD 4.023 vs 4.292) are small and could be within random variation. Please provide standard deviations over at least three random seeds or bootstrapped 95% confidence intervals, plus a paired significance test (e.g., Wilcoxon signed-rank) for the headline metrics.
  4. [Sec. 4.2; Sec. E.2] The Same Position Encoding (SPE) strategy is a core contribution, but its ablation appears only as qualitative examples in the appendix (Fig. 10, 'w/ SPE' vs 'w/o SPE'), with no quantitative numbers on the test set. The claim that SPE 'reduces spatial drift' is not supported by the reported metrics. Please add a quantitative SPE on/off ablation reporting LCD, mIoU, ΔCS, and BPS.
minor comments (6)
  1. [Sec. 3.3, Eq. (3)] The threshold tau in the BPS definition is not specified in the main text or appendix; please give its value and a brief sensitivity check.
  2. [Sec. 5.1] The test set is described as '159 test samples randomly selected at the source-image level'; please clarify whether any source image appears in more than one test sample and whether the train/test split is at the source-image level.
  3. [Sec. 4.2 vs Table 3] The fixed-prompt baseline is described in Table 3 as initialized with 'following the sketch lines' while Section 4.2 says the learnable token is initialized with the mean of 'follow the sketch lines'; please make the phrasing consistent.
  4. [Sec. 4.2] The statement that the <sk> token is added to 'each text encoder' is vague; please specify which text encoders are actually used in FLUX.1 Kontext and whether the token is added to all of them.
  5. [Appendix A.1] Please provide training details for reproducibility: batch size, gradient accumulation steps, learning-rate schedule, effective number of training hours, and the exact version and access date of the Nano Banana Pro model.
  6. [Fig. 5 and Fig. 6] The 'Overlay Img' columns show the sketch overlaid on the source, but the raw binary sketch is never displayed separately; please label the sketch itself or include a separate panel so the reader can see the stroke constraint.

Circularity Check

0 steps flagged · score 0.0 of 10

No formal circularity: training and evaluation are separate, metrics are not the training objective, and the flow-sketch proxy is a task-definition assumption rather than a reduction of the claimed result to its inputs.

full rationale

SI-Edit's derivation chain is not circular. SI-Data is constructed by an external pipeline (Qwen3-VL instructions, Nano Banana Pro targets, SEA-RAFT optical-flow sketches), and the model is trained with a rectified-flow objective on a training split, with results reported on 159 test source images excluded at the source-image level. The evaluation metrics (LCD, mIoU, Delta CS, BPS) are applied post hoc to all methods and are not used as training losses or fitted to maximize SI-Edit's score. The flow-derived sketch is explicitly presented as a proxy for user strokes ('contour-like constraints resembling hand-drawn strokes'), and the paper supplies qualitative hand-drawn generalization evidence (Fig. 9, Sec. E.4) plus an external Motion-X subset, so the central claim does not reduce by definition to the benchmark construction. Self-citations [20, 36, 37] are general references on positional embeddings and prior work and are not load-bearing. The in-domain-versus-out-of-domain comparison and the validity of motion boundaries as a sketch proxy are legitimate methodological concerns, but they are not instances of a prediction being equivalent to its inputs by construction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The central claim rests on a chain of automated data construction choices: MLLM instruction generation, teacher-model target synthesis, and optical-flow sketch extraction. Each step is a domain assumption without independent verification. The paper contributes no new mathematical machinery, so the ledger is dominated by hand-set thresholds and model hyperparameters rather than derived constants.

free parameters (5)
  • optical flow gradient threshold = 0.06
    Used in Section 3.2 step (4) to binarize motion boundaries into sketches; chosen by hand, not justified, and directly shapes every sketch in SI-Data and the test set.
  • BPS edit/background threshold tau = unspecified
    Equation 3 defines M_edit using a threshold tau on the source-target difference, but the paper never gives its value; BPS scores depend on it.
  • LCD ROI dilation distance = unspecified
    Section 3.3 dilates the input sketch using the Euclidean Distance Transform to define the ROI; the dilation radius is not specified and affects LCD and mIoU.
  • LoRA rank and alpha = r=256, alpha=256
    Section 5.1 and A.1; capacity choice for fine-tuning, affects the model's ability to fit geometric edits.
  • sketch overlay opacity = 0.8
    Section 5.1 and A.1; sketches are integrated as 0.8-opacity overlays during training; opacity is a hand-set hyperparameter.
assumptions (6)
  • domain assumption Dominant apparent displacement between source and target arises from the intended geometric deformation.
    Section 3.2 step (4) justifies deriving sketch S_k from optical flow; if false, sketches encode global motion or artifacts instead of the edit.
  • domain assumption Nano Banana Pro produces target images that faithfully execute the MLLM-generated instruction with only the intended local deformation.
    Section 3.2 step (3); the teacher model creates ground-truth targets; errors propagate into SI-Data.
  • domain assumption Qwen3-VL generated instructions are semantically correct and spatially explicit for the source image.
    Section 3.2 step (2) and Appendix A.2; instructions are model-generated without manual verification except screening.
  • domain assumption SAM segmentation with bounding box and centroid of the sketch recovers the object contour in the generated image.
    Section 3.3; LCD and mIoU depend on SAM contour extraction; a wrong segmentation would corrupt geometric fidelity scores.
  • standard math Standard rectified flow matching objective and FLUX.1 Kontext backbone are appropriate for this task.
    Section 4.1; the paper builds on a published method, not proved here.
  • ad hoc to paper The learned <sk> embedding initialized with the mean of 'follow the sketch lines' provides a valid semantic anchor.
    Section 4.2; no independent justification beyond the paper's ablations.
invented entities (1)
  • learnable <sk> task-trigger token
    purpose: Prefixes every instruction and acts as a global semantic anchor steering the model toward sketch-guided editing (Section 4.2).
    It exists only inside the proposed model; its benefit is demonstrated in the paper's own ablations (Table 3, Figure 11), not by an external benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SI-Edit: Toward Sketch-Instruction Guided Local Image Editing with Pixel-Level Precision." pith.science (2026). https://pith.science/paper/G2JIOX56

@misc{pith2026260809097,
  author       = {Pith},
  title        = {Pith review of: SI-Edit: Toward Sketch-Instruction Guided Local Image Editing with Pixel-Level Precision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G2JIOX56}},
  note         = {Machine review of arXiv:2608.09097}
}
read the original abstract

Despite rapid advances in generative models, achieving pixel-level precision in sketch-based image editing remains a persistent challenge, particularly for fine-grained local deformations. This gap stems primarily from the critical shortage of high-quality, publicly available benchmark datasets that jointly provide geometric constraints and semantic instructions. To address this issue, we first introduce **SI-Data**, a high-quality dataset specifically designed for instruction-guided local sketch editing. We develop an automated pipeline leveraging Multimodal Large Language Models (MLLMs) to synthesize comprehensive quadruplets comprising original images, local geometric sketches, semantic instructions, and corresponding edited images. By providing both reliable spatial anchors and explicit semantic intent, SI-Data uniquely enables collaborative spatial-semantic learning. Building upon this, we propose a collaborative framework called **SI-Edit** that integrates semantic instructions with precise geometric constraints. Furthermore, to address the lack of standardized evaluation, we establish a comprehensive set of metrics designed to measure both structural fidelity (e.g., sketch-to-edge alignment) and semantic adherence. Experimental results demonstrate that SI-Edit provides more reliable structural control than baselines for sketch-based image editing, and achieves precise, pixel-level local refinements aligned with user intent. The data and code are released on the [project page](https://github.com/ywxsuperstar/SIEdit).

Figures

Figures reproduced from arXiv: 2608.09097 by the authors.

Figure 1
Figure 1. Motivation. Left (a, b): Case 1 (Linguistic Bottleneck): text-only instructions lack precise spatial anchors, resulting in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of different paradigms for image edit [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the constructed dataset. Left: distribution of edit types (inner ring) and target object categories (outer [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Overview of SI-Edit. (a) Data construction: Qwen3-VL generates an instruction from [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison of SI-Edit against baseline methods. Across [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: SI-Edit on sketch-guided motion editing. Given a source pose and a target-pose sketch, our method synthesizes an [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Some examples from our constructed dataset. For each case, we display the original image (left), the sketch condition [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results on robustness to diverse textual prompts. Columns (a)-(d) correspond to the four prompts listed in [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Generalization to hand-drawn sketches. For the same source image, each hand-drawn sketch specifies a different [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Qualitative results of the SPE strategy. Without (w/o) SPE, the model produces artifacts such as hollow boundaries in [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Qualitative results on the position of the task-trigger token [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 14
Figure 14. Figure 14: Aggregated results. The bars report the preference [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 12
Figure 12. Figure 12: User study start page [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 21 canonical work pages

  1. [1]

    Saquib Alam. 2025. OminiKontext: Multi-image references for image to image instruction models. (2025)

    work page 2025

  2. [2]

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2022. InstructPix2Pix: Learning to Follow Image Editing Instructions.arXiv preprint arXiv:2211.09800 (2022)

    arXiv 2022

  3. [3]

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging Properties in Self-Supervised Vision Transformers. In2021 IEEE/CVF International Conference on Computer Vision (ICCV). 9630–9640. doi:10.1109/ICCV48922.2021.00951

    arXiv 2021

  4. [4]

    Wengling Chen and James Hays. 2018. Sketchygan: Towards diverse and realistic sketch to image synthesis. InProceedings of the IEEE conference on computer vision and pattern recognition. 9416–9425

    work page 2018

  5. [5]

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. InProceedings of the 41st International Conference on Machine Learning...

    2024

  6. [6]

    Haoqiang Fan, Hao Su, and Leonidas Guibas. 2017. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. In2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2463–2471. doi:10.1109/CVPR.2017.264

    doi:10.1109/cvpr.2017.264 2017

  7. [7]

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan

  8. [8]

    Google. 2025. Nano Banana Pro. https://blog.google/innovation-and-ai/products/ nano-banana-pro/

    work page 2025

Show all 50 references
  1. [9]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilis- tic Models. InAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Asso- ciates, Inc., 6840–6851. https://proce...

    2020

  2. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [cs.CL] https://arxiv.org/abs/2106.09685

    2021 arXiv

  3. [11]

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. 2024. PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code.International Conference on Learning Representations (ICLR)(2024)

    2024

  4. [12]

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2023. Imagic: Text-Based Real Image Editing with Diffusion Models. InConference on Computer Vision and Pattern Recognition 2023

    2023

  5. [13]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything.arXiv:2304.02643(2023)

    2023 arXiv

  6. [14]

    Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini,...

    2025 arXiv

  7. [15]

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023. GLIGEN: Open-Set Grounded Text- to-Image Generation. arXiv:2301.07093 [cs.CV] https://arxiv.org/abs/2301.07093

    2023 arXiv

  8. [16]

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Hao- qian Wang, and Lei Zhang. 2023. Motion-X: A Large-scale 3D Ex- pressive Whole-body Human Motion Dataset. InAdvances in Neu- ral Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Har...

    2023

  9. [17]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2023. Flow Matching for Generative Modeling. arXiv:2210.02747 [cs.LG] https://arxiv.org/abs/2210.02747

    2023 arXiv

  10. [18]

    Xingchao Liu, Chengyue Gong, and Qiang Liu. 2022. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003(2022)

    2022 arXiv

  11. [19]

    Zichen Liu, Yue Yu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Wen Wang, Zhiheng Liu, Qifeng Chen, and Yujun Shen. 2025. MagicQuill: An Intelligent Interactive Image Editing System. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13072–13082

    2025

  12. [20]

    Chengxi Min, Wei Wang, and Yao Zhao. 2026. Compass-RoPE: Isotropic Ro- tary Position Embeddings for Vision Transformers. InForty-third International Conference on Machine Learning. https://openreview.net/forum?id=kbIXDF7PD0

    2026

  13. [21]

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. 2024. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. InProceedings of the AAAI conference on artificial intelligence, Vol. 38. 4296–4304

    2024

  14. [22]

    William Peebles and Saining Xie. 2023. Scalable diffusion models with transform- ers. InProceedings of the IEEE/CVF international conference on computer vision. 4195–4205

    2023

  15. [23]

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. 2023. Unicontrol: A unified diffusion model for controllable visual generation in the wild.arXiv preprint arXiv:2305.11147(2023)

    2023 arXiv

  16. [24]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.000...

    2021 arXiv

  17. [25]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.J. Mach. Learn. Res.21, 1, Article 140 (Jan. 2020), 67 pages

    2020

  18. [26]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen

  19. [27]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752 [cs.CV] https://arxiv.org/abs/2112.10752

    2022 arXiv

  20. [28]

    Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388

    2025 arXiv

  21. [29]

    Yingjie Tian, Minghao Liu, Haoran Jiang, Yunbin Tu, and Duo Su. 2025. SketchRe- finer: Text-Guided Sketch Refinement Through Latent Diffusion Models.IEEE Transactions on Visualization and Computer Graphics31, 12 (2025), 10711–10722. doi:10.1109/TVCG.2025.3613388

    2025

  22. [30]

    Unsplash. 2020. The Unsplash Dataset. https://github.com/unsplash/datasets. Accessed for research and machine learning

    2020

  23. [31]

    Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. 2023. Sketch-Guided Text- to-Image Diffusion Models. InACM SIGGRAPH 2023 Conference Proceedings(Los Angeles, CA, USA)(SIGGRAPH ’23). Association for Computing Machinery, New York, NY, USA, Article 55, 11 pages. doi:10.1145/3588...

    2023

  24. [32]

    Fleet, Radu Soricut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan

    Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Soricut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan. 2023. Imagen Editor and EditBench: Advancing and Evaluat...

    2023

  25. [33]

    Yihan Wang, Lahav Lipson, and Jia Deng. 2025. SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow. InComputer Vision – ECCV 2024, Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (Eds.). Springer Nature Switzerland, Cham, 36–54

    2025

  26. [34]

    Bovik, H.R

    Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity.IEEE Transactions on Image Processing13, 4 (2004), 600–612. doi:10.1109/TIP.2003.819861

    2004

  27. [35]

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models.arXiv preprint arxiv:2308.06721(2023)

    2023 arXiv

  28. [36]

    Weixin Ye, Wei Wang, Yahui Liu, Yue Song, Bin Ren, Wei Bi, Rita Cucchiara, and Nicu Sebe. 2025. A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models.IEEE Transactions on Pattern Analysis and Machine Intelligence 48, 2 (2025), 1873–1887. doi:10.1109/TPAMI.202...

    2025

  29. [37]

    Weixin Ye, Hongguang Zhu, Wei Wang, Yahui Liu, Mengyu Wang, and Xuecheng Nie. 2026. All-in-One Slider for Attribute Manipulation in Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 18693–18702

    2026

  30. [38]

    Yu Zeng, Zhe Lin, and Vishal M. Patel. 2021. SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches. arXiv:2111.15078 [cs.CV] https://arxiv.org/ abs/2111.15078

    2021 arXiv

  31. [39]

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. 2023. MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing. InAdvances in Neural Information Processing Systems

    2023

  32. [40]

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional con- trol to text-to-image diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    2023

  33. [41]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang

  34. [42]

    Yabo Zhang, Xinpeng Zhou, Yihan Zeng, Hang Xu, Hui Li, and Wangmeng Zuo

  35. [43]

    Bend the plant stem into a graceful curve

    Jun Zhou, Jiahao Li, Zunnan Xu, Hanhui Li, Yiji Cheng, Fa-Ting Hong, Qin Lin, Qinglin Lu, and Xiaodan Liang. 2025. FireEdit: Fine-grained Instruction-based Im- age Editing via Region-aware Vision Language Model. arXiv:2503.19839 [cs.CV] https://arxiv.org/abs/2503.19839 SI-Edit...

    2025 arXiv

  36. [48]

    building

    Only describe what you ACTUALLY see (if you see a building, say “building”, not “tree”)

  37. [49]

    the left/right/center one

    Choose ONE part. When there are MULTIPLE similar elements (e.g., several flowers, multiple trees, many rocks), specify which one using position words like: “the left/right/center one”, “the top/bottom one”, “the foreground/background one”, “the largest/smallest one”

  38. [50]

    Bend/Deform [the part] into a curve/shape/position

    Write an instruction to modify that part: - “Bend/Deform [the part] into a curve/shape/position” - “Enlarge/Expand [the part]” - “Stretch/Move [the part] horizontally/vertically/left/right/up /down” - “Add a small [element]” Output Format (JSON only): { "target_subject": "the ...

    2026

  39. [2018]

    InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  40. [2022]

    arXiv:2204.06125 [cs.CV] https://arxiv.org/abs/2204.06125

    Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv:2204.06125 [cs.CV] https://arxiv.org/abs/2204.06125

    arXiv

  41. [2024]

    InInternational Conference on Learning Representations (ICLR)

    Guiding Instruction-based Image Editing via Multimodal Large Language Models. InInternational Conference on Learning Representations (ICLR)

  42. [2025]

    InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 18121–18131. MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Weixin Ye, Wei Wang, Hongguang Zhu, and Xuecheng Nie

    2026

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.