Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

RePlan shows that a reasoning-driven region planner plus training-free attention masking lets open-source diffusion editors handle complex, fine-grained image edits with far fewer artifacts and better grounding.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 15:23 UTC pith:ZFN2O7BM

load-bearing objection Clever training-free attention injection and a new benchmark, but the Weighted metric in Table 1 is arithmetically impossible and the evaluation is too self-referential to support the headline claim. the 4 major comments →

arxiv 2512.16864 v2 pith:ZFN2O7BM submitted 2025-12-18 cs.CV

RePlan: Reasoning-guided Region Planning for Complex Instruction-based Image Editing

classification cs.CV
keywords instruction-based image editingregion planningattention maskingmultimodal diffusion transformerGRPO reinforcement learninginstruction-visual complexityimage editing benchmarkgrounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

RePlan argues that state-of-the-art instruction-based editors fail on cluttered scenes because vision-language models serve only as coarse semantic encoders. The paper's central claim is that making the VLM output explicit region-aligned guidance — bounding boxes plus per-region edit hints — and then constraining the diffusion model's attention so each region only attends to its own hint yields substantially more precise edits. With just about 1,000 instruction-only examples used to reinforce the planner via GRPO, RePlan reports consistent gains over open-source baselines trained on orders of magnitude more data, and closes much of the gap with proprietary editors on the new IV-Edit benchmark. The paper also introduces IV-Complexity as a challenge axis and IV-Edit as a curated testbed for it.

Core claim

On the paper's own terms, the discovery is that a reasoning-driven planner plus attention masking is enough to unlock precise multi-region editing in a pretrained multimodal diffusion transformer. The VLM planner (Qwen2.5-VL 7B, reinforced with GRPO) decomposes an instruction into structured reasoning and a set of bounding-box/hint pairs, including negative hints that tell regions to stay unchanged. The editor then applies a training-free binary attention mask with five rules: text hints are isolated from each other, each image region attends only to its own hint and the global hint, background tokens attend only to the global hint, and image–latent tokens remain fully connected. This keeps

What carries the argument

The attention region injection mask is the central mechanism: a binary mask over the token sequence of an MMDiT (a diffusion transformer that jointly attends to text, image, and noise-latent tokens). The mask enforces five rules — intra-group connection, hint isolation, full image–latent interaction, region constraint (region tokens see only their own hint and the global hint), and background constraint (background tokens see only the global hint). It is training-free and applied at every attention layer, and it is what lets the model execute the planner's region-aligned guidance precisely.

Load-bearing premise

The evaluation premise: Gemini-2.5-Pro's five-point ratings on IV-Edit are a valid, bias-free measure, even though the benchmark's instructions and filters were generated with the same model family and the consistency dimension favors edits that leave regions untouched.

What would settle it

A human-rated study on a random subset of IV-Edit comparing RePlan against Flux.1 Kontext dev on the target and effect dimensions; if humans do not rate RePlan higher on those dimensions, the central claim weakens. A sharper ablation: feed the same region hints to Flux as a global prompt without attention injection — if scores are unchanged, the attention mask is not the cause of the gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Open-source MMDiT-based editors can be made substantially more precise on complex grounding tasks without retraining the diffusion model.
  • Multiple region edits can be executed in a single pass, avoiding the accumulated errors and extra compute of iterative inpainting.
  • A small reinforcement-learned planner (about 1k instruction-only examples) can produce reliable region guidance, reducing dependence on large paired editing datasets.
  • Region-level negative prompts are supported directly, preventing unintended changes to semantically similar areas.
  • The IV-Edit benchmark provides a public testbed for instruction–visual complexity, with fine-grained grounding and knowledge-intensive edits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the attention-isolation principle transfers, other diffusion architectures (e.g., cross-attention U-Nets) could gain similar spillover resistance by constraining which text tokens each spatial region attends to.
  • The structured region-hint output makes interactive editing practical: a user can directly adjust a bounding box or hint and re-run the same one-pass pipeline.
  • One testable extension: swap the planner for a stronger base VLM without RL to isolate how much of the gain comes from the attention mask versus the reinforced reasoning.
  • The benchmark's consistency dimension is structurally aligned with the method's design, so future work should report human ratings or an independent judge before generalizing the claimed margin.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RePlan, a plan-then-execute framework for instruction-based image editing under 'Instruction-Visual Complexity'. A VLM planner (Qwen2.5-VL-7B) is trained via two-stage GRPO with roughly 1k instruction-only examples to output region-aligned guidance (bounding boxes plus per-region hints). The diffusion decoder (Flux.1 Kontext or Qwen-Image-Edit, both MMDiT) executes this guidance through a training-free attention-region injection mechanism governed by five attention rules. The authors introduce IV-Edit, an 800-pair benchmark emphasizing fine-grained grounding and knowledge-intensive edits, and evaluate RePlan against several open and closed baselines using Gemini-2.5-Pro as a judge. They report consistent improvements in Overall and Consistency, along with ablations on planner choice, reasoning, and staged RL, and a bbox-perturbation robustness test.

Significance. If the quantitative claims are valid, the central contribution is notable: a small RL-trained planner plus a training-free attention regime can substantially improve open-source MMDiT editors on hard grounding tasks without requiring paired editing data. The attention-region injection mechanism is clearly specified, and the internal ablations (planner choice in Table 2, reasoning on/off and staged RL in Table 3) support the importance of the planner and the RL training. The bbox perturbation robustness test (Table 5) is a genuine and commendable probe of the method's dependence on region accuracy. However, the evaluation protocol currently prevents acceptance: the Weighted metric used for the headline comparisons is arithmetically inconsistent with the reported numbers, and the benchmark construction/judge share the same model family, with no human evaluation. The claimed edge over baselines is therefore not yet established.

major comments (4)
  1. [§5.1, Eq. (Weighted) and Table 1] The definition 'Weighted = (1/N) Σ (Target+Quality+Effect+Effect×Consistency)/4' cannot reproduce Table 1. Since all ratings are in [1,5], Effect×Consistency ≥ 1 for every sample. For the Flux baseline, the lower bound of the weighted average is (3.93+3.34+2.73+1)/4 = 2.75, but the table reports 2.49. The implied mean of Effect×Consistency would have to be about −0.04, which is impossible. Because Weighted is the metric introduced to counter consistency inflation, the central comparison (e.g., +0.06 for Flux, +0.29 for Qwen) is unsupported. Please correct the formula or provide per-sample computation and verify all Weighted values in Tables 1, 2, 3, and 5.
  2. [§4 Evaluation Protocol and Appendix C] The benchmark construction and evaluation are not independent: Gemini-2.5-Pro was used to generate and filter instructions (Appendix C), Qwen2.5-VL-72B annotated ground-truth bounding boxes, and Gemini-2.5-Pro also serves as the grading judge (Section 4). The judge therefore shares a model family with the benchmark generator. More seriously, RePlan's largest measured gains are in Consistency (+0.76 on Flux, +1.45 on Qwen in Table 1), and consistency is directly hard-enforced by attention rules 4 and 5, which restrict background tokens to attend only to the global text. Without a human evaluation or an independently developed judge, the risk that Gemini's ratings reward what the mechanism trivially enforces cannot be dismissed. Provide a human study on a representative sample or use a second, independently built VLM judge, and report agreement between judges.
  3. [Table 1, Effect column] The Effect dimension regresses on both backbones (Flux 2.73→2.59; Qwen 3.24→3.16), while the largest gains are in Consistency. This contradicts the paper's claim of 'consistently outperforming' and 'improving overall fidelity' in the abstract and introduction. The Weighted metric was designed to handle exactly this trade-off, but it is arithmetically broken (see above). Please provide a per-task breakdown, test whether the Effect regression is statistically significant, and discuss whether the attention rules trade away editing strength for preservation. Without this, the net benefit of RePlan on edit quality remains unclear.
  4. [§5.1 / all tables] The quantitative claims lack statistical validation. The reported comparisons are aggregate 5-point means over roughly 800 samples, with no error bars, confidence intervals, or significance tests. The headline Weighted improvement for Flux is +0.06 (2.49→2.55), a difference likely below inter-annotator noise. Report bootstrap confidence intervals, per-sample distributions, or paired significance tests (e.g., Wilcoxon) for the main comparisons, and make clear whether the reported numbers are means over all samples or medians.
minor comments (5)
  1. [Throughout] Typos: 'Specificly' in Section 4, 'anlysis' in §3.2, 'ensuring' in §3.3 ('This ensuring global stylistic coherence'), and inconsistent naming 'Gemini2.5-pro' in Table 2 vs 'Gemini-2.5-Pro' elsewhere. The formula in §5.1 uses 'Weighted=P samples...' due to typesetting; please use proper summation notation.
  2. [Related Work / References] The citation to Goodfellow et al. (2016) in the context of VLM reasoning abilities is inappropriate (it is a deep-learning textbook). Please replace it with a relevant reference or remove it.
  3. [§4 / Appendix C] The paper mentions 'manual verification' and 'manual sample-by-sample filtering' for the test set, but provides no details on annotator expertise, number of annotators, or inter-annotator agreement. Add these details for reproducibility.
  4. [§3.4] The 'Reasoning quality reward' uses the length of <think> content as a proxy, with a cap. This is an ad-hoc axiom; please justify this proxy or provide evidence that longer reasoning correlates with plan quality. Also clarify whether the '1k instruction-only examples' are used for supervised alignment before RL and what that alignment consists of.
  5. [Table 1, row ordering] Consider marking the second-best open-source model clearly; the underline is easy to miss in the printed table. Also, the closed-source models are included without discussion of their access/cost differences.

Circularity Check

0 steps flagged

No circular derivation found: RePlan's reported gains come from a stated training-free attention mask and separate RL training, not from fitting the evaluation score.

full rationale

The paper's central chain is plan-then-execute: a VLM planner outputs region–hint pairs, and a diffusion MMDiT decoder executes them under explicit attention rules (§3.2–3.3). These rules are mechanism design, not fitted parameters; the Consistency gain is a direct consequence of rules 4–5, so it is an engineered property being measured rather than a circular prediction. The GRPO training (§3.4) optimizes a reward that resembles the evaluation metric (consistency reweighted by effect), but the model is still evaluated on a separately filtered IV-Edit test set with manual sample-by-sample filtering (Appendix C), so the evaluation is not the training set and does not reduce by construction. The benchmark construction and grading by Gemini-2.5-Pro / Qwen2.5-VL-72B is a legitimate validity concern about judge alignment, but it is not a self-definitional derivation circle in the paper's equations. The §5.1 Weighted formula appears arithmetically inconsistent with Table 1 (the implied mean of Effect×Consistency would be negative), which is a serious correctness/reproducibility issue, not a circularity issue. No load-bearing self-citation or imported uniqueness theorem is used; self-citations in Related Work are non-essential. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No new physical or conceptual entities are postulated; IV-Complexity is a framing label for an evaluation setting, not an entity. The paper's load-bearing assumptions are the validity of the VLM judges/rewards, the correctness of the bbox-to-patch mapping, and the coherence-preserving property of full image-latent attention.

free parameters (2)
  • λ (format-reliability weight in Stage-2 reward) = not stated ('a small weight')
    R^(2) = R_T + R_E + R'_C + λR^(1) in Section 3.4; the trade-off between image-level rewards and the Stage-1 format/reasoning reward is hand-set and unreported, and the ablation does not sweep it.
  • Reasoning-quality reward length cap = capped maximum, value not stated
    Stage-1 'reasoning quality' is measured as think-tag text length up to a cap (Section 3.4); the cap is a hand-chosen proxy that directly shapes planner behavior and is never reported.
axioms (5)
  • domain assumption Gemini-2.5-Pro ratings are valid ground-truth edit quality for Target/Effect/Consistency/Quality
    The evaluation protocol in Section 4 rests entirely on an unvalidated proprietary judge; no human evaluation or judge-human correlation study is provided.
  • domain assumption IV-Edit's test set is an unbiased sample of IV-Complexity editing tasks
    Appendix C: instructions generated by Gemini-2.5-Pro from sampled category combinations, filtered by Qwen2.5-VL-72B bbox annotation and Gemini-2.5-Pro ambiguity filtering, then manually filtered; the same model family is used for grading.
  • domain assumption Pixel-space bounding boxes map correctly to VAE patch-grid token groups for attention masking
    Equation 4 groups image patch tokens by bbox B_k; there is no verification that token-grid boundaries align with the VAE spatial layout other than the robustness test in Appendix G.
  • domain assumption Full image-latent self-attention preserves global coherence so region-restricted text attention does not degrade out-of-region content
    Attention rule 3 asserts edits may extend beyond the bbox 'when necessary'; Appendix E shows cutting image-region attention creates visible boundaries, so the design relies on this empirical balance.
  • ad hoc to paper Longer <think> text is a positive proxy for plan reasoning quality
    Stage-1 reward R_R increases with think-tag length up to a cap (Section 3.4); this is an ad hoc proxy for reasoning fidelity, not a measure of reasoning correctness.

pith-pipeline@v1.3.0-alltime-deepseek · 12764 in / 16770 out tokens · 159885 ms · 2026-08-03T15:23:23.600017+00:00 · methodology

0 comments
read the original abstract

Instruction-based image editing enables natural-language control over visual modifications, yet existing models falter under Instruction-Visual Complexity (IV-Complexity), where intricate instructions meet cluttered or ambiguous scenes. We introduce RePlan (Region-aligned Planning), a plan-then-execute framework that couples a vision-language planner with a diffusion editor. The planner decomposes instructions via step-by-step reasoning and explicitly grounds them to target regions; the editor then applies changes using a training-free attention-region injection mechanism, enabling precise, parallel multi-region edits without iterative inpainting. To strengthen planning, we apply GRPO-based reinforcement learning using 1K instruction-only examples, yielding substantial gains in reasoning fidelity and format reliability. We further present IV-Edit, a benchmark focused on fine-grained grounding and knowledge-intensive edits. Across IV-Complex settings, RePlan consistently outperforms strong baselines trained on far larger datasets, improving regional precision and overall consistency. Our project page: https://replan-iv-edit.github.io

Figures

Figures reproduced from arXiv: 2512.16864 by Bei Yu, Bohao Peng, Dong Yu, Jiaya Jia, Lei Ke, Longxiang Tang, Tianyuan Qu, Xiaohang Zhan, Yuqi Liu.

Figure 1
Figure 1. Figure 1: We define Instruction-Visual (IV) Complexity as the challenges that arise from com￾plex input images, intricate instructions, and their interactions—for example, cluttered layouts, fine-grained referring, and knowledge-based reasoning. Such tasks require models to conduct fine￾grained visual reasoning. To address this, we propose RePlan, a framework that leverages the inher￾ent visual understanding and rea… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our RePlan framework. The bottom part of the figure shows the overall architecture. Given an input image and text instruction, the VLM analyzes them via chain-of-thought reasoning and produces region-aligned guidance, where each guidance includes a region bbox and its editing hint. Each hint is futher encoded by a text encoder into a feature token, while image patch tokens are obtained by VAE e… view at source ↗
Figure 3
Figure 3. Figure 3: VLM output format Example Output Format. We require the VLM to output structured text for convenient post-processing, with explicit markers separating reasoning, global edits, and region guidance. Region guidance are expressed in JSON format. An example is shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Attention rule visualization. We use different highlight colors to indicate dif￾ferent rules, which correspond to Hint iso￾lation, Region constraint, Background con￾straint and Image–latent full interaction. Image encoding and patch grouping. The image I is first processed by the VAE encoder to produce a spatial feature map F img which can be reshaped into patch tokens {fi,j}i=1..M, j=1..N . Each editing r… view at source ↗
Figure 5
Figure 5. Figure 5: Overview of our IV-Edit Benchmark. (a) and (b) respectively shows the distribution of referring types and task types across the dataset. IV-Edit is explicitly designed to reflect the IV￾Complexity challenge, where user instructions require aligning fine-grained language with rich and diverse visual contexts. (c) presents visual examples spanning a wide range of real-world scenar￾ios and fine-grained instru… view at source ↗
Figure 6
Figure 6. Figure 6: Editing results comparison. We use Flux.1 Kontext dev as the backbone of RePlan. Notably, GPT-4o enforces fixed aspect ratios, leading to unavoidable cropping for non-standard images [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of expected editing re￾gion counts per instruction General Statistics of IV-Edit. In [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The right image is the original, and the left image shows the editing result where the [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: We decouple the noise patches and image patches, preserving their respective self [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The left image shows the editing result, and the right image visualizes the editing region [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Overlapping Case 1 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Overlapping Case 2 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: More Comparative Results. We use Flux.1 Kontext dev as the backbone of RePlan. For [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: More Comparative Results. We use Flux.1 Kontext dev as the backbone of RePlan. [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: More Comparative Results under text editing scenario. We use Flux.1 Kontext dev as the [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GMO-E$^2$DIT: Grounded Multi-Operation Editing for E-Commerce Images

    cs.CV 2026-07 unverdicted novelty 6.0

    GMO-E²DIT is an agentic editing framework that decouples VLM-based planning from mask-conditioned rendering and uses reflection to execute multi-operation e-commerce image edits with error recovery.

  2. GMO-E$^2$DIT: Grounded Multi-Operation Editing for E-Commerce Images

    cs.CV 2026-07 unverdicted novelty 6.0

    GMO-E²DIT is an agentic framework that decouples VLM-based edit planning from mask-conditioned rendering using reflection loops for reliable multi-operation e-commerce image editing.

  3. Rethinking Scribble-Guided Image Editing: Generalization, Instruction Adherence, and Multi-Tasking

    cs.CV 2026-05 unverdicted novelty 6.0

    Empirical studies reveal instruction-level generalization as the main bottleneck in scribble-guided editing; three strategies (curriculum, multi-task mosaicking, edit-focused loss) achieve SOTA on VIBE benchmark.

  4. DDA-Thinker: Decoupled Dual-Atomic Reinforcement Learning for Reasoning-Driven Image Editing

    cs.CV 2026-04 unverdicted novelty 6.0

    DDA-Thinker decouples planning from generation and applies dual-atomic RL with checklist-based rewards to boost reasoning in image editing, yielding competitive results on RISE-Bench and KRIS-Bench.

  5. Making Image Editing Easier via Adaptive Task Reformulation with Agentic Executions

    cs.CV 2026-04 unverdicted novelty 6.0

    An MLLM agent reformulates image editing tasks into executable operation sequences to improve reliability on challenging cases across existing generative backbones.

  6. Making Image Editing Easier via Adaptive Task Reformulation with Agentic Executions

    cs.CV 2026-04 conditional novelty 6.0

    An MLLM agent that profiles, routes, and reformulates image-editing queries into better-conditioned multi-step operations consistently improves existing editors on hard cases.

  7. ProductConsistency: Improving Product Identity Preservation in Instruction-Based Image Editing via SFT and RL

    cs.CV 2026-06 unverdicted novelty 5.0

    Introduces ProductConsistency dataset, benchmark, and Cyclic Consistency reward to fine-tune image editing models, achieving a 5x reduction in character error rate for product identity preservation.

  8. OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration

    cs.CL 2026-05 unverdicted novelty 4.0

    OmniVerifier-M1 is a generalist visual verifier using symbolic outputs for meta-verification and decoupled RL to outperform joint optimization for robust verification and agentic self-correction.

Reference graph

Works this paper leans on

4 extracted references · cited by 6 Pith papers

  1. [1]

    Directly extract the core action, the target object, and any specific attributes

  2. [2]

    If the request involves multiple distinct steps, break it down into separate commands, one per line

  3. [3]

    Keep the commands as short and direct as possible

  4. [4]

    WAITED 8

    Your response must contain ONLY the rephrased command(s). Do not add any explana- tions, apologies, or conversational text. Instruction: E ATTENTIONRULEDISCOVERY When experimenting with different attention rules, we observed several interesting phenomena. First, if we cut off the attention across different regions of an image, very clear boundaries appear...