{"id":"f511afcf-76d3-4ba1-86a4-1b10432114e9","arxiv_id":"2608.09097","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new dataset and method let local image edits be guided by both a sketch and a text instruction, improving geometric and semantic control over sketch-only or text-only baselines.","lead":"This paper introduces SI-Data, a dataset of 6.5K image quadruplets that pair a photo with a local sketch, an editing instruction, and an edited target, plus SI-Edit, a model fine-tuned on that data. The combination lets users guide local edits with both a drawn sketch and a text instruction, which the authors claim gives more precise geometric control than sketch-only or text-only methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The optical-flow sketches in Sec. 3.2 step (4) are motion-boundary contours, not the target-contour strokes users would draw; this proxy shapes both SI-Data and the LCD/mIoU metrics, so the benchmark may measure alignment to flow artifacts rather than to user intent.","rationale":"The central claim is that SI-Edit achieves 'precise, pixel-level local refinements aligned with user intent' and that SI-Data is a valid benchmark for this task. Both halves depend on the sketch S_k being a faithful stand-in for what a user would draw. The construction in Sec. 3.2 step (4) chooses optical-flow motion boundaries as that stand-in, and the evaluation metrics LCD and mIoU use these same boundaries as ground truth. If the proxy mismatches real hand-drawn sketches—which is plausible because motion boundaries are closed contours around moving regions, whereas users typically draw open strokes along the desired target contour—then the training signal and the evaluation are both misaligned with the stated goal. The paper's qualitative hand-drawn results suggest the model can generalize to some hand-drawn inputs, but they do not quantify how different those inputs are from the flow-extracted sketches, nor how much performance drops when the sketch distribution shifts. This is precisely the assumption the reader flagged, and I agree it is the most load-bearing. A secondary concern is the unfair comparison to zero-shot baselines, but the sketch proxy is more foundational: even a perfectly fair comparison would not rescue the 'user intent' claim if the benchmark's ground-truth sketches are unrepresentative. The proposed human-sketching study directly tests whether the proxy lands, and the outcome would either validate the benchmark or require reinterpreting the quantitative results. Since the paper already received a CONDITIONAL verdict on addressable issues, this concern does not by itself change the verdict; it sharpens what condition must be met.","tokens_in":18068,"tokens_out":7101,"duration_ms":87929,"concrete_test":"Select 50 SI-Data test samples; for each, show the source image and instruction to at least 5 human annotators and ask them to draw the geometric edit they intend (without showing the target). Compute agreement between human sketches and the flow-extracted S_k (e.g., Chamfer distance or F1 after alignment). Then run SI-Edit on the same source+instruction pairs using the human sketches instead of S_k, and recompute LCD/mIoU. If the human-to-flow sketch agreement is low (e.g., mean Chamfer distance > 10 px) or the human-sketch LCD increases substantially relative to the flow-sketch LCD, the proxy does not represent user intent and the benchmark/metrics are measuring an artifact rather than editing quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 step (4) defines the user sketch S_k as motion boundaries: dense optical flow (SEA-RAFT) between I_src and I_tgt, Sobel gradients on each flow channel, thresholded at 0.06. Motion boundaries are edges in the flow field—closed contours around regions whose apparent displacement changed—not the open target-contour strokes a user would draw for an instruction like 'bend the stem into an S-shape' (compare Fig. 1, where the user sketch is a single Target Contour stroke). Consequently, SI-Data quadruplets pair instructions with flow-derived contours, and the evaluation metrics LCD and mIoU in Sec. 3.3 treat those same contours as ground truth. If human sketches systematically differ from flow-boundary contours (e.g., users draw the desired new shape or centerline rather than the motion discontinuity), then a model trained and evaluated on SI-Data can score well by reconstructing the teacher's motion artifacts while failing on genuine hand-drawn input. The paper provides only qualitative hand-drawn examples (Fig. 6, Sec. E.4) and no quantitative evidence that the flow proxy matches human sketching. This assumption is load-bearing because it underlies both the dataset's validity and the headline claim of pixel-level alignment with user intent: a benchmark built on an unrepresentative sketch proxy cannot substantiate that claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SI-Data, a dataset of 6.5K quadruplets (source image, target image, local sketch, instruction) for fine-grained geometric image editing, constructed by using Qwen3-VL to generate spatially explicit instructions, Nano Banana Pro to synthesize target images, and SEA-RAFT optical flow to extract binary sketches from motion boundaries. It also proposes SI-Edit, which fine-tunes FLUX.1 Kontext with a learnable <sk> task-trigger token and a same-position encoding (SPE) strategy, and evaluates on a 159-sample test set with four metrics: LCD, mIoU, ΔCS, and BPS. The paper reports that SI-Edit outperforms SketchEdit, MagicQuill, and FramePainter on all metrics, and includes ablations on the token configuration and qualitative ablations for SPE.","tokens_in":18375,"tokens_out":5797,"duration_ms":61769,"significance":"If the flow-derived sketch proxy is accepted as representative of user sketches, SI-Data is the first public quadruplet dataset for sketch-instruction guided local editing, and SI-Edit demonstrates a plausible way to combine semantic and geometric control through a learnable task token and shared positional encoding. The paper releases data and code, and the ablations on token placement and learnability support several design choices. However, the quantitative evidence for the central claim of pixel-level alignment with user intent is weakened by the issues below: the sketch proxy and the evaluation metrics are derived from the same pipeline, the baselines are not trained on the benchmark, and the reported numbers lack variance estimates.","major_comments":[{"comment":"The binary sketch S_k is defined as optical-flow motion boundaries (Sobel gradients thresholded at 0.06), not as the target-contour strokes a user would draw for an instruction like 'bend the stem into an S-shape'. The same flow-derived sketches are then used as ground truth for LCD and mIoU in Section 3.3, and the test set of 159 samples is drawn from SI-Data. Consequently, the quantitative results in Table 2 measure alignment to flow-derived contours, not to genuinely hand-drawn input, which is the stated use case. The only hand-drawn evaluation is qualitative (Sec. E.4, Fig. 9 and Fig. 6 col. 5). This is load-bearing for the claim of pixel-level precision aligned with user intent. A concrete remedy: collect hand-drawn sketches on a subset of test images and report LCD/mIoU against those hand-drawn sketches, or compare the distribution of flow-derived sketches with hand-drawn sketches to validate the proxy.","section":"Sec. 3.2, step (4); Sec. 3.3"},{"comment":"SI-Edit is fine-tuned on SI-Data, while the three baselines (SketchEdit, MagicQuill, FramePainter) are applied zero-shot without any adaptation to the quadruplet format or fine-tuning on the training split. The claim of 'best performance among all competing approaches across all evaluation metrics' is therefore not a controlled comparison: the margins may reflect access to the SI-Data training pairs rather than the proposed architecture. The paper should either fine-tune the baselines on SI-Data (where the model formulation permits it), or evaluate SI-Edit in a zero-shot setting, and report both conditions so the dataset contribution and the method contribution can be separated.","section":"Sec. 5.2.2, Table 2"},{"comment":"All metrics are reported as single point estimates on a 159-sample test set, with no error bars, confidence intervals, or significance tests. The LCD difference between SI-Edit (4.292) and MagicQuill (7.434) is large, but the ablation differences in Table 3 (e.g., ΔCS 0.0149 vs 0.0174, LCD 4.023 vs 4.292) are small and could be within random variation. Please provide standard deviations over at least three random seeds or bootstrapped 95% confidence intervals, plus a paired significance test (e.g., Wilcoxon signed-rank) for the headline metrics.","section":"Sec. 5.2.2, Table 2; Table 3"},{"comment":"The Same Position Encoding (SPE) strategy is a core contribution, but its ablation appears only as qualitative examples in the appendix (Fig. 10, 'w/ SPE' vs 'w/o SPE'), with no quantitative numbers on the test set. The claim that SPE 'reduces spatial drift' is not supported by the reported metrics. Please add a quantitative SPE on/off ablation reporting LCD, mIoU, ΔCS, and BPS.","section":"Sec. 4.2; Sec. E.2"}],"minor_comments":[{"comment":"The threshold tau in the BPS definition is not specified in the main text or appendix; please give its value and a brief sensitivity check.","section":"Sec. 3.3, Eq. (3)"},{"comment":"The test set is described as '159 test samples randomly selected at the source-image level'; please clarify whether any source image appears in more than one test sample and whether the train/test split is at the source-image level.","section":"Sec. 5.1"},{"comment":"The fixed-prompt baseline is described in Table 3 as initialized with 'following the sketch lines' while Section 4.2 says the learnable token is initialized with the mean of 'follow the sketch lines'; please make the phrasing consistent.","section":"Sec. 4.2 vs Table 3"},{"comment":"The statement that the <sk> token is added to 'each text encoder' is vague; please specify which text encoders are actually used in FLUX.1 Kontext and whether the token is added to all of them.","section":"Sec. 4.2"},{"comment":"Please provide training details for reproducibility: batch size, gradient accumulation steps, learning-rate schedule, effective number of training hours, and the exact version and access date of the Nano Banana Pro model.","section":"Appendix A.1"},{"comment":"The 'Overlay Img' columns show the sketch overlaid on the source, but the raw binary sketch is never displayed separately; please label the sketch itself or include a separate panel so the reader can see the stroke constraint.","section":"Fig. 5 and Fig. 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be salvageable: the dataset and the method are both potentially useful, and the authors have released code and data. However, the comparison fairness and the gap between flow-derived sketches and user-drawn sketches need substantive work. I would ask the authors to add the validation experiments described in the major comments and to temper the claims of 'pixel-level precision aligned with user intent' until the hand-drawn generalization is quantified. The zero-shot baseline comparison could be reframed as a benchmark-style comparison, but as written the 'outperforms' claim is too strong."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper's real contribution is SI-Data, the first public quadruplet dataset for sketch-instruction editing, with code and data released. That fills a genuine gap: SketchEdit and FramePainter never released their training data. The method—LoRA fine-tuning of FLUX.1 Kontext with the <sk> token and shared position encoding—is straightforward and the ablations actually support the design choices, including a trade-off between LCD/BPS and ΔCS for the token configuration. The qualitative results are strong, and the hand-drawn sketch examples in the appendix are the most convincing evidence that the model generalizes beyond its training distribution.\n\nThe soft spots are real, but they're the usual kind for a dataset+method paper. First, the sketch proxy: the training sketches are motion boundaries from optical flow, not target contours a user would draw. A bend instruction gives you a double-edge contour around the deformed region rather than a single curve. Since the test sketches and the LCD/mIoU metrics both come from the same flow pipeline, the benchmark measures agreement with the teacher's motion artifacts. The hand-drawn generalization results are encouraging, but there's no quantitative test on human sketches, so the load-bearing claim that the model aligns with user intent is under-supported. Second, the evaluation is one-sided: the baselines are zero-shot while SI-Edit is trained on the test benchmark's own data, the test set is 159 images, and the numbers come without error bars or significance tests. The user study helps, but it's also self-selected. Finally, \"pixel-level precision\" is asserted more strongly than the metrics prove.\n\nNone of these are fatal. The central idea—synthesize paired data with an LLM and a teacher generator, extract geometric anchors with flow, and fine-tune a diffusion transformer—works. The dataset alone is worth peer review, and the method is a reasonable baseline for anyone working on this task. I'd send it to review and ask for three things: a study comparing flow sketches against actual hand-drawn contours, training-fair baselines or at least error bars, and more careful wording about pixel-level claims.\n\nWho's it for: people working on interactive image editing, sketch-conditioned generation, and dataset synthesis. A serious referee should engage with it.","headline":"A solid dataset-plus-method paper; the dataset is the real contribution, the flow-based sketch proxy and one-sided evaluation are the main soft spots.","tokens_in":18903,"tokens_out":3511,"would_cite":true,"duration_ms":36351,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that local image editing becomes pixel-precise only when a sketch and a text instruction are conditioned jointly, and it introduces both the dataset and the model to demonstrate it.","keywords":["sketch-guided image editing","local geometric deformation","diffusion models","instruction following","dataset construction","optical flow sketches","pixel-level editing","spatial-semantic conditioning"],"falsifier":"Take a set of source images, apply a camera-only pan or global color change with no local geometric deformation, and run the SI-Data sketch-extraction pipeline; if it yields nonzero sketches and the localized chamfer distance reports finite alignment error, the optical-flow proxy is measuring displacement rather than user intent, and both the dataset and geometric metrics need re-grounding.","tokens_in":17847,"feed_emoji":"✏️","tokens_out":6366,"duration_ms":69067,"temperature":0.7,"pith_summary":"This paper argues that text-only and sketch-only editing fail for different, complementary reasons—text lacks precise spatial anchors, and sketches lack semantic intent—and that combining the two in one model can achieve pixel-level local geometric changes. To make that combination learnable, it introduces SI-Data, a 6.5K-sample dataset of (source, target, sketch, instruction) quadruplets built automatically from an MLLM-generated instruction, a generator-synthesized target, and an optical-flow-derived sketch. It then proposes SI-Edit, which fine-tunes a latent diffusion transformer with same-position encoding between the source and the sketch overlay and with a learnable task-trigger token. The paper reports that SI-Edit outperforms all compared sketch-based baselines on geometric alignment, semantic fidelity, and background preservation, with a localization error of 4.292 versus 7.434 for the best baseline. A sympathetic reader would take the central claim to be that spatial-semantic collaboration, not a single modality, is what unlocks precise local editing.","feed_headline":"Pairing sketches with text instructions makes edits pixel-precise","feed_subtitle":"A new quadruplet dataset and a collaborative conditioning model keep local geometry on-sketch while following the text intent.","key_machinery":"The load-bearing object is the quadruplet of source image, target image, local sketch, and instruction, and the mechanism that makes it work is Same Position Encoding: the source image and the sketch-overlay image are patchified into token sequences that share identical temporal and spatial rotary positional coordinates, so the model cannot treat the sketch as an independent drifting condition. The second mechanism is the learnable task-trigger token, initialized from the mean embedding of the phrase 'follow the sketch lines' and placed at the beginning of the instruction, which switches the diffusion transformer into sketch-guided editing mode. These two devices, trained with low-rank adapters on a rectified-flow matching objective, are what carry the alignment between semantic intent and geometric constraint.","core_discovery":"The paper's core discovery is that the bottleneck in sketch-based local editing is not generative capacity but the joint conditioning signal: when a sketch and a text instruction are aligned at the token level, the model can execute fine geometric deformations while preserving object identity and background. Technically, this is achieved by overlaying the binary sketch on the source image, giving source and overlay tokens identical 3D rotary positional coordinates, and prefixing the instruction with a learnable task-trigger token that anchors the editing mode. Evaluated on 159 test quadruplets, the method reports the lowest localized chamfer distance, the highest mask intersection-over-union, the largest improvement in CLIP-based semantic alignment, and the highest background-preservation score among the compared sketch-based baselines, and a pairwise user study confirms the trend. The same recipe is shown to transfer to human-pose movement editing using hand-drawn motion sketches.","pith_inferences":["The authors' ground-truth targets are themselves outputs of a generative model following an MLLM-written instruction, so the reported numbers certify alignment with a synthetic interpretation of intent; a human-drawn-sketch test with independently verified targets would test the claim more directly.","Thresholding motion-boundary gradients at 0.06 likely captures large smooth deformations better than small articulated ones, so the paper's 'pixel-level precision' is demonstrated on the dataset's deformation distribution, and an extension would build a test set of small-scale edits.","A directly testable consequence of the same-position-encoding mechanism is that alignment should degrade gracefully as the sketch overlay is eroded, occluded, or blurred; measuring localized chamfer distance under those perturbations would isolate whether the shared positional encoding carries the alignment.","The conditioning design invites an interactive editing loop, because the model accepts new sketches at inference time without retraining, so a deployable system could let users draw, generate, inspect, and redraw iteratively."],"forward_implications":["If the central claim holds, the practical interface for local geometric editing should give users both a sketch and a short text instruction, since neither alone disambiguates the intended edit.","SI-Data's public quadruplet format gives other research groups a benchmark for training and comparing sketch-instruction editors, closing the data gap the paper identifies.","The combined set of localized chamfer distance, mask IoU, CLIP-score improvement, and background-preservation score can be reused as a standard evaluation protocol for local geometric editing, separating geometric adherence from semantic adherence and background preservation.","The same same-position-encoding and task-trigger recipe should transfer to other diffusion backbones and to pose or motion editing, because the mechanism is a conditioning-level device rather than a task-specific module."],"supporting_citations":[{"why":"It supplies the latent diffusion transformer backbone and native in-context conditioning that SI-Edit fine-tunes.","marker":"[14]"},{"why":"It computes the dense optical flow from which SI-Data's binary sketches are extracted.","marker":"[33]"},{"why":"It synthesizes the target images that realize the generated editing instructions.","marker":"[8]"},{"why":"It generates the localized geometric instructions and their augmented variants for SI-Data.","marker":"[28]"},{"why":"It provides the low-rank adapters used to fine-tune the backbone's attention layers.","marker":"[10]"},{"why":"It extracts object contours from generated images for the localized chamfer distance and mask IoU metrics.","marker":"[13]"},{"why":"It defines the symmetric chamfer distance that underlies the localized chamfer distance metric.","marker":"[6]"},{"why":"It serves as a sketch-based inpainting baseline whose semantic blindness motivates the need for the instruction signal.","marker":"[38]"},{"why":"It serves as an interactive baseline that infers intent from strokes, and its inferred intent can deviate from the desired edit.","marker":"[19]"},{"why":"It serves as a motion-based baseline that reformulates image editing as image-to-video generation without explicit semantic instructions.","marker":"[42]"}],"fun_headline_variants":["Sketch plus text: pixel-perfect local image edits","Aligning sketches and text sharpens local edits","Pixel-precise local edits from sketch-text pairing","Joint sketch-text conditioning nails pixel-level editing","Sketch-text alignment yields pixel-accurate local edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the dominant apparent displacement between source and target images, captured as optical-flow motion boundaries thresholded at 0.06, equals the geometric deformation a user would draw; if hand strokes do not coincide with those boundaries, the dataset's sketch signal and geometric metrics are mislabeled.","fun_headline_variants_meta":{"raw":{"variants":["Sketch plus text: pixel-perfect local image edits","Aligning sketches and text sharpens local edits","Pixel-precise local edits from sketch-text pairing","Joint sketch-text conditioning nails pixel-level editing","Sketch-text alignment yields pixel-accurate local edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000631,"raw_usage":{"total_tokens":2919,"prompt_tokens":958,"completion_tokens":1961,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1886}},"tokens_in":574,"tokens_out":1961,"duration_ms":14497,"temperature":1.0,"reasoning_tokens":1886,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:35:07.733626+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of source images, apply a camera-only pan or global color change with no local geometric deformation, and run the SI-Data sketch-extraction pipeline; if it yields nonzero sketches and the localized chamfer distance reports finite alignment error, the optical-flow proxy is measuring displacement rather than user intent, and both the dataset and geometric metrics need re-grounding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It computes the dense optical flow from which SI-Data's binary sketches are extracted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It synthesizes the target images that realize the generated editing instructions."},{"cited_title":"SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches","cited_arxiv_id":"2111.15078","evidence_quote":"It serves as a sketch-based inpainting baseline whose semantic blindness motivates the need for the instruction signal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It serves as an interactive baseline that infers intent from strokes, and its inferred intent can deviate from the desired edit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It serves as a motion-based baseline that reformulates image editing as image-to-video generation without explicit semantic instructions."}],"review_version":1}