{"id":"018fcdd5-3e49-441d-8584-25211fc106df","arxiv_id":"2412.01223","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PainterNet is a diffusion-model plugin that uses local prompts, attention supervision, and diverse masks to improve text-consistent image inpainting.","lead":"PainterNet adds a trainable branch to diffusion models to improve image inpainting when users describe the masked area with a short local prompt. It also introduces a new dataset and benchmark with diverse mask shapes and localized text prompts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is confounded: PainterBench's local prompts and diverse masks match PainterNet's training distribution, while baselines trained on global prompts and segmentation masks are evaluated without adaptation, biasing Table 1.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the evaluation benchmark matches PainterNet's training distribution while baselines are not adapted. This is not a minor issue; it directly undermines the paper's headline claim of surpassing state-of-the-art models. The concern is concrete and testable. I find no internal inconsistency that would warrant a stronger verdict, and the proposed method and dataset could still be useful contributions if the comparison were made fair. The original CONDITIONAL verdict remains appropriate: the paper should either provide neutral-benchmark results or scale back the SOTA claim. I agree with the reader's assessment and do not see grounds to move the verdict. A secondary technical concern is that Eq. 4's ATAL loss compares an average over actual-token attention maps to a binary mask; since the average is bounded by 1/L_S, the loss cannot reach zero, but this does not change the main verdict because the central claim is empirical and would need to be re-established under a fair evaluation regardless.","tokens_in":14222,"tokens_out":4421,"duration_ms":43439,"concrete_test":"Re-run the comparison on the original BrushBench benchmark under identical inference settings, using each method's native global prompts and recommended hyperparameters. If PainterNet does not also achieve the best or near-best results on BrushBench across IR, AS, CLIP Sim, Local CLIP Sim, and Gdino Acc, then the Table 1 superiority is specific to PainterBench and the claimed state-of-the-art status should be downgraded. As a second check, on PainterBench, provide baselines with their original global prompts and compute CLIP Sim against those global prompts; if PainterNet's lead shrinks or vanishes, the local-prompt advantage is an artifact of the evaluation design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PainterNet 'surpasses existing state-of-the-art models' rests entirely on Table 1, which is measured on PainterBench. PainterBench is constructed by the authors using the same mask-generation strategy as PainterData (Section 4.1 and supplementary Section 6.2) and local textual prompts produced by the same MLLM pipeline (Section 6.1). Baselines such as BrushNet, ControlNet-Inpainting, and HD-Painter were trained on global prompts and segmentation-based masks, per the paper's own descriptions (Sections 2.2 and 3.4). When evaluated on PainterBench, these baselines are given inputs that are out-of-distribution for them: local prompts and irregular or bounding-box masks. The metrics most favorable to PainterNet—Local CLIP Sim and Gdino Acc—directly reward matching the local prompt in the masked region, yet the baselines were never trained to use local prompts. The paper does not report performance on a neutral benchmark such as the original BrushBench, nor does it fine-tune baselines on PainterData. Thus the reported improvements (e.g., Local CLIP Sim 22.67 vs 22.35, Gdino Acc 0.96 vs 0.94) may reflect benchmark/training distribution alignment rather than a genuine architectural advantage. This is a load-bearing concern: if the evaluation is unbalanced, the abstract's SOTA claim is unsupported even if the method is sound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PainterNet, a plug-and-play control branch for diffusion-based image inpainting. The method freezes the Stable Diffusion U-Net and inserts features from a trainable branch at both layer outputs and cross-attention outputs (\"Attention Control Points\"), using local textual prompts in both branches. An additional Actual-Token Attention Loss (ATAL) regularizes the branch's cross-attention maps toward the mask region. The authors also construct a new training set PainterData and a benchmark PainterBench, with local captions generated by an MLLM and diverse masks (bounding box, irregular scribbles, segmentation). Quantitative results on PainterBench (Table 1) and ablations (Table 2) are used to claim state-of-the-art performance on image quality and global/local text consistency.","tokens_in":14515,"tokens_out":9932,"duration_ms":84750,"significance":"The paper's architecture is a natural and reasonably clean extension of BrushNet: it retains cross-attention in the control branch, adds attention-level injection, and introduces a simple attention regularizer. The dataset and benchmark construction is described in enough detail in the supplementary (including Algorithm 1 and the CLIP threshold) to be reproducible in principle. I do not find a definitional circularity between the ATAL objective and the evaluation metrics. The problem is that the empirical support for the \"surpasses state-of-the-art\" claim is not yet convincing: PainterBench is built with the same prompt and mask generation pipeline as PainterData, while none of the baselines is adapted or fine-tuned for that setting, and the reported margins on the headline metrics are small and without error bars. The paper would be a solid contribution if the authors added a neutral benchmark evaluation, an adapted-baseline comparison, and basic statistical reporting.","major_comments":[{"comment":"The state-of-the-art claim is confounded by the choice of benchmark. PainterBench is generated using the same mask-generation strategy (Section 3.4, Eq. (6); supplementary Section 6.2, Algorithm 1) and the same local-prompt MLLM pipeline (supplementary Section 6.1, Fig. 6) that are used to create PainterData. Baselines such as BrushNet and HD-Painter are trained on global prompts and segmentation-based masks (Sections 2.2 and 3.4). Evaluating them on PainterBench therefore gives PainterNet an in-distribution advantage while the baselines are out-of-distribution, and the metrics that most favor PainterNet (Local CLIP Sim, Gdino Acc) directly reward matching the local text in the masked region. The paper must additionally report results on BrushBench with the original global prompts and segmentation masks, or fine-tune the baselines on PainterData, or both. Without such a controlled comparison, the abstract's statement that PainterNet surpasses existing state-of-the-art models is not supported.","section":"Section 4.1, Table 1"},{"comment":"No error bars or significance tests are reported. The advantages on the headline metrics are small (Local CLIP Sim 22.67 vs 22.35; Gdino Acc 0.96 vs 0.94 on SD1.5), and the SDXL row actually shows a loss on IR and AS relative to SDXLI. The paper should report means and standard deviations over at least three runs (or bootstrap confidence intervals) and a paired test over the same masks and prompts, so the reader can judge whether the ranking is robust rather than noise.","section":"Section 4.2, Table 1"},{"comment":"The evaluation protocol for the baselines is underspecified. The text states that inference follows the baselines' open-source code and recommended hyperparameters, but it never states whether the PainterBench local prompts were fed verbatim to all models, whether a global caption was also provided, or how bounding-box and irregular masks were converted for methods that expect segmentation masks or specific input formats. This must be stated explicitly, and the paper should also report each baseline's performance under its intended input format on a common benchmark. As written, the comparison is not well-defined.","section":"Section 4.2"},{"comment":"The ablation does not isolate the contribution of ACP. Adding ACP alone decreases Local CLIP Sim (22.53 to 22.50) while increasing CLIP Sim, and the large jump to 22.67 appears only after ATAL is added. The paper does not report a variant with ATAL but without ACP, so the claim that both modules each contribute meaningfully is not demonstrated. Please add the missing ablation cell and, if the interaction is real, discuss it.","section":"Section 4.5, Table 2"}],"minor_comments":[{"comment":"There are several typos, including \"PatinerNet\" and \"BrtushNet\" in Section 3.2 and \"ACtual-Token Attention Loss\" in Section 3.3; the manuscript should be proofread.","section":"Section 3.2, 3.3"},{"comment":"The sentence \"our PainterData utilized only 8% of the data from BrushData, amounting to 500,000 data\" is unclear: does 8% correspond to 500,000 samples, and is PainterData a subset or a re-processed version of BrushData? Please clarify.","section":"Section 4.2"},{"comment":"The hyperparameters β, the mask-selection thresholds in Eq. (6), and the CLIP threshold 0.2 in the supplementary are introduced without an ablation. Please report sensitivity or justify the chosen values.","section":"Eq. (5), Eq. (6), Supplementary 6.1"},{"comment":"The CLIP Sim and Local CLIP Sim values (e.g., 26.12, 22.67) appear to be on a different scale from the usual CLIP cosine similarity in [0,1]; please state the exact formula and scaling used for these metrics.","section":"Table 1"},{"comment":"The paper does not mention code or data release. Since the contributions include a dataset and a benchmark, providing them (or a link) would be important for reproducibility and for the community to use PainterBench.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core technical idea is reasonable, and I do not see a fundamental flaw in the method itself. The main issue is that the empirical comparison as presented cannot support the claimed state-of-the-art result. A revision that adds a neutral benchmark or fine-tuned baselines and error bars would be sufficient. If the authors are unwilling to add those experiments, the claims in the abstract and conclusion should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main take: PainterNet is a plausible extension of BrushNet with a genuinely new attention loss and a useful local-prompt dataset, but the paper's headline SOTA claim is not supported by the evidence as reported. The stress-test is right: Table 1 is measured on PainterBench, which is built with the same mask-generation strategy and MLLM prompt pipeline as PainterData, while BrushNet, ControlNet-Inpainting, HD-Painter, and SDI are evaluated with local prompts and coarse masks that are out-of-distribution for them. That doesn't make the method bad, but it makes \"surpasses SOTA\" an overstatement.\n\nWhat's new: ATAL is the most interesting piece. It explicitly penalizes the cross-attention maps of actual text tokens for not matching the mask, which is a simple, direct way to tie prompt tokens to the inpainted region. Keeping cross-attention in the control branch, rather than removing it as BrushNet does, is a reasonable design choice, and the ACP ablation shows a small but consistent gain. The PainterData pipeline—cropping the masked object, having ShareGPT caption it, shortening with ChatGLM, and filtering by CLIP similarity—is a solid practical contribution, provided the data is actually released. The mask diversity (bbox, irregular, segmentation) is a nice touch for real-world use.\n\nSoft spots: The evaluation is the weak link. There are no error bars; differences like Local CLIP Sim 22.67 vs 22.35 and Gdino Acc 0.96 vs 0.94 are small. More importantly, baselines are not adapted to local prompts, and the paper does not report results on BrushBench or any neutral benchmark. The authors could fix this by fine-tuning baselines on PainterData or evaluating on a benchmark with global prompts, and by reporting variance. The claim about \"surpassing existing state-of-the-art\" should be scaled down to \"performs competitively on our benchmark\" until that is done. The ablation is okay but doesn't isolate the contribution of the local prompt itself.\n\nWho this is for: researchers working on text-guided inpainting or control-based diffusion editing. The ATAL idea and the dataset construction are worth a read even if the benchmark needs work.\n\nRecommendation: send it to peer review. The method and dataset are substantive enough to warrant referee time, but the paper needs a major revision of the evaluation section before publication.","headline":"PainterNet has a genuinely new attention loss and a useful local-prompt dataset, but the SOTA claim is not backed by the evaluation because the benchmark matches the method's training distribution and not the baselines'.","tokens_in":67,"tokens_out":2957,"would_cite":false,"duration_ms":74236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PainterNet claims to beat prior inpainting models by steering diffusion attention onto masked regions with local prompts and an attention loss.","keywords":["image inpainting","diffusion models","text-guided inpainting","cross-attention control","actual-token attention loss","plug-and-play","mask generation","PainterBench"],"falsifier":"Rerun Table 1 on the original BrushBench (global prompts, segmentation masks) and on a held-out set of unscripted user-drawn scribbles; if PainterNet's margins over BrushNet and HD-Painter shrink or reverse, the state-of-the-art claim rests on benchmark overlap with the training distribution rather than on a general architectural advantage.","tokens_in":14009,"feed_emoji":"🎨","tokens_out":10025,"duration_ms":73180,"temperature":0.7,"pith_summary":"The paper sets out to show that a plug-and-play inpainting plugin for Stable Diffusion can achieve better text-image consistency in masked regions than dedicated inpainting models, provided it is trained on local prompts and realistic, user-like masks. It proposes PainterNet, a dual-branch control network that keeps the full cross-attention structure of the base model, and adds an Actual-Token Attention Loss that pushes cross-attention maps onto the masked area so the generated pixels follow the local caption. A new training set, PainterData, provides local captions from multimodal LLMs and three mask styles (bounding boxes, irregular scribbles, and segmentation masks), while a new benchmark, PainterBench, evaluates on the same distribution. The paper reports that on this benchmark PainterNet beats BrushNet, ControlNet-Inpainting, HD-Painter, and Stable Diffusion inpainting on image reward, aesthetic score, global and local CLIP similarity, and grounding accuracy, and that the gains transfer to SDXL and to stylized downstream models.","feed_headline":"Plugin beats prior inpainting models on local prompt consistency","feed_subtitle":"A drop-in control branch trained on user-like masks and local captions, with a new benchmark to back the claim.","key_machinery":"The core mechanism combines (1) a dual-branch control architecture, inherited from BrushNet but retaining cross-attention layers, where the trainable branch receives a 9-channel input (noisy latent, masked image latent, downsampled mask) and is fused into the frozen SD U-Net through zero convolutions at both layer control points and attention control points; and (2) the Actual-Token Attention Loss, $\\mathcal{L}_{ATAL} = \\frac{1}{N}\\sum_{i=1}^{N}\\| \\frac{1}{L_S}\\sum_{j \\in S} A_{i,j} - m_i \\|_2^2$, where $A_{i,j}$ is the cross-attention map of layer $i$ for text token $j$, $S$ is the set of actual text tokens (excluding start/end specials), and $m_i$ is the mask resized to that layer's attention map. This loss forces the model's cross-attention to concentrate on the masked region rather than on the whole image, which is what aligns generated pixels with the local prompt. The third component is the training-time mask mixer: for each sample, a random $k\\in[0,1]$ selects a bounding-box mask (25%), an irregular finger-like scribble (50%), or the segmentation mask (25%), so the model learns to handle coarse, user-like inputs.","core_discovery":"PainterNet is a plug-and-play branch for Stable Diffusion-based inpainting that claims to fix two problems in existing control-based methods: global prompts that do not describe the masked region, and masks that do not match how users actually draw them. The paper replaces global captions with local prompts, keeps the full cross-attention layers of the trainable branch (unlike BrushNet, which removes them), and adds Attention Control Points plus an Actual-Token Attention Loss (ATAL) that drives the cross-attention maps of every layer toward the mask region. On the authors' PainterBench (built from PainterData with three mask types and MLLM-generated local prompts), PainterNet reports the best scores among BrushNet, ControlNet-Inpainting, HD-Painter, and Stable Diffusion inpainting on image reward, aesthetic score, global and local CLIP similarity, and Grounding DINO accuracy, with Local CLIP Sim of 22.67 on SD 1.5 and 23.06 on SDXL, and Gdino accuracy of 0.96 and 0.95. The ablation shows each component matters: the branch raises Local CLIP Sim from 22.40 to 22.53, ACP raises global CLIP Sim from 25.82 to 25.98, and ATAL lifts all key metrics.","pith_inferences":["A fair reading of the benchmark suggests part of the measured advantage may reflect that PainterBench matches PainterNet's training distribution (local prompts, mixed masks) while baselines trained on global prompts and segmentation masks face an out-of-distribution test; this distributional overlap, not just the architecture, could drive the reported margins.","The ATAL loss is generic enough to transfer beyond inpainting: any diffusion task that must respect a spatial region (object insertion, attribute editing, layout control) could reuse the same masked-attention objective.","The dataset recipe (MLLM captioning of cropped regions, short-caption post-processing, and CLIP-similarity filtering) is reusable for building localized caption datasets for other editing benchmarks, not just inpainting.","Varying the mask-mixing probabilities (fixed here at 25% box, 50% irregular, 25% segmentation) would be a natural next study; the paper does not test whether the reported robustness depends on that exact ratio."],"forward_implications":["PainterNet can be plugged into arbitrary Stable Diffusion-based models (the paper demonstrates anime, Van Gogh-style, and specific-role models) while keeping the base style intact.","Local prompts become the conditioning signal, so a user can describe just the masked object (e.g., 'a pink sweater') and the model aligns the inpainted pixels with that description rather than the full scene caption.","The ATAL loss compels cross-attention in every layer to concentrate on the mask, which the ablation ties to gains in all key metrics, especially local CLIP similarity and grounding accuracy.","Training on mixed mask shapes (box, irregular, segmentation) means users can draw coarse masks without following object contours, making the model usable with the masks people actually produce.","The same plugin trained on SD 1.5 transfers to SDXL, raising local CLIP similarity from 19.49 (SDXL-inpainting) to 23.06, suggesting the attention-control mechanism is backbone-agnostic."],"supporting_citations":[{"why":"BrushNet; supplies the dual-branch plug-and-play architecture that PainterNet extends by keeping cross-attention layers, and is the primary baseline.","marker":"[16]"},{"why":"ControlNet; provides the zero-convolution control-branch insertion method and is a baseline.","marker":"[52]"},{"why":"CLIP; the text encoder whose tokens define the 'actual tokens' in ATAL and whose similarity filters generated local prompts.","marker":"[32]"},{"why":"ShareGPT; generates the local textual prompts for masked regions in PainterData.","marker":"[5]"},{"why":"ChatGLM; post-processes the long local prompts into short object-specific captions.","marker":"[41]"},{"why":"Stable Diffusion; the frozen base model that PainterNet's branch conditions, and its inpainting variant is a baseline.","marker":"[35]"},{"why":"HD-Painter; a prompt-faithful text-guided inpainting baseline that PainterNet must beat on attention consistency.","marker":"[26]"},{"why":"Grounding DINO; the detector used to compute Gdino accuracy of local generation in PainterBench.","marker":"[22]"}],"fun_headline_variants":["PainterNet plug-in tops inpainting benchmarks for local text","Adaptive inpainting branch boosts text-local alignment","New plug-in makes inpainting follow user prompts precisely","Diverse-mask training helps inpainting match local captions","Attention-focused plugin beats prior inpainting on text fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that evaluating every method on PainterNet's own style of local prompts and user-like masks is a fair comparison, even though the baselines were trained on global prompts and segmentation masks, so PainterBench may systematically favor PainterNet.","fun_headline_variants_meta":{"raw":{"variants":["PainterNet plug-in tops inpainting benchmarks for local text","Adaptive inpainting branch boosts text-local alignment","New plug-in makes inpainting follow user prompts precisely","Diverse-mask training helps inpainting match local captions","Attention-focused plugin beats prior inpainting on text fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001061,"raw_usage":{"total_tokens":4485,"prompt_tokens":1014,"completion_tokens":3471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":3387}},"tokens_in":630,"tokens_out":3471,"duration_ms":20807,"temperature":1.0,"reasoning_tokens":3387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:32:44.909297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun Table 1 on the original BrushBench (global prompts, segmentation masks) and on a held-out set of unscripted user-drawn scribbles; if PainterNet's margins over BrushNet and HD-Painter shrink or reverse, the state-of-the-art claim rests on benchmark overlap with the training distribution rather than on a general architectural advantage.","supporting_citations":[{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"ControlNet; provides the zero-convolution control-branch insertion method and is a baseline."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"CLIP; the text encoder whose tokens define the 'actual tokens' in ATAL and whose similarity filters generated local prompts."},{"cited_title":"Chatglm: A family of large language mod- els from glm-130b to glm-4 all tools","cited_arxiv_id":null,"evidence_quote":"ChatGLM; post-processes the long local prompts into short object-specific captions."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion; the frozen base model that PainterNet's branch conditions, and its inpainting variant is a baseline."}],"review_version":1}