{"id":"e9803f02-0edc-4bc3-9194-92638f263bd9","arxiv_id":"2506.23482","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"MTADiffusion improves text-guided object inpainting by training on a new 5M-image mask-text dataset with edge prediction and style-consistency losses.","lead":"MTADiffusion is a diffusion model for filling in masked parts of images with objects described by text prompts. It combines a large automatically annotated dataset with edge and style losses to make the generated objects more semantically accurate, structurally stable, and stylistically consistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The style-consistency loss (Eq. 7, delta=100) is unvalidated: VGG is applied to 4-channel latents, and Sections 3.4/4.1 ambiguously train the 'style extractor,' risking a meaningless or gameable objective.","rationale":"The reader's weakest_assumption identifies the VGG-on-latent transfer as a key risk; I agree that this is a serious concern. However, I find an even more concrete internal inconsistency: the paper first calls the VGG 'pre-trained' and later explicitly trains the 'VGG latent style extractor' with a learning rate of 1e-7. If the extractor is trainable, the Gram-matrix loss is not a fixed perceptual objective and can be gamed, which would undermine the claimed style-consistency contribution regardless of whether VGG features transfer. The large weight (delta=100) makes this loss dominant, so any failure mode is amplified. The paper does provide useful evidence elsewhere: Table 3 (dataset ablation) and Table 4 (caption model comparison) partially support the MTADataset contribution, and the user study is a real if small-scale evaluation. But the proposed style loss is never ablated in the main text, and the edge-prediction loss also lacks a main-text ablation. These gaps in verification are consistent with the reader's CONDITIONAL verdict; my concern does not change that verdict, but it sharpens the condition: the authors must clarify the VGG's trainability and provide the missing loss ablations before the style-consistency claim can be accepted.","tokens_in":11881,"tokens_out":10420,"duration_ms":96864,"concrete_test":"Run the full training protocol (Section 4.1) three times: (a) with delta=100 as reported, (b) with delta=0, and (c) with delta=100 but a frozen ImageNet-pretrained VGG instead of the trainable 'style extractor.' Compare style-consistency user-study votes or a style metric (e.g., LPIPS on masked area vs. background) on BrushBench. If (b) and (c) do not statistically differ from (a) on style, the style loss is not the cause of the reported style gains; if (c) differs from (a), the frozen/trainable ambiguity is material to the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim depends on the style-consistency loss as one of the three proposed improvements (Abstract; user study in Table 5 gives style consistency 54% vs. 22% for SDI). The loss in Eq. 7 compares Gram matrices of VGG features of the predicted latent X_{t-1} (Eq. 5) and the ground-truth noised latent X\\tilde_{t-1} (Eq. 6) at each training timestep. Two unaddressed issues make this objective unreliable: (1) The VGG's first conv layer is expanded from 3 to 4 channels to match latent space (Section 4.1), but there is no evidence that ImageNet-trained VGG features transfer to VAE latents, which have different statistics and are noisy at intermediate steps. (2) The paper is internally inconsistent about whether the VGG is frozen: Section 3.3.2 says 'pre-trained VGG network,' but Section 3.4 says the 'VGG latent style extractor' is trained (learning rate 1e-7 in Section 4.1). If the VGG is trainable, the Gram loss can be minimized by the extractor collapsing to trivial features, not by improving style. With delta=100, gamma=1, eta=0.1 (Section 4.1), this loss dominates the total loss, so if it is not measuring style, it can actively distort the denoising process. No main-text ablation of the style loss is presented; the paper directs readers to supplementary ('See also supplementary material for the ablation studies of different losses'), but the supplied text does not include those results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MTADiffusion, a text-guided object inpainting model with three main contributions: (i) an automatic annotation pipeline (MTAPipeline) and a large dataset (MTADataset, 5M images and 25M mask-text pairs), (ii) a multi-task training strategy that jointly predicts noise and Sobel edge maps, and (iii) a style-consistency loss computed with a VGG network and Gram matrices on denoised latent maps. The model is evaluated on BrushBench and EditBench against four baselines (SDI, PowerPaint, CNI, BrushNet), and the authors claim state-of-the-art performance in semantic alignment, structural stability, and style consistency.","tokens_in":12311,"tokens_out":4100,"duration_ms":42839,"significance":"If fully validated, the paper would provide a valuable large-scale mask-text annotation resource for inpainting and a set of training objectives that could generalize beyond the specific architecture. The dataset scale (5M images, 25M mask-text pairs) is a substantial contribution, and the idea of jointly training edge prediction with inpainting is plausible. However, the central SOTA claim is currently supported only by small or inconsistent metric differences, and the style-consistency loss is not adequately validated. The ablation study for the losses is deferred to a supplementary that is not included in the submitted text, which makes the individual contributions of the proposed components impossible to assess.","major_comments":[{"comment":"The style-consistency loss is a core contribution, but its validity is not established. The first VGG convolutional layer is expanded from 3 to 4 channels (Section 4.1) and applied to noisy latent maps X_{t-1} and X~_{t-1} at intermediate denoising steps, yet no evidence is given that ImageNet-pretrained VGG features remain meaningful for VAE latents at those noise levels. Moreover, the manuscript is internally inconsistent about whether the VGG is frozen: Section 3.3.2 calls it a 'pre-trained VGG network,' while Section 3.4 says the 'VGG latent style extractor' is trained and Section 4.1 gives it a learning rate of 1e-7. If the extractor is trainable, minimizing the Gram loss in Eq. (7) can be trivially achieved by feature collapse, not by improving style consistency. With delta=100 in Eq. (8), this unvalidated objective dominates the total loss and could actively distort denoising. No main-text ablation isolates the effect of L_style; the pointer in Section 5 to supplementary material for loss ablations is unfulfilled in the submitted text. This issue is load-bearing because the style-consistency loss is one of the three named contributions in the Abstract and is credited for the style gains in the user study.","section":"Section 3.3.2/3.4/4.1, Eq. (5)-(8)"},{"comment":"The state-of-the-art claim is not supported by the quantitative tables. On BrushBench, the gains over BrushNet are extremely small: PSNR 31.87 vs 31.82, LPIPS 18.94 vs 18.95, and MSE 0.80 vs 0.82. On EditBench, the proposed method is worse than BrushNet on PSNR (33.31 vs 33.66) and MSE (0.65 vs 0.63), and essentially tied on LPIPS. The claimed superiority rests entirely on IR, CLIP, and VQA scores. The paper reports no error bars, no multiple-seed results, and no significance tests. Given the small and sometimes contradictory margins, the abstract's 'state-of-the-art performance' claim is overstated and should either be supported with statistical evidence or appropriately qualified.","section":"Tables 1 and 2"},{"comment":"The multi-task edge prediction is a central mechanism for the claimed structural stability improvement, but no ablation demonstrates its contribution. The only ablation pointer in Section 5 ('See also supplementary material for the ablation studies of different losses') is not backed by content in the submitted manuscript. Since the user study attributes structural stability to the full system, the individual effects of L_structure and of the proposed dataset cannot be disentangled. The authors should provide main-text ablations that remove or reweight each loss component, and ideally also isolate the dataset contribution, to substantiate the structural-stability claim.","section":"Section 3.2 and Section 5"}],"minor_comments":[{"comment":"The summation index i is reused for both the batch and the style-feature count; the second sum should use a different index (e.g., j) for the N style features.","section":"Eq. (7)"},{"comment":"The phrase 'the outputs of convi 1, where the i = (1, . . . ,5) layers' is unclear; it should read 'the outputs of the conv1_i layers for i = 1, ..., 5'.","section":"Section 4.1"},{"comment":"Table 1 reports a VQA Score but Table 2 does not; please either add the same metric to both tables or explain the omission.","section":"Tables 1 and 2"},{"comment":"The user study reports only aggregate vote percentages; providing per-image vote distributions or confidence intervals would strengthen the claim that the differences are meaningful.","section":"Section 4.6"},{"comment":"The statement that each image contains 'approximately 5 masks' and the dataset contains '25 million mask-text pairs' is consistent only if exactly 5 masks per image is the average; please clarify that this is an average, not an exact count.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claims depend on the MTADataset and the proposed losses, but the submitted version does not include the supplementary ablation material and gives no indication of code or data release. For a systems paper, the small and inconsistent metric differences against a single strong baseline (BrushNet) need more careful statistical treatment. I would be willing to re-review a revised version that includes the missing ablations and clarifies the style extractor's training status."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The MTADataset and the MTAPipeline are the strongest parts of this paper. A 5M-image, 25M-mask-text-pair dataset built with Grounded-SAM and LLaVA is a substantial engineering contribution, and the controlled comparisons in Tables 3 and 4 give real evidence that the dataset and the detailed captions drive the measured improvements. The user study, while not beyond methodological quibbles, is an honest attempt to evaluate the three claimed axes. The multi-task edge prediction is a reasonable idea, and the paper is transparent about the known limitation that captions can be inaccurate.\n\nThe soft spot is the style-consistency loss. The paper is genuinely ambiguous about whether the VGG extractor is frozen: Section 3.3.2 calls it pre-trained, but Section 3.4 and the implementation details say it is trained, with a learning rate of 1e-7. If it is trainable at any rate, the Gram loss can be minimized in part by the extractor distorting its own features, and there is no main-text ablation to show the loss even helps. Applying ImageNet VGG to 4-channel VAE latents at intermediate denoising steps is a strong assumption that needs validation. With delta=100, this unvalidated loss dominates the total objective, so the concern is not minor.\n\nThat said, the central SOTA claim rests more on the dataset than on this loss. The metric gaps over BrushNet on BrushBench are small but mostly positive; on EditBench the PSNR and MSE actually go the wrong way. No error bars, no significance tests, and no release of code or data. A reader should treat the SOTA phrasing with caution.\n\nFor a field that increasingly relies on large auto-annotated datasets, this paper is worth refereeing. It needs a revision that adds ablations for each loss component, clarifies the VGG training status, and reports variance. I would not desk-reject it.","headline":"The dataset and annotation pipeline are the real contribution; the style-consistency loss is under-validated and the SOTA claim is thinner than the abstract suggests, but the paper deserves a serious referee.","tokens_in":12813,"tokens_out":2129,"would_cite":false,"duration_ms":24032,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MTADiffusion claims that combining automatically annotated mask-text pairs, a joint edge-prediction task, and a VGG Gram-matrix style loss yields state-of-the-art object inpainting on BrushBench and EditBench.","keywords":["object inpainting","text-guided diffusion","mask-text alignment","edge prediction","style consistency","Gram matrix","VGG feature loss","dataset annotation pipeline"],"falsifier":"Train the same model with $L_{\\text{style}}$ removed or replaced by a style loss computed on decoded clean images, and check whether the user-study style-consistency votes and the reported gains collapse; if they do not, the latent Gram-loss mechanism is not the carrier of the claim.","tokens_in":11693,"feed_emoji":"🎨","tokens_out":4493,"duration_ms":39982,"temperature":0.7,"pith_summary":"The paper claims that object inpainting can be made simultaneously more semantically accurate, structurally stable, and stylistically consistent by combining three ingredients: a large automatically annotated dataset of masks with detailed text descriptions, a multi-task training objective that also predicts edge maps, and a Gram-matrix style loss computed on VGG features of intermediate diffusion latents. These ingredients are assembled in MTADiffusion, a dual-branch diffusion model built on Stable Diffusion and the BrushNet architecture. Evaluated on BrushBench and EditBench, the method reports state-of-the-art results on image quality, masked-region preservation, and text-alignment metrics, plus a user study favoring its outputs.","feed_headline":"Auto-annotated masks plus edge and style losses beat prior inpainting","feed_subtitle":"MTADiffusion tops BrushBench and EditBench with 25M mask-text pairs, edge prediction, and VGG style loss.","key_machinery":"The architecture is a UNet branch (Stable Diffusion) plus a brush branch whose multi-resolution self-attention blocks take the noisy latent, masked image latent, and downsampled mask, feeding into each UNet layer through zero convolutions. The brush branch's final layer predicts a Sobel-edge map, adding the structural loss $L_{\\text{structure}}$. For style, predicted and ground-truth latents at timestep $t{-}1$, obtained via the denoise and noise-adding functions, are passed through a VGG network whose first layer is expanded from 3 to 4 input channels, and $L_{\\text{style}}$ compares Gram matrices of VGG features, weighted by $\\delta = 100$.","core_discovery":"On the paper's own terms, MTADiffusion establishes that the missing ingredient in text-guided object inpainting is tight alignment between the mask and a detailed description of the object inside it, rather than a caption of the whole image. The authors build MTAPipeline to produce such pairs at scale: Grounded-SAM extracts masks and labels, LLaVA writes content-and-style descriptions, and the resulting 5-million-image, 25-million-pair MTADataset is used to train a dual-branch diffusion model alongside an edge-prediction head and a VGG-based latent style loss. The combined objective $L = \\gamma L_{\\text{noise}} + \\delta L_{\\text{style}} + \\eta L_{\\text{structure}}$ yields gains on both object-mask and random-mask benchmarks.","pith_inferences":["The same MTAPipeline could be applied to other region-level tasks such as image harmonization or object replacement, where local text-style grounding matters.","Because caption quality is named as the bottleneck, upgrading LLaVA to a reasoning-stronger vision-language model may improve color and attribute fidelity more than further scaling the data.","A risk the paper does not analyze is that the 4-channel VGG input adaptation may distort early-layer features; a calibration study comparing channel-expansion strategies would test this directly."],"forward_implications":["Automatic mask-text annotation can substitute for human-labeled inpainting data, making large-scale training practical.","Auxiliary edge prediction strengthens structure without requiring edge maps at inference time.","Latent-space style loss with VGG Gram matrices improves style consistency without extra decoding steps.","The training strategies are model-agnostic and can be lifted into other inpainting or editing models."],"supporting_citations":[{"why":"Supplies the dual-branch architecture that MTADiffusion adapts and the BrushBench benchmark used for evaluation.","marker":"[9]"},{"why":"Generates the masks, labels, and bounding boxes that feed the annotation pipeline.","marker":"[24]"},{"why":"Writes the detailed content-and-style captions that form the mask-text pairs.","marker":"[15]"},{"why":"Provides the 5 million source images from which masks and captions are produced.","marker":"[28]"},{"why":"Serves as the frozen base generative model whose UNet is extended by the brush branch.","marker":"[25]"},{"why":"Provides the pretrained features used to compute the Gram-matrix style loss in latent space.","marker":"[29]"},{"why":"Provides the EditBench evaluation set with random masks and captions.","marker":"[34]"}],"fun_headline_variants":["Aligned mask-text pairs improve object inpainting","Auto-annotated mask-text data boosts inpainting","Edge and style losses refine diffusion inpainting","MTADiffusion: align masks to detailed captions","25M mask-text pairs for text-guided inpainting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The style-consistency loss is trusted to work on 4-channel noisy diffusion latents through a VGG network pretrained on clean 3-channel RGB images, and that trust carries a weight of 100 in the final loss.","fun_headline_variants_meta":{"raw":{"variants":["Aligned mask-text pairs improve object inpainting","Auto-annotated mask-text data boosts inpainting","Edge and style losses refine diffusion inpainting","MTADiffusion: align masks to detailed captions","25M mask-text pairs for text-guided inpainting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1601,"prompt_tokens":900,"completion_tokens":701,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":623}},"tokens_in":516,"tokens_out":701,"duration_ms":7511,"temperature":1.0,"reasoning_tokens":623,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:40:37.734045+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model with $L_{\\text{style}}$ removed or replaced by a style loss computed on decoded clean images, and check whether the user-study style-consistency votes and the reported gains collapse; if they do not, the latent Gram-loss mechanism is not the carrier of the claim.","supporting_citations":[{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Writes the detailed content-and-style captions that form the mask-text pairs."},{"cited_title":"Laion-5b: An open large-scale dataset for training next generation image-text models","cited_arxiv_id":null,"evidence_quote":"Provides the 5 million source images from which masks and captions are produced."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Serves as the frozen base generative model whose UNet is extended by the brush branch."},{"cited_title":"Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting","cited_arxiv_id":null,"evidence_quote":"Provides the EditBench evaluation set with random masks and captions."}],"review_version":1}