{"id":"182c0e08-1c00-42f7-a314-c79ba8f50dd9","arxiv_id":"2501.09194","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":12,"one_line_summary":"Combining ControlNet and GLIGEN, ObjectDiffusion conditions Stable Diffusion on bounding boxes and open-ended object names, reporting improved AP50, AR, and FID on COCO2017.","lead":"ObjectDiffusion is a text-to-image system that adds bounding-box and object-name control by fusing ControlNet's trainable-copy design with GLIGEN's grounding attention layers. The authors report state-of-the-art AP50, AR, and FID on COCO2017, but the comparison relies on published baseline numbers and a test-time model swap that is not ablated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SOTA numbers come from an inference-time hybrid that swaps the SD base used in training for pretrained GLIGEN; the paper provides no ablation isolating the trained GroundNet's contribution, so the central claim is unsubstantiated.","rationale":"The reader's weakest-assumption identification is exactly the load-bearing issue: the evaluated system is not the trained model. Section 4.1 states that the SD model is replaced with pretrained GLIGEN at inference, while all training was performed against a locked SD v1.4 backbone. Because GroundNet's zero-convolution outputs are added to the frozen backbone's features, and GLIGEN's backbone includes additional gated self-attention layers that alter those features, the transfer is not guaranteed and must be empirically validated. The paper provides no such validation: it reports only the hybrid's metrics and even remarks that GLIGEN 'yields more controllable images,' implying the training-time model was worse. This single omission is sufficient to reject the SOTA claim; secondary issues like copied baselines and FID protocol differences would also need correction, but the base swap is the primary defect. A controlled re-evaluation with the training-time base and GLIGEN-alone, under a unified detection and FID protocol, would settle whether the proposed method contributes anything beyond the pretrained GLIGEN base.","tokens_in":19560,"tokens_out":4428,"duration_ms":42139,"concrete_test":"Compute AP50, AR, and FID on COCO2017-val using the identical YOLOv8m detection pipeline and pytorch-fid (resize to 299x299) for three configurations: (A) SD-v1.4 base + trained GroundNet (the training-time model, Figure 2); (B) GLIGEN base + trained GroundNet (the inference-time model, Figure 4); and (C) pretrained GLIGEN alone with the same grounding inputs. If (A) is substantially below (B), or (C) matches (B), the headline gains come from the base swap, not from GroundNet. Additionally, run (B) with the zero-convolution outputs zeroed to measure GroundNet's marginal contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper trains ObjectDiffusion with a frozen Stable Diffusion v1.4 base (Section 3.3, Eq. 8), where the trainable GroundNet's zero-convolution outputs are added to the frozen SD features. At inference (Section 4.1, Inference; Figure 4), the authors replace the SD base with a pretrained GLIGEN network, which has its own injected gated self-attention layers and therefore produces different feature activations at the insertion points. GroundNet's additive corrections were calibrated to SD's feature statistics during 100k iterations of fine-tuning; applying them to GLIGEN's features is an uncontrolled distribution shift. The paper explicitly states that GLIGEN 'yields more controllable images,' confirming the swap changes behavior, yet it reports only the hybrid's metrics (AP50=46.6, AR=44.5, FID=19.8) and omits the training-time configuration entirely. Without an ablation comparing (i) SD base + GroundNet, (ii) GLIGEN base + GroundNet, and (iii) GLIGEN alone under the same evaluation protocol, the reported gains cannot be attributed to the proposed training; they may stem entirely from the pretrained GLIGEN base. This directly undermines the abstract's claim that ObjectDiffusion outperforms SOTA models.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ObjectDiffusion, a layout-to-image model that combines ControlNet-style zero-convolution injection with GLIGEN-style gated self-attention grounding. The authors freeze a Stable Diffusion v1.4 base and fine-tune a parallel GroundNet on COCO2017; at inference, however, they replace the frozen SD base with a pretrained GLIGEN network. They report AP50=46.6, AR=44.5, and FID=19.8 and claim state-of-the-art performance among models trained on open-source datasets, supported by qualitative closed-set and open-set examples.","tokens_in":19841,"tokens_out":9225,"duration_ms":86999,"significance":"If the quantitative claims were obtained by the proposed training method, the paper would offer a lightweight approach for adding bounding-box grounding to a frozen text-to-image model. The authors are transparent about their open-source evaluation and include a limitations section. However, the headline results are generated by a hybrid system that is not the model defined by the training objective, and the baseline numbers are not measured under the same protocol; the central empirical claim is therefore currently unsubstantiated.","major_comments":[{"comment":"The reported metrics are produced by replacing the Stable Diffusion base used during training with a pretrained GLIGEN network, while GroundNet's zero-convolution outputs were trained as additive corrections to SD features. Equation (8) defines the model as f_{θ,θ'} = f_θ(SD) + Z(f_θ'), but at inference f_θ is a GLIGEN network whose internal gated self-attention layers alter the feature activations at the injection points. The paper offers no ablation comparing (i) SD + GroundNet, (ii) GLIGEN + GroundNet, (iii) GLIGEN alone, and (iv) SD alone under the same evaluation protocol. Without such an ablation, the reported AP50=46.6, AR=44.5, and FID=19.8 cannot be attributed to the proposed fine-tuning; they may stem entirely from the pretrained GLIGEN base.","section":"§4.1 (Inference), §3.3 (Eq. 8), Figure 4"},{"comment":"The state-of-the-art comparison is not controlled. Tables 1, 3, and 4 state that baseline values are taken from [28], while the authors compute their own numbers with a pretrained YOLOv8m detector and the pytorch-fid implementation [56]. If GLIGEN's published numbers used a different detector or a different FID implementation and image preprocessing, part or all of the reported improvement may be an artifact of the evaluation pipeline. The preprocessing also differs from GLIGEN's: the authors resize with bicubic interpolation and do not crop, whereas GLIGEN center-crops (§4.1, Figure 3). The authors should rerun GLIGEN and GLIGEN (fine-tuned) under the exact same preprocessing, sampling, detection, and FID protocol before claiming superiority on AP50, AR, and FID.","section":"§4.1 (Evaluation Benchmarks), Tables 1 and 4"},{"comment":"The AR upper bound is reported as 49.7, the same value as the AP upper bound in Table 1. Average recall is a different evaluation statistic, and its upper bound should be derived from the detector's recall on the ground-truth boxes, not from the AP value. The statement that ObjectDiffusion's AR of 44.5 is 'only five points behind the upper bound' is therefore unsupported unless the upper bound is explicitly defined and recomputed.","section":"§4.2 (Table 3)"},{"comment":"The description of GroundNet's construction and initialization is internally inconsistent. Section 3.3 says the encoder and middle blocks are cloned from the frozen Stable Diffusion model and then augmented with gated self-attention layers, while Figure 2 and Section 4.1 state that GroundNet consists of, and is initialized from, the encoder and middle blocks of GLIGEN. Because GroundNet is the only trainable component of the proposed method, this ambiguity prevents readers from knowing which weights are actually fine-tuned and is a barrier to reproducibility.","section":"§3.3, Figure 2, §4.1 (Initialization)"}],"minor_comments":[{"comment":"The detector is referred to as YOLOv8m with citation [49], but [49] is the original YOLO paper; please cite the Ultralytics YOLOv8 implementation actually used.","section":"§4.1 (Evaluation Benchmarks)"},{"comment":"Reference [13] contains a malformed URL ('https://https://huggingface.co/gligen/diffusers-generation-text-box'); it should be corrected.","section":"References"},{"comment":"The sentence 'Our model increased the zero-shot AP score of GLIGEN (zero-shot) from 19.1 to 27.4' is misleading because ObjectDiffusion is fine-tuned on COCO; this is a comparison against a zero-shot baseline, not an improvement in zero-shot capability.","section":"§4.2 (Quantitative Results)"},{"comment":"AP/AR are computed on 5k generated images while FID uses 20k generated images; please clarify whether the 5k set is a subset of the 20k set and report variance over multiple random seeds or sample sets.","section":"§4.1 (Dataset)"}],"recommendation":"reject","confidential_remarks":"The inference-time base swap means the evaluated model is not the model defined by Equation (8), and every headline metric is affected. The controlled-baseline issue compounds this. If the authors can provide a clean evaluation in which the trained model is also the evaluated model, rerun all baselines under an identical protocol, and clarify GroundNet's initialization, a fresh submission of the underlying idea could be worth considering. The current manuscript, however, does not support its central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper combines ControlNet's trainable-copy setup with GLIGEN's gated self-attention grounding, injecting grounding tokens at every encoder scale and the middle block. That's a sensible incremental idea, and the paper is honestly written. But the headline SOTA numbers come from an inference-time model that swaps the frozen Stable Diffusion base used during training for a pretrained GLIGEN, and no ablation separates GroundNet's contribution from GLIGEN's. So the abstract's claim is not established.\n\nWhat's actually good: the architecture is clearly specified, training details are concrete, and the authors openly list failure cases (faces, hands, text, spatial relations). The multi-scale injection is a reasonable extension of ControlNet's single injection, and initializing GroundNet from a GLIGEN checkpoint is sensible. The related work covers the main baselines, and the citation pattern is appropriate.\n\nWhere it falls down: first, the inference swap. GroundNet was trained to add corrections to SD features. At inference it is attached to a GLIGEN base whose features differ because GLIGEN inserts its own gated attention. The paper even says GLIGEN gives more controllable images. The reported AP50 of 46.6 and FID 19.8 are from this hybrid, so we cannot tell whether the fine-tuned GroundNet helps, hurts, or is just along for the ride. You would need at least three runs—SD+GroundNet, GLIGEN alone, GLIGEN+GroundNet—under the same detector and FID pipeline. Second, the baselines are taken from the GLIGEN paper rather than re-run. Different detectors and FID implementations make the 10-point AP50 gap uninterpretable. Third, no code or checkpoints are released, so even the promising numbers cannot be independently reproduced. Minor: the sentence about increasing GLIGEN's zero-shot AP from 19.1 to 27.4 compares their model to a different model, which is misleading.\n\nNet: the method is plausible and the write-up is fine, but the central claim is unsubstantiated as submitted. It deserves a serious referee because the flaws are fixable through re-evaluation and ablations; a referee should insist on those before any acceptance. If GroundNet turns out to add little over GLIGEN alone, the contribution shrinks to a modest variant, which is still publishable somewhere but not as a SOTA result.","headline":"The SOTA claim rests on an unablated inference-time swap from SD to GLIGEN plus uncontrolled baselines; the architecture is a plausible incremental combination, but the numbers as reported don't support the headline.","tokens_in":20401,"tokens_out":3860,"would_cite":false,"duration_ms":38680,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ObjectDiffusion grounds text-to-image diffusion on bounding boxes, reporting AP50 46.6, AR 44.5, and FID 19.8, beating open-source-trained layout-to-image models on all three metrics.","keywords":["text-to-image generation","layout-to-image generation","bounding box grounding","diffusion models","ControlNet","GLIGEN","controllable image synthesis","open-set grounding"],"falsifier":"Run the exact same training procedure but keep the frozen Stable Diffusion base at inference (no swap to GLIGEN), then recompute AP50, AR, and FID on the same COCO2017 validation annotations; if the numbers collapse toward or below the GLIGEN fine-tuned baseline, the reported gains depend on the swapped base rather than on the GroundNet training.","tokens_in":1813,"feed_emoji":"🎯","tokens_out":2371,"duration_ms":71040,"temperature":0.7,"pith_summary":"ObjectDiffusion is a conditional image generation model that lets a pretrained text-to-image diffusion model place objects described in free-form text into specified bounding boxes. The authors claim that by combining the frozen-backbone architecture of ControlNet with the gated-attention grounding mechanism of GLIGEN, they get both high placement precision and high image quality. Fine-tuned on COCO2017 and evaluated on its validation split, the model reports AP50 of 46.6, AR of 44.5, and FID of 19.8, beating open-source-trained baselines on all three metrics. The point of the work is to show that an existing frozen diffusion base can be extended with grounding capability without retraining the base and without losing its generation knowledge.","feed_headline":"Box-guided diffusion model beats open-source SOTA on grounding","feed_subtitle":"Fusing ControlNet's frozen-backbone design with GLIGEN's grounding places described objects precisely in their boxes.","key_machinery":"GroundNet: a trainable parallel network formed from the encoder and middle blocks of the diffusion backbone, modified by inserting a gated self-attention layer (with tanh gating) into each attention block, and connected to the frozen base via 1x1 zero-initialized convolution layers. It processes a control layout g = [(e1,b1),...,(eN,bN)] where each entity ei is embedded by a frozen CLIP text encoder and each box bi by a Fourier positional embedding; an MLP fuses the two into a 768-dimensional token that the gated self-attention layers inject into the visual features. The zero-convolutions and the tanh gates protect the pretrained weights from the newly introduced grounding signal during early training.","core_discovery":"The central claim is that a trainable GroundNet—a cloned, modified copy of the diffusion model's encoder and middle blocks with GLIGEN-style gated self-attention layers folded in—can be attached to a frozen Stable Diffusion model through zero-initialized convolutions and fine-tuned on detection annotations, yielding a layout-to-image model whose grounding precision and image quality exceed the current state of the art among models trained on open-source data. The grounding input is a set of tokens pairing a CLIP-encoded open-ended entity description with a Fourier-embedded bounding box, fused by an MLP and injected at every encoder layer and the middle block. At inference the frozen base is swapped for a pretrained GLIGEN, which the authors say produces more controllable images; all reported numbers come from that hybrid.","pith_inferences":["A controlled comparison that keeps the frozen base fixed (Stable Diffusion throughout training and inference) would separate how much of the reported precision comes from the GroundNet fine-tuning and how much from swapping in GLIGEN at inference.","The large per-class AP spread (cat AP50 96.7 versus sports ball 10.9) suggests that reweighting the grounding loss or oversampling low-AP classes could lift overall precision without changing the architecture.","Because the grounding tokens are produced by a frozen CLIP encoder and injected through gated self-attention, the same recipe could likely be attached to other frozen text-to-image backbones, making box grounding a modular add-on rather than a per-model retraining task.","Testing the model with deliberately corrupted or partially dropped grounding tokens would reveal whether the 10% random-drop training makes it robust to missing boxes, which is directly relevant to real-world use."],"forward_implications":["Layout-to-image generation can be added to a frozen diffusion backbone by fine-tuning only a small parallel network, so the base model's broad generation knowledge is preserved.","The reported AP50 46.6 versus 42.3 for GLIGEN (fine-tuned) and AR 44.5 versus 30.7 suggest that multi-scale injection of grounding tokens reduces both object misplacement and object omission.","An FID of 19.8 versus 21.58 for GLIGEN (fine-tuned) indicates that image quality does not have to be sacrificed for control.","Open-set qualitative results imply that the grounding works beyond the 80 COCO categories, as long as the CLIP text encoder recognizes the entity phrase.","Initializing GroundNet from a pretrained GLIGEN checkpoint and fine-tuning only 460.8M of 1.32B parameters substantially reduces training cost."],"supporting_citations":[{"why":"Supplies the ControlNet architecture: freezing the pretrained diffusion backbone and training a parallel copy connected by zero convolutions; the paper modifies this design for grounding.","marker":"[74]"},{"why":"GLIGEN supplies the grounding method—gated self-attention layers, CLIP plus Fourier plus MLP grounding tokens—and the pretrained checkpoint used to initialize GroundNet.","marker":"[28]"},{"why":"Stable Diffusion is the frozen text-to-image base whose generation knowledge is preserved, and the source of the cloned encoder and middle blocks.","marker":"[50]"},{"why":"COCO2017 is the dataset used for fine-tuning and for all quantitative and qualitative evaluation.","marker":"[30]"},{"why":"The YOLOv8m detector is the pretrained object detector used to compute AP and AR on generated images.","marker":"[49]"},{"why":"FID is the metric used to measure image quality of generated samples.","marker":"[17]"},{"why":"CLIP ViT-L/14 is the frozen text encoder that turns open-ended entity descriptions into the semantic grounding tokens.","marker":"[46]"},{"why":"ReCo supplies the open-ended text-plus-box conditioning format that the paper adopts for entity descriptions.","marker":"[73]"},{"why":"The pretrained GLIGEN checkpoint provides the weight initialization for the trainable GroundNet, giving grounding knowledge without COCO fine-tuning.","marker":"[13]"},{"why":"The Fourier embedding is the positional encoding used to turn bounding-box coordinates into periodic features.","marker":"[34]"}],"fun_headline_variants":["ObjectDiffusion: ControlNet and GLIGEN fusion tops open-source SOTA","Box-guided diffusion beats open-source benchmarks with fused architecture","Hybrid model improves grounding and image quality over open-source SOTA","ObjectDiffusion achieves record AP and FID with precise box control","Combining ControlNet and GLIGEN: new SOTA for layout-to-image"],"cache_read_input_tokens":22400,"weakest_assumption_plain":"The reported metrics all come from a hybrid where the fine-tuned GroundNet is attached to a pretrained GLIGEN base at inference instead of the Stable Diffusion base it was trained with; if the zero-convolution injections trained against Stable Diffusion do not transfer to GLIGEN's own grounding pathway, the numbers would not reflect the proposed training method.","fun_headline_variants_meta":{"raw":{"variants":["ObjectDiffusion: ControlNet and GLIGEN fusion tops open-source SOTA","Box-guided diffusion beats open-source benchmarks with fused architecture","Hybrid model improves grounding and image quality over open-source SOTA","ObjectDiffusion achieves record AP and FID with precise box control","Combining ControlNet and GLIGEN: new SOTA for layout-to-image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1930,"prompt_tokens":984,"completion_tokens":946,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":852}},"tokens_in":600,"tokens_out":946,"duration_ms":9665,"temperature":1.0,"reasoning_tokens":852,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:12.665278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact same training procedure but keep the frozen Stable Diffusion base at inference (no swap to GLIGEN), then recompute AP50, AR, and FID on the same COCO2017 validation annotations; if the numbers collapse toward or below the GLIGEN fine-tuned baseline, the reported gains depend on the swapped base rather than on the GroundNet training.","supporting_citations":[{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the ControlNet architecture: freezing the pretrained diffusion backbone and training a parallel copy connected by zero convolutions; the paper modifies this design for grounding."},{"cited_title":"Gligen: Open-set grounded text-to-image generation","cited_arxiv_id":null,"evidence_quote":"GLIGEN supplies the grounding method—gated self-attention layers, CLIP plus Fourier plus MLP grounding tokens—and the pretrained checkpoint used to initialize GroundNet."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion is the frozen text-to-image base whose generation knowledge is preserved, and the source of the cloned encoder and middle blocks."},{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"COCO2017 is the dataset used for fine-tuning and for all quantitative and qualitative evaluation."},{"cited_title":"You only look once: Unified, real-time object de- tection","cited_arxiv_id":null,"evidence_quote":"The YOLOv8m detector is the pretrained object detector used to compute AP and AR on generated images."},{"cited_title":"Gans trained by a two time-scale update rule converge to a local nash equilib- rium","cited_arxiv_id":null,"evidence_quote":"FID is the metric used to measure image quality of generated samples."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"CLIP ViT-L/14 is the frozen text encoder that turns open-ended entity descriptions into the semantic grounding tokens."},{"cited_title":"https : / / https : / / huggingface","cited_arxiv_id":null,"evidence_quote":"The pretrained GLIGEN checkpoint provides the weight initialization for the trainable GroundNet, giving grounding knowledge without COCO fine-tuning."}],"review_version":1}