{"id":"c9ed80df-4774-4bee-9db5-cb14bd289a3e","arxiv_id":"2505.05678","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A training-free pipeline that uses attention-derived instance masks and LLM-assigned per-segment instructions to improve multi-object, multi-attribute text-to-image generation.","lead":"InstanceGen is a training-free way to make text-to-image models follow complex prompts with multiple objects, attributes, and spatial relations. It starts from a rough first image, segments the objects in it, asks a language model to assign each segment, then re-generates the image under those assignments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests mainly on a self-built GPT-4o-judged benchmark with no human validation, no error bars, and no independently usable release; the anchor-point assumption is a real but secondary risk.","rationale":"The reader's weakest_assumption focuses on the anchor-point guarantee, which is an honest and clearly stated empirical assumption but is partly mitigated by seed search and instance copying, and is not the main pillar of the SOTA claim. The reader's rationale does mention the LLM judge and missing artifacts, and those are where I see the more load-bearing concern: the largest reported gains are on a self-constructed benchmark scored by an LLM judge with no human grounding and no variance estimates. This concern does not overturn the paper: GenEval and DrawBench provide independent evidence, the ablations are informative, and the limitations section candidly describes known failure modes. However, the specific margin claimed on CompoundPrompts should not be treated as established until a human-validated, reproducible evaluation is available. Since the reader already returned CONDITIONAL, my stress-test does not change the verdict; it sharpens the condition: before accepting the headlinenumbers, run the human-judge check and release the benchmark and evaluation harness. I would not move to REJECT because the method is plausible, the mechanism is well described, and the independent GenEval result supports at least part of the claim. I would not move to ACCEPT because the central benchmark result is currently unverifiable from the paper alone.","tokens_in":19778,"tokens_out":4280,"duration_ms":52199,"concrete_test":"Recruit three independent human annotators to answer the tier-specific yes/no questions on a stratified random sample of 120 images per method from CompoundPrompts (40 per tier), with images blinded to method. Compare human-accuracy gaps against the GPT-4o gaps and report two-sided 95% confidence intervals bootstrapped over unique prompts and seeds. If the InstanceGen-versus-Emu gap on Tiers B and C is not reproduced by human judgments (e.g., shrinks below 0.05), or if human-GPT-4o agreement on individual questions is below roughly 80%, then Table 2's headline gap is not established and the central claim should be downgraded from 'significantly outperforms' to 'may improve on some cases'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim — that InstanceGen significantly outperforms prior models on prompts compounding object counts, instance-level attributes, and spatial relations — is carried by Table 2: VQA Accuracy 0.60 versus 0.43 for Emu, with the largest gaps on Tier B (0.57 vs 0.40) and Tier C (0.50 vs 0.30). These numbers are produced by GPT-4o answering yes/no questions on CompoundPrompts, a benchmark designed by the same group that proposes the method. There is no human validation of the judge, no per-prompt variance or confidence intervals, and no public code, evaluation harness, or independently reusable form of the benchmark. Because the question templates are derived from the prompts, a systematic GPT-4o bias toward 'yes' for template-consistent or spatially tidy images would inflate InstanceGen's scores more than baseline scores, and the single-point estimates cannot reveal such a bias. GenEval (Table 6) provides independent support and is a genuine point in the paper's favor, but its categories do not cover the Tier B/C combination of instance-level attributes with spatial relations that motivates the method. The anchor-point assumption in Section 3.1.2 is a real additional risk, but seed search and instance copying are explicit fallbacks, and the qualitative results show the pipeline often succeeds even when segmentation is imperfect; it is not the single most load-bearing issue. The load-bearing issue is that the headline advantage is measured by an unvalidated automated judge on an unreleased benchmark, so the claimed gap is not yet independently checkable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"InstanceGen is a training-free, inference-time method for text-to-image generation aimed at prompts that combine object counts, instance-level attributes, and spatial relations. The pipeline first generates an initial image with a pretrained diffusion model (Emu), extracts anchor points from aggregated cross-attention maps, and constructs a fine-grained instance segmentation by combining Mask R-CNN with SAM2. A Llama-based LLM parses the prompt and assigns each segment an object label plus optional instance attributes. The final image is produced by a conditioned diffusion stage that optimizes latents against cross-attention losses, applies attention masking, and regularizes the background against the initial image. The paper also introduces CompoundPrompts, a 540-prompt benchmark with three difficulty tiers, and reports experiments on DrawBench, GenEval, and CompoundPrompts, claiming state-of-the-art performance on the new benchmark.","tokens_in":20070,"tokens_out":5472,"duration_ms":65238,"significance":"If the empirical claims hold, InstanceGen is a meaningful step for a genuine failure mode of text-to-image models: it replaces coarse LLM-generated bounding boxes with attention-derived instance masks, uses per-segment text conditioning, and explicitly targets counting, attribute-binding, and spatial errors. The independent GenEval results (Table 6) and DrawBench results (Table 1) are real points in the paper's favor, as are the detailed implementation hyperparameters and the candid limitations section. The central caveat is that the headline comparison on CompoundPrompts is not yet independently verifiable: the benchmark is self-constructed, GPT-4o is both part of the pipeline and the judge, and no human validation or uncertainty quantification is reported. The anchor-point assumption is a secondary but real correctness risk. These issues are addressable with additional experiments and a corrected presentation, so the contribution is potentially solid.","major_comments":[{"comment":"The central SOTA claim rests on VQA Accuracy numbers produced by GPT-4o answering yes/no questions on CompoundPrompts, a benchmark constructed by the same authors, with no human validation of the judge, no confidence intervals or significance tests, and no public release of the benchmark or evaluation harness. A systematic evaluator bias, for example toward spatially tidy layouts or toward template-consistent compositions, could inflate the reported 0.60 versus 0.43 gap. Please add a human-validated subset with inter-annotator agreement, report per-prompt variance and paired significance tests for the main comparisons, and release the benchmark and evaluation code; without these, the phrase \"significantly surpasses\" in Section 4.3 is stronger than the evidence supports.","section":"Section 4.3, Table 2"},{"comment":"Equation (3) as printed multiplies the cross-attention value by delta=-1.5 when m_i=1, i.e., inside the segment assigned to the word, and leaves the value unchanged when m_i=0, i.e., outside the segment. This is the opposite of the stated goal of masking attention in segments to which the word is not assigned, and negative attention weights are not explained anywhere in the text. Since the ablation in Table 3 shows that attention masking has a strong effect on Tier C performance, the implementation of this component is load-bearing. Please correct the condition (presumably m_i=0) or, if the negative multiplier is intentional, give a precise account of how negative attention values are used by the diffusion backbone.","section":"Section 3.3.2, Eq. (3)"},{"comment":"The completeness of the proposed layout generation relies on the empirical claim that \"every object instance produces at least one such anchor point\" in the aggregated cross-attention map, but no failure-rate statistics are reported. When this assumption fails, the method falls back to seed search and random instance copying, which the paper itself describes as degrading layout quality and visual plausibility. Please report, on a sample of CompoundPrompts prompts, the fraction of object instances that yield no anchor point, the success rate of the seed search, and the frequency with which instance copying is needed, together with its effect on final VQA Accuracy.","section":"Section 3.1.2 and Supplementary B.1.3"},{"comment":"The ablation study is difficult to interpret as reported. The \"Ours\" row (0.76/0.72/0.60/0.69) does not match the full-benchmark result in Table 2 (0.72/0.57/0.50/0.60), because a \"representative subset\" is used, but the subset is not defined. In addition, \"w/o Seed Search\" improves Tier A from 0.76 to 0.84 and leaves VQA Sim unchanged at 0.90, which undermines the text's conclusion that this component is \"not the primary factor driving our increased performance.\" Without confidence intervals or a significance test, the reader cannot distinguish real ablation effects from noise. Please report ablations on the identical full benchmark with variance estimates.","section":"Section 4.4, Table 3"}],"minor_comments":[{"comment":"The phrase \"adhere to all parts of the text prompt\" overstates the quantitative results, since the full-benchmark VQA Accuracy in Table 2 is 0.60 and Fig. 9 documents failures on each tier; \"better adherence\" would be more precise.","section":"Abstract and Section 1"},{"comment":"The supplementary results folder is described as containing images for \"all150 prompts\" in CompoundPrompts, while the main text says the benchmark contains 540 prompts; please reconcile this discrepancy.","section":"Supplementary C.3"},{"comment":"The DrawBench evaluation is described only as using \"the procedure used in [Phung et al. 2024]\"; please specify which prompt subset was used and how the counting and spatial accuracies were computed, so that the numbers in Table 1 can be reproduced.","section":"Section 4.3, Table 1"},{"comment":"The caption states that the initial images are \"post our robust initialization stage,\" but the seed search stage only selects a seed and does not edit the image; please clarify what post-processing was applied to the displayed initial images.","section":"Fig. 7 caption"},{"comment":"The json layout summary labels attention scores as \"object_probabilities\" and \"attribute_probabilities,\" even though they are averaged cross-attention values and not probabilities; please rename these fields to avoid misleading readers.","section":"Supplementary B.1.3"}],"recommendation":"major_revision","confidential_remarks":"The GenEval and DrawBench results give me enough confidence that the method itself is not a fabrication, so I would not reject the paper. However, because CompoundPrompts is presented as a contribution and the headline claim is measured on it, I would make public release of the benchmark and evaluation harness a condition of acceptance, and I would require the authors to resolve the Eq. (3) inconsistency before the paper can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"InstanceGen is a well-engineered inference-time method for compositional text-to-image prompts. The new part is the close coupling of attention-derived instance masks from an initial generation with LLM assignment of per-segment objects and attributes, followed by attention-based optimization of the latent. That combination is not in CountGen or SLD, and the paper makes a credible case that fine-grained structural anchors beat bounding boxes. The ablations are real and informative, and Section C.5 honestly reports failure modes. Credit where it is due: this is a serious piece of work.\n\nThe soft spots are in the evidence for the headline claim. Table 2, the core result, is VQA Accuracy scored by GPT-4o on CompoundPrompts, a benchmark the authors constructed. There are no error bars, no human validation of the judge, and no public release of the benchmark or code. The Tier B/C gaps (0.57 vs 0.40; 0.50 vs 0.30) could plausibly be inflated by a systematic 'yes' bias in GPT-4o for template-consistent or spatially tidy images; the single-point estimates cannot rule that out. GenEval is a genuine independent check, but its categories only partially overlap with the combined attribute-plus-spatial case that motivates the method.\n\nThe anchor-point assumption is a real secondary risk, but the paper has fallbacks and the qualitative results show the pipeline often works despite imperfect segmentation. The larger issue is the overclaim: “adhere to all parts of the text prompt” does not follow from a 0.60 VQA Accuracy, which means 40% of prompts still fail some question. The abstract and conclusion should be tempered.\n\nThis paper deserves a serious referee: the method is novel enough and the results promising enough to justify referee time. The referee should require release of the benchmark and evaluation harness, human-agreement numbers for the GPT-4o judge, error bars, and a revised claims section. With those changes, it would be a solid contribution. For now, treat the SOTA claim as conditional.","headline":"A well-engineered inference-time method for compositional prompts that is worth a serious referee, but its headline SOTA claim rests on a self-built GPT-4o-judged benchmark with no error bars or human validation.","tokens_in":20622,"tokens_out":2696,"would_cite":true,"duration_ms":26867,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InstanceGen couples attention-derived instance segmentation with LLM-written per-segment instructions to make text-to-image models obey object counts, attributes, and spatial relations without retraining.","keywords":["text-to-image generation","diffusion models","instance-level attributes","attention maps","layout guidance","prompt fidelity","object counting","CompoundPrompts benchmark"],"falsifier":"Run the pipeline on prompts with many small, partially occluded instances, such as ten birds on a branch with several behind leaves, and record how often the number of anchor points in the initial image falls below the requested count across many seeds. A setup with a high miss rate would show that the anchor-point premise, and therefore the layout quality, does not hold in the regime the method targets.","tokens_in":19558,"feed_emoji":"🎨","tokens_out":8072,"duration_ms":80673,"temperature":0.7,"pith_summary":"This paper claims that a text-to-image model can follow complex prompts—exact object counts, attributes tied to specific instances, and spatial relations—without retraining and without user-supplied bounding boxes, by using its own first draft as the structural guide. The proposed pipeline, InstanceGen, generates an initial image, extracts an instance-level segmentation from the cross-attention maps (local maxima serve as anchor points for each object instance), and asks an LLM to assign each segment an object label and optional attributes. It then regenerates the image while applying attention losses, attention masking, and a background-preserving term so the final image realizes the assignment while staying close to the initial composition. On the introduced CompoundPrompts benchmark, the paper reports 0.60 VQA Accuracy against 0.43 for the underlying Emu model, with the largest gains on prompts that compound attributes and spatial arrangements.","feed_headline":"Attention anchors lift complex-prompt accuracy from 43% to 60%","feed_subtitle":"The method turns attention peaks into labeled regions and regenerates, beating base models on multi-object prompts.","key_machinery":"The load-bearing object is the anchor point: a local maximum of the aggregated cross-attention map for an object word, taken as evidence that one instance of that object occupies that image region. Anchor points convert the base model's own attention signal into segmentation seeds, letting Mask R-CNN and SAM2 masks be filtered, merged, and assigned to object words and attributes. Everything downstream—LLM assignment, attention losses, and attention masking—assumes these anchor points localize every requested instance.","core_discovery":"The central claim is that a fine-grained structural signal with instance-level semantics can be obtained from the base diffusion model itself and fed back into that same model to fix complex-prompt adherence. InstanceGen first renders an initial image and aggregates cross-attention maps; local maxima of those maps serve as anchor points for Mask R-CNN and SAM2 segments, producing a layout of instance masks. An LLM, given the parsed prompt and per-segment attention scores, assigns each segment an object word and any instance attributes, deleting segments that do not fit. A second diffusion pass then optimizes latents with a weighted object-attention binary cross-entropy loss, an attribute cross-entropy loss, attention masking to stop semantic leakage, and a background-preserving regularization, so the output stays close to the initial composition while conforming to the layout instructions. The paper reports that this yields the strongest results among tested baselines on CompoundPrompts, DrawBench, and GenEval, with the largest gains on attribute and spatial tiers.","pith_inferences":["Because InstanceGen never retrains the base model, its ceiling should rise as base diffusion models improve; better attention maps would make anchor-point detection more reliable and reduce the need for seed search and mask copying.","The anchor-point premise implies a bottleneck at attention resolution: scenes with many small or overlapping instances may exhaust the supply of distinct peaks, so a direct test of that regime would map the method's practical limits.","The paper's front/back failure hints that injecting depth or occlusion ordering into the LLM assignment prompt could resolve spatial instructions that 2D segment coordinates cannot express.","CompoundPrompts-style evaluation with an MLLM judge could become a reusable stress test for compositional fidelity, but its scores will be sensitive to the choice of judge model and question wording."],"forward_implications":["Users can obtain fine-grained multi-object layouts from a single text prompt without drawing boxes, because the base model itself supplies the layout.","Instance-level attributes and spatial relations, the weakest spot of current text-to-image models, become addressable at inference time and can be bolted onto any diffusion model that exposes cross-attention maps.","Attention masking is doing much of the spatial work: removing it drops Tier C accuracy from 0.60 to 0.32 on the paper's ablation subset.","The background-preservation loss trades a little prompt fidelity for much better composition quality, per the ablation's VQA Similarity results.","CompoundPrompts gives the community a three-tier stress test so future methods can be compared separately on counting, attribute binding, and spatial fidelity."],"supporting_citations":[{"why":"Serves as the pretrained text-to-image model for both the initial image and the final regeneration, and as the strongest baseline on CompoundPrompts.","marker":"[Dai et al. 2023]"},{"why":"Supplies the object attention loss, self-attention background masking, and the idea of deriving layouts from attention, which InstanceGen extends to attributes.","marker":"[Binyamin et al. 2024]"},{"why":"Provides the bounding-box-guided Attention Refocusing baseline and the DrawBench evaluation procedure used for comparison.","marker":"[Phung et al. 2024]"},{"why":"Gives the initial object instance masks that are filtered by anchor point association in the layout segmentation stage.","marker":"[He et al. 2017]"},{"why":"Supplies keypoint-based SAM2 segmentation that recovers instances outside Mask R-CNN's vocabulary.","marker":"[Kirillov et al. 2023]"},{"why":"Llama 3.3 performs prompt parsing and per-segment instance assignment, with error feedback and in-context examples.","marker":"[Dubey et al. 2024]"},{"why":"Contributes the VQA yes/no evaluation protocol and the ConceptMix question style that CompoundPrompts adopts.","marker":"[Wu et al. 2024b]"},{"why":"Bounded Attention is a main bounding-box baseline whose attention-masking ideas the method builds on.","marker":"[Dahary et al. 2025]"}],"fun_headline_variants":["Attention anchors lift complex-prompt accuracy from 43% to 60%","Self-attention anchors + LLM labels fix instance-level generation","Cross-attention peaks become labeled masks for better image prompts","Model's own attention guides multi-object text-to-image fidelity","InstanceGen: from attention maps to instance-aware output"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every requested object instance must leave at least one detectable peak in the aggregated attention maps of the initial image; if instances overlap heavily or are too small to form a peak, segmentation misses them and the only fallbacks are repeated seeds or copying masks, which reduce layout quality.","fun_headline_variants_meta":{"raw":{"variants":["Attention anchors lift complex-prompt accuracy from 43% to 60%","Self-attention anchors + LLM labels fix instance-level generation","Cross-attention peaks become labeled masks for better image prompts","Model's own attention guides multi-object text-to-image fidelity","InstanceGen: from attention maps to instance-aware output"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1623,"prompt_tokens":859,"completion_tokens":764,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":679}},"tokens_in":475,"tokens_out":764,"duration_ms":8312,"temperature":1.0,"reasoning_tokens":679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:00:46.708769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on prompts with many small, partially occluded instances, such as ten birds on a branch with several behind leaves, and record how often the number of anchor points in the initial image falls below the requested count across many seeds. A setup with a high miss rate would show that the anchor-point premise, and therefore the layout quality, does not hold in the regime the method targets.","supporting_citations":[],"review_version":1}