{"id":"449bfbbe-2221-4042-aede-29f3fc5f62db","arxiv_id":"2411.18810","paper_version":5,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Certain random seeds yield consistently more accurate compositional text-to-image outputs, and mining these seeds plus fine-tuning on the resulting self-generated images improves numerical and spatial composition accuracy.","lead":"This paper shows that the random seed used to start a text-to-image diffusion model strongly affects whether it correctly renders object counts and spatial relations, and that certain seeds are consistently more reliable than others. The authors mine these reliable seeds automatically, then fine-tune models on images generated with them, improving compositional accuracy on Stable Diffusion and PixArt-alpha.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The mining of 100 seeds and selection of top-3 (Sec. 4.2) lacks a multiple-comparison control: with 60 images per seed, the best of 100 binomial estimates can exceed the mean by chance alone, so the reported sampling/fine-tuning gains may be inflated by best-of-100 selection rather than reflecting…","rationale":"The paper is a careful empirical study: it uses a held-out test set with unseen categories and backgrounds, evaluates with GPT-4o and validates against human judgments (Table 3), includes ablations on the number of reliable seeds (Table 10), and shows consistent gains across two architectures. The attention-map analysis in Sec. 3 provides a plausible mechanism. My concern is not about the existence of seed effects—the top-50 result and the five-seed chi-squared test support that—but about the stability of the specific top-3 ranking extracted from 100 noisy estimates. The reader's weakest assumption points at transfer; I agree partially because transfer to unseen categories is actually demonstrated on the test set, whereas the statistical reliability of the mining step itself is not tested. A split-half replication of the mining procedure would settle whether the gains are a reproducible property of the selected seeds or an artifact of selecting the maximum of 100 binomial estimates. Given the evidence already presented, I would not reject the paper, but the conditional verdict is appropriate: the method's headline numbers should be accompanied by a stability analysis of the seed ranking.","tokens_in":18790,"tokens_out":7568,"duration_ms":68794,"concrete_test":"Split the 60 mining prompts used per seed in Sec. 4.2 into two disjoint halves (30 prompts each), re-rank the 100 seeds independently on each half, and compute the Spearman rank correlation between the two rankings. Then select the top-3 seeds from one half only and evaluate them on the Comp90 test set with GPT-4o, comparing against the reported 43.0 (numerical) and 23.4 (spatial) for Stable Diffusion 2.1. If the split-half rank correlation is high (ρ > 0.5) and the half-A top-3 reproduce the test-set gains within ~2 points, the ranking is stable and the concern does not land. If ρ is near zero or the gains do not reproduce, the reported effect is largely a best-of-100 selection artifact and the central claim needs to be reframed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 selects the top-3 of 100 candidate seeds from per-seed accuracy estimates computed on only 60 generated images per quantity (numerical) and 80 per relation (spatial). For a binomial proportion with p≈0.3 and n=60, the standard error is about 5.9 percentage points, so the maximum of 100 independent estimates is expected to lie roughly 10–12 points above the mean even under the null that all seeds are equally reliable. The paper's only significance test (chi-squared, p=1.2e-4, Sec. 3.3) is for five hand-picked seeds, not for the mined 100-seed ranking. Consequently, the observed gap between top and bottom seeds in Fig. 6, and the +5.5/+5.6 point gains from 'sampling with reliable seeds' (Tables 1–2), may be inflated by best-of-100 selection. The fine-tuning results are less vulnerable because the reliable-seed data is evaluated on the held-out test set after training, but the curated data themselves are generated by the same selected seeds, so a portion of the gain could still reflect selection luck. The top-50 result (Table 10: 40.8 vs 37.5) suggests a genuine but weaker seed effect; the question is whether the specific top-3 ranking is stable enough to be the load-bearing component of the method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the effect of the initial noise seed on compositional text-to-image generation. Using Stable Diffusion 2.1 and PixArt-α, the authors show that different seeds produce characteristic object layouts, that these layouts correlate with compositional correctness, and that per-seed accuracy varies substantially. They construct a new benchmark, Comp90, with disjoint train/test categories and backgrounds, mine 100 candidate seeds with CogVLM2 on a small subset, and then either sample with the top-3 seeds or fine-tune only the Q/K attention projections on images generated with those seeds, with optional CogVLM2-based recaptioning. On held-out test prompts, sampling with reliable seeds improves numerical and spatial accuracy by roughly 3 to 6 points, and fine-tuning on reliable, rectified self-generated data yields larger gains (e.g., 37.5 to 51.3 numerical and 17.8 to 36.6 spatial for Stable Diffusion 2.1). The authors include ablations over top-k, comparisons with random-seed and rectified baselines, a human evaluation of GPT-4o scoring, and diversity/image-quality metrics.","tokens_in":19097,"tokens_out":9287,"duration_ms":80666,"significance":"The paper identifies a genuinely useful and simple phenomenon: seed choice is not neutral for compositional correctness, and a small curated set of reliable seeds can improve both inference-time sampling and fine-tuning. The main empirical claims are supported by held-out category evaluation, human validation of the GPT-4o metrics, top-k ablations, and comparisons against random-seed fine-tuning and recaptioning baselines. The release of code and the Comp90 benchmark are valuable assets. The main weaknesses are statistical: the top-3 seed selection is made from 100 candidates without a multiple-comparison control, the transfer of the seed ranking across categories is not directly quantified, and the main tables lack error bars. These issues are fixable with additional analyses and do not, in my reading, invalidate the central phenomenon.","major_comments":[{"comment":"The top-3 seeds are selected from 100 candidate seeds using per-seed accuracies estimated on only 60 images per quantity (numerical) and 80 images per relation (spatial). For a binomial proportion with p≈0.3 and n=60, the standard error is about 5.9 percentage points, so the maximum of 100 independent estimates is expected to exceed the mean by roughly 10-12 points even under the null of equal seed reliability. The chi-squared test in Sec. 3.3 covers five hand-picked seeds and does not control this multiple-comparison. Although evaluating the selected seeds on the held-out test set reduces the risk of selection inflation, the magnitude of the sampling gains in Tables 1-2 and the spread in Fig. 6 remain unquantified. The top-50 result in Table 10 (40.8 vs 37.5) suggests a genuine but weaker effect, so the specific top-3 ranking is load-bearing. Please add a permutation or bootstrap test under seed exchangeability (e.g., the null distribution of the best-of-100 accuracy on the mining subset) and report confidence intervals for the sampling gains.","section":"Sec. 4.2, Fig. 6, Tables 1-2"},{"comment":"The central generalization claim is that seed reliability transfers from the 15 mining categories and 4 settings to unseen categories, backgrounds, and tasks. The only quantitative evidence is the aggregate test-set improvement of the top-3 seeds. The paper does not report the correlation of per-seed accuracies between disjoint category splits, nor any split-half reliability of the mined ranking. Given that the top-50 gain (40.8 vs 37.5) is much smaller than the top-3 gain (43.0 vs 37.5), the observed top-3 improvement could be dominated by a small number of unstable seeds. Please provide a per-seed correlation between two disjoint category splits or a split-half analysis of the mining ranking, and report per-seed test accuracies on held-out categories.","section":"Sec. 4.2 / Sec. 5.1, Tables 1-2, Table 10"},{"comment":"All main quantitative comparisons are reported as point estimates without standard deviations, confidence intervals, or the number of evaluation seeds. Several differences that are used to support design choices are small, for example PixArt-α spatial accuracy: fine-tuning (reliable) 27.5 vs fine-tuning (reliable + rectified) 27.2 vs fine-tuning (random + rectified) 26.6 in Table 2. Please report means and standard deviations over at least three evaluation runs or seeds, or otherwise justify that the reported differences exceed evaluation noise.","section":"Tables 1, 2, 4, and Appendix A.4"},{"comment":"The five candidate seeds used for the chi-squared test are not described as randomly selected or pre-registered; if they were chosen after inspecting their accuracies, the reported p=1.2e-4 is not a valid test of the null. Please state how these five seeds were selected, and consider reporting a significance test on the full 100-seed ranking instead of only on a hand-picked subset.","section":"Sec. 3.3"}],"minor_comments":[{"comment":"The claim of a '6% accuracy improvement' is imprecise; Tables 1-2 show gains ranging from about 2.9 to 6.0 points. Please report the exact numbers.","section":"Abstract and Introduction"},{"comment":"The visual claim that seeds produce distinct object arrangements would be strengthened by a quantitative measure, such as the centroid or entropy of the binarized attention maps, rather than relying only on visual inspection.","section":"Figure 2"},{"comment":"The description of the data curation step is ambiguous: 'For each generated image, we use CogVLM2 to check its correctness' does not state whether incorrect images are removed, kept, or recaptioned. Please clarify the exact curation pipeline and report the fraction of generated images that were rectified or discarded, since this affects interpretation of the 'reliable' vs 'reliable + rectified' variants.","section":"Sec. 4.3"},{"comment":"The labels 'Fine-tuned' and 'Fine-tuned (Ours)' are easy to confuse; consider renaming them to 'Fine-tuned (random seeds)' and 'Fine-tuned (reliable seeds + rectified)'.","section":"Figure 7"},{"comment":"The 'Potential Biases' paragraph is a useful limitation statement but appears only in the appendix; consider moving a condensed version to the main text, since the CogVLM2-in-the-loop design is central to the method.","section":"Appendix A.3.1"},{"comment":"The caption says 'Our method significantly improves the accuracy' but no significance test is reported; please qualify the language or add the corresponding statistical test.","section":"Table 11 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published at ICLR 2025, so the journal version should focus on adding the statistical rigor requested in the major comments. The CogVLM2-in-the-loop issue (used both for mining and for rectification) is a mild circularity concern but is mitigated by GPT-4o and human evaluation on held-out sets. The fit with the journal's scope is good, and the Comp90 benchmark and code release are assets. The requested analyses are straightforward and should be feasible within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading it. First, it gives a genuinely useful empirical result: certain random seeds in Stable Diffusion and PixArt-α consistently produce better compositional layouts, and you can mine those seeds automatically with a VLM and then fine-tune the model on self-generated data. Second, the size of the seed-sampling effect is probably overestimated. The stress-test is right: picking the top-3 of 100 seeds from accuracy estimates on 60 images per seed means the best seed will look much better than average even if all seeds are equal. The reported +5.5 points from sampling with reliable seeds is therefore not trustworthy as a measure of the true seed effect. That said, the paper does not rest entirely on that number. The fine-tuning results are much less vulnerable to selection artifacts, because the model is evaluated on held-out categories after training. The gap between fine-tuning on reliable-seed data and random-seed data is real and substantial (e.g., 48.5 vs 41.8 for SD numerical, 28.6 vs 22.0 for spatial), even before rectification. The ablation with k=50 still shows a ~3-point gain, so there is a genuine, if weaker, seed effect underneath the multiple-comparison noise. What is actually new here is the identification of seed-specific layout patterns as a cause of compositional failure, and the automatic reliable-seed mining plus self-generated fine-tuning pipeline. The paper does a solid job on evaluation: held-out categories, ablations, comparisons against LMD/MultiDiffusion/Ranni, and a human validation of the GPT-4o metric. The code is public. My main soft spot, beyond the multiple-comparison issue, is that the mining and rectification both use CogVLM2, so there is a mild circularity in the data-construction loop. The human evaluation shows the final metrics are not VLM hallucinations, but the seed ranking itself is never independently validated. Also, the authors report no error bars or significance tests for the fine-tuning comparisons, which makes it hard to judge how much of the 2-7 point differences are real. The paper is written clearly, and the claims are appropriately hedged for the most part. I think this deserves a serious referee. For a reader in T2I or compositional generation, it is a worthwhile contribution and a practical trick. I would bring it to a reading group and probably cite it for the seed-mining method, even if I would not treat the sampling gains at face value.","headline":"A practical and well-evaluated paper showing that some random seeds reliably improve compositional T2I generation, with the honest caveat that the headline seed-sampling gains are likely inflated by best-of-100 selection.","tokens_in":19616,"tokens_out":3240,"would_cite":true,"duration_ms":30562,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a large share of compositional failures in text-to-image diffusion models is caused by the initial random seed, since different seeds bias objects into different image layouts and some layouts are consistently more…","keywords":["text-to-image generation","diffusion models","compositional generation","random seed reliability","initial noise","cross-attention maps","numeracy and spatial relations","self-generated data fine-tuning"],"falsifier":"Re-run the seed-mining protocol exactly as described, then evaluate the top-3 seeds against the full Comp90 test set (30 unseen categories, 4 unseen backgrounds) using human raters instead of GPT-4o; if the top seeds' accuracy is within noise of a random seed baseline — or if swapping the mining subset to a disjoint set of categories changes which seeds are top-ranked — the transferability claim is falsified. A second decisive check is to measure whether the same seeds remain reliable when the object categories are paraphrased in the prompt (e.g., 'a pair of dogs' vs 'two dogs'), since the paper's explanation predicts layout, not lexical, control.","tokens_in":18581,"feed_emoji":"🎲","tokens_out":10103,"duration_ms":81453,"temperature":0.7,"pith_summary":"Text-to-image diffusion models often draw the wrong number of objects or put them in the wrong place, and this paper claims the culprit is frequently the random starting noise, not the prompt or the model weights. Different initial seeds systematically bias where objects appear: some seeds spread objects into separated, readable layouts, while others bunch them into regions where they are missed, so some seeds are simply more reliable for compositional prompts. The paper proves the point by mining 100 seeds with a vision-language model on a small prompt subset, then shows two uses of the reliable seeds: sampling with them improves accuracy by roughly 3–6% with no extra computation, and fine-tuning only the attention projections on self-generated reliable-seed images raises numerical accuracy by 29.3% (Stable Diffusion) and 19.5% (PixArt-α), and spatial accuracy by 60.7% and 21.1% relative. The improvement matters because it is automatic, annotation-free, costs almost nothing at inference, and works on two different model architectures with minimal loss of aesthetic quality or diversity. If the seed-reliability ranking transfers as claimed, then many real-world compositional failures can be fixed without layout conditioning or retraining from scratch.","feed_headline":"Mining good seeds boosts AI image composition by up to 60%","feed_subtitle":"Pre-ranked noise patterns lift spatial accuracy 60.7% and counting accuracy 29.3% on two diffusion models.","key_machinery":"The load-bearing mechanism is the seed-conditioned layout prior carried in cross-attention maps: what a random seed dictates is not the object content but where the model's object tokens attend, producing distinct average image arrangements per seed. Because some arrangements (separated, evenly spread, vertically stacked) are far more likely to render counts and spatial relations correctly than clustered ones, compositional accuracy becomes a measurable property of the seed. The paper operationalizes this with (1) Comp90, a 90-category prompt set split into disjoint train/test categories and backgrounds, (2) a mining protocol that scores 100 candidate seeds with CogVLM2 on a small prompt subset and sorts them per quantity/relation, and (3) a parameter-restricted fine-tuning recipe that updates only query/key projections in attention, leaving the rest of the model frozen. The same machinery supplies both the training-free sampling gain and the self-generated training data.","core_discovery":"The central discovery is that compositional correctness in text-to-image diffusion models is strongly conditioned on the initial seed, because each seed biases the cross-attention maps toward a particular object arrangement. Averaged binarized attention maps show that correct images share consistent layouts (separated regions, grids, vertical stacking for 'on top of') while incorrect images cluster objects in small areas, and this association is visible in both Stable Diffusion 2.1 and PixArt-α. Consequently, with a small batch of candidate seeds scored by CogVLM2 on a subset of the paper's Comp90 prompts, a stable reliability ranking emerges: the top seeds outperform the bottom seeds by large margins on held-out categories (38.3% vs 17.5%), and using the top seeds either at sampling time or as a source of self-generated fine-tuning data transfers the gain to unseen test prompts. The fine-tuning step achieves seed-independent improvement by updating only the query and key projections of attention layers, so that random seeds inherit the reliable-layout behavior.","pith_inferences":["Editorial inference: if seed-layout association is baked into diffusion training, the same mining protocol should transfer to other denoising-based generators — e.g., layout-free video or 3D generation — where structured composition also fails, making seed reliability a general property of generative samplers rather than a text-to-image curiosity.","Editorial inference: a direct test of the paper's mechanism would be to measure whether reliability ranking is monotone under prompt paraphrase and under small weight perturbations; monotonicity would support a geometric interpretation ('good seeds sit in a favorable basin of the sampling trajectory'), whereas fragility would suggest the ranking is partly an artifact of the Comp90 prompt distribut","Editorial inference: the large spatial gains suggest reliable-seed sampling may be a cheaper substitute for explicit layout conditioning in production pipelines; a natural comparison is whether combining top seeds with an LLM-generated bounding-box layout produces additive gains or saturates, which the paper does not test.","Editorial inference: because only Q/K projections are updated, the fine-tuned model likely encodes a 'reliable layout prior' that could be distilled into a seed-independent adapter, letting users keep full seed diversity while inheriting reliable arrangements."],"forward_implications":["Using the top-3 reliable seeds at sampling time gives a 3–6 point absolute accuracy improvement over random seeds on numerical and spatial composition with no retraining and no added inference cost, and the effect persists up to top-50 seeds.","Fine-tuning only attention Q/K projections on self-generated images produced with reliable seeds yields relative gains of 29.3% (numerical, Stable Diffusion), 19.5% (numerical, PixArt-α), 60.7% (spatial, Stable Diffusion), and 21.1% (spatial, PixArt-α) while keeping aesthetic score and recall losses small.","The gains transfer to multiple-category prompts such as 'three tigers and two penguins' and to out-of-scope quantities like seven or eight objects, without re-mining seeds for those cases.","Rectifying the self-generated captions with CogVLM2 (re-captioning) stacks with reliable-seed fine-tuning, and sampling a fine-tuned model with reliable seeds yields the best numbers reported (56.2% numerical accuracy on Stable Diffusion).","The authors position reliable-seed mining as a general, model-agnostic inference-time scaling technique that needs no manual annotation and no layout inputs, unlike LLM-based layout methods that sacrifice diversity and aesthetic quality."],"supporting_citations":[{"why":"Supplies the Stable Diffusion 2.1 base model whose compositional failures and seed behavior are analyzed.","marker":"Rombach et al., 2022"},{"why":"Supplies the PixArt-α base model, the second architecture used to show the seed effect and fine-tuning gains generalize.","marker":"Chen et al., 2023a"},{"why":"CogVLM2 is the vision-language model that scores candidate seeds, rectifies prompts, and checks generated-image correctness.","marker":"Hong et al., 2024"},{"why":"GPT-4o is the independent evaluator of generated quantities and spatial relations, and filters unreasonable Comp90 spatial prompts.","marker":"Achiam et al., 2023; OpenAI"},{"why":"Motivates the parameter-restricted fine-tuning choice by warning that training on self-generated data can exacerbate model biases.","marker":"Shumailov et al., 2023"},{"why":"Prior evidence that initial noise quality strongly influences output coherence, which this paper extends to compositional accuracy.","marker":"Guo et al., 2024a"},{"why":"Prior demonstration that carefully selected seeds can unlock rare concepts in Stable Diffusion, supporting the seed-reliability hypothesis.","marker":"Samuel et al., 2024"},{"why":"Cross-attention control work used as the rationale that attention maps, and thus Q/K projections, govern object arrangement.","marker":"Hertz et al., 2022"}],"fun_headline_variants":["Good seeds lift text-to-image composition up to 60%","Seed reliability unlocks 60% better AI image composition","Top seeds improve image composition by up to 60%","Reliable seeds yield 60% gains in AI image composition","Seed choice matters: up to 60% compositional gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes that the reliability ranking of seeds, measured by CogVLM2 on a small subset of Comp90 (15 categories and 4 backgrounds for numerical, 20 scenes and 4 settings for spatial), is stable enough to transfer to unseen test categories, unseen backgrounds, quantities beyond 6, and to GPT-4o/human evaluation; the paper itself acknowledges that CogVLM2 can miscount underrepresented objects and large numbers, so a ranking error on the mining subset would propagate into both the sampling gains and the fine-tuning data.","fun_headline_variants_meta":{"raw":{"variants":["Good seeds lift text-to-image composition up to 60%","Seed reliability unlocks 60% better AI image composition","Top seeds improve image composition by up to 60%","Reliable seeds yield 60% gains in AI image composition","Seed choice matters: up to 60% compositional gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001598,"raw_usage":{"total_tokens":6377,"prompt_tokens":964,"completion_tokens":5413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":5331}},"tokens_in":580,"tokens_out":5413,"duration_ms":33263,"temperature":1.0,"reasoning_tokens":5331,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:51:42.784639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the seed-mining protocol exactly as described, then evaluate the top-3 seeds against the full Comp90 test set (30 unseen categories, 4 unseen backgrounds) using human raters instead of GPT-4o; if the top seeds' accuracy is within noise of a random seed baseline — or if swapping the mining subset to a disjoint set of categories changes which seeds are top-ranked — the transferability claim is falsified. A second decisive check is to measure whether the same seeds remain reliable when the object categories are paraphrased in the prompt (e.g., 'a pair of dogs' vs 'two dogs'), since the paper's explanation predicts layout, not lexical, control.","supporting_citations":[{"cited_title":"Generating images of rare concepts using pre-trained diffusion models","cited_arxiv_id":null,"evidence_quote":"Prior demonstration that carefully selected seeds can unlock rare concepts in Stable Diffusion, supporting the seed-reliability hypothesis."}],"review_version":1}