{"id":"f80c0a4a-10c8-4e64-9598-4ef7bddb885a","arxiv_id":"2607.05382","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Selective search plus generator-reasoner co-training improves knowledge-grounded image generation, but the reported gains are scored by the same VLM judge used to train the system.","lead":"A new 20K-prompt benchmark shows that open image generators collapse by up to 40 points when users ask for recent or culturally specific knowledge, while naive search-augmentation makes things worse by injecting noise. The paper's fix is a 'teach-then-search' co-training loop that teaches the generator what it can memorize and calibrates the search reasoner to fetch only the rest.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 9b does not measure the boundary shift it claims: it plots no-search quality, not the Definition-1 gap ΔQ = Q(SEARCH) − Q(∅), so the 'newly internalized knowledge' region is unsupported.","rationale":"The reader identified the Gemini-3-Flash judge fidelity as the weakest assumption. That is a serious concern, but the most load-bearing flaw is internal and can be stated even if the judge were perfect: the paper defines the knowledge boundary via the search-augmentation gap ΔQ, yet its direct evidence for boundary migration plots only the no-search quality. The inference from 'no-search quality improved' to 'knowledge units migrated from Kctx to Kint' is invalid because the gap could stay constant or grow if search-augmented quality also improves—which is likely, since Phase-1 DPO is trained on search-augmented inputs. The paper even misdescribes Figure 9b in §3.2 as measuring the with/without-search gap. This is not an external critique about metric choice; it is a gap between the formal definition and the empirical demonstration. The released corpus should make the missing ΔQ analysis straightforward. If the ΔQ distribution does not shift left, the conceptual contribution is weakened to a claim about score improvement, not boundary discovery. That said, the benchmark and the naive-search failure analysis remain valuable, and the requested re-analysis is feasible, so the reader's CONDITIONAL verdict remains appropriate rather than a rejection.","tokens_in":28943,"tokens_out":7441,"duration_ms":81374,"concrete_test":"From the released SEARCHGEN harness, compute for the 647-prompt eval set the per-prompt scores Q(Gθ, p, ∅) and Q(Gθ, p, SEARCH(k)) for both the base generator and the Phase-1 DPO generator, using the same judge and protocol as the paper. Plot the CDF of ΔQ = Q(SEARCH) − Q(∅) for each generator and report the fraction of prompts with ΔQ > ε for a fixed ε (e.g., 5 points). If this fraction does not decrease after DPO, Figure 9b's shaded region does not correspond to internalization, and the 'discovered boundary' claim should be downgraded to 'co-training improves scores.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evidence that co-training discovers the knowledge boundary does not measure the quantity Definition 1 defines. Def. 1 partitions knowledge units by the per-prompt quality gap ΔQ = Q(Gθ, p, SEARCH(k)) − Q(Gθ, p, ∅) relative to a tolerance ε; a unit migrates from Kctx to Kint only when ΔQ falls below ε. The paper's direct evidence, Figure 9b and the surrounding text in §4.1, plots the CDF of no-search quality Q(Gθ, p, ∅) before and after Phase-1 DPO. A rightward shift in this CDF is consistent with DPO improving general rendering or instruction-following and does not imply that ΔQ shrank. In fact, because Phase-1 DPO trains on search-augmented inputs, Q with search may improve as much or more, leaving the boundary unchanged. The text in §3.2 even claims that Figure 9b measures 'the per-prompt quality gap with and without search,' which the figure does not do. Without the ΔQ distribution, the shaded 'newly internalized knowledge' region is not identified, and the paper's central claim that a knowledge boundary is discovered—rather than merely that judge scores improve—lacks its stated evidence. This concern is independent of, and compounds, the separate issue of whether the Gemini-3-Flash judge is a faithful reward.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SEARCHGEN-20K/SEARCHGEN-BENCH, a long-tailed, bilingual benchmark of world-knowledge-intensive text-to-image prompts with fine-grained checklists and rubrics, together with a pre-executed multimodal search corpus (SEARCHGEN-CORPUS-1M) enabling offline, replayable evaluation. The authors show that frontier open generators collapse to 21–28/100 on search-intensive prompts, that naive search can degrade performance, and propose a two-phase co-training recipe (Phase 1: online DPO on the generator; Phase 2: rejection-sampling finetuning of an 8B reasoner) to discover a generator-specific 'knowledge boundary' and improve search-augmented generation. The central claim is that a minimal co-training recipe yields monotonic improvement and that an 8B reasoner calibrated by this procedure can match or slightly exceed a frontier oracle on a 4B generator.","tokens_in":29317,"tokens_out":3997,"duration_ms":44741,"significance":"If the central claims hold, the paper makes a useful contribution: it identifies a real evaluation gap, provides a substantial and reproducible infrastructure (prompts, reasoning traces, cached search sessions), and proposes a concrete, low-compute co-training loop for agentic visual generation. The offline replay harness is a particularly valuable resource for the community. The paper is also commendably explicit about limitations (single-iteration recipe, VLM judge noise, commercial-API interpretation caveats). The 'knowledge boundary' concept is intuitively appealing and connects to selective-retrieval literature, but its empirical demonstration is currently not tight enough to support the strongest statements in the abstract and §4.1.","major_comments":[{"comment":"Definition 1 defines the knowledge boundary through the per-prompt quality gap ΔQ = Q(Gθ,p,SEARCH(k)) − Q(Gθ,p,∅) and the tolerance ε (Eq. 1). The evidence in Figure 9b, however, plots the CDF of no-search quality Q(Gθ,p,∅) before and after Phase 1 DPO. A rightward shift in this CDF does not imply that ΔQ shrank; it is consistent with DPO improving general rendering or instruction-following, or with search-augmented quality improving by an equal or larger amount. The text in §4.1 ('CDF of per-prompt no-search quality') is accurate, but §3.2 explicitly claims that 'Figure 9b makes this concrete: we measure the per-prompt quality gap with and without search,' which the figure does not do. Without the joint distribution of Q(SEARCH) and Q(∅) — or a per-prompt migration table across the ε threshold — the shaded 'newly internalized knowledge' region is unsupported. This is the load-bearing ev","section":"§3.2, §4.1, Fig. 9b"},{"comment":"The main quantitative claim is monotonic improvement across co-training phases, but all numbers in Table 6 are single-run point estimates with no error bars, confidence intervals, or significance tests. The monotonicity claim is already internally contradicted by the NoSearch column: Klein-4B drops from 54.6 (Phase 0) to 54.0 (Phase 1), and Bagel drops from 52.6 to 52.4. While the Search/Overall columns improve monotonically, the NoSearch regression is small and could easily be within run-to-run noise. Since the paper's prediction is explicitly 'without regressing on any stratum,' these two regressions require at least a paired analysis across prompts, multiple seeds, or a statement of variance. The absence of statistical support also affects the matched-compute comparison (RFT-8B + Klein-4B-DPO: 31.8 vs. ORACLE: 31.2), which is a one-point difference on a single run.","section":"§4.1, Table 6"},{"comment":"The same VLM-judge protocol supplies the DPO reward (Phase 1: 'Score(p,x_j)' in Algorithm 1, line 8; described as Qwen3-VL-8B in Appendix E.2), the RFT reward (Algorithm 1, line 17), and the final evaluation scores (Gemini-3-Flash judge in §4). Even if the judge is not literally the same model, both are VLM judges using the same scoring rubrics. This creates a circularity risk: the 'knowledge boundary shift' in Figure 9b and the gains in Table 6 may partly reflect optimization of the judge's preferences rather than genuine world-knowledge grounding. The reported human correlation (Spearman ρ=0.87 on 500 pairs) is reassuring but does not show that the co-trained models improve on human judgment relative to the baselines, nor that the ΔQ change is human-visible. I recommend evaluating the co-trained outputs on a held-out human-rated subset and/or with an independent judge family (e.g., a d","section":"§2.3, §3.3, Algorithm 1, Appendix D.2/E.2"}],"minor_comments":[{"comment":"The dataset size is stated inconsistently: 20,839 prompts (Abstract), 20,939 prompt records (§6), and 20,188 training rows with 751 test prompts in §2.2. Please reconcile these numbers and clarify whether 20,839/20,939 includes the test and validation splits.","section":"Abstract, §2.2, §6"},{"comment":"The caption says the CDF is computed on the '647-prompt eval set,' while §4 says all evaluation is on the 751-prompt test set. If the boundary analysis uses a subset (e.g., search-intensive only), state this explicitly and justify why the full test set is not used.","section":"§4, Fig. 9b"},{"comment":"§2.3 states the evaluation has four knowledge-sensitive components, but Table 3 lists five (Checklist, Rubric, Prompt Faithfulness, Visual Reference Fidelity, Textual Knowledge Fidelity). Please align the count and describe which components are included in the overall mean.","section":"§2.3, Table 3"},{"comment":"The Phase 0 SFT data size is given as '~10,000 expert-annotated trajectories' in §3.3 but '∼20K SFT trajectories' in Appendix E.1 and '~20,000' in the Appendix overview. Please correct the inconsistency.","section":"§3.3, Appendix E.1"},{"comment":"The 'answer-first' synthesis strategy means prompts carry knowledge-gap references and checklists by construction. This is efficient but risks making the benchmark somewhat circular for the knowledge-gap detection task. Please provide a human validation of the checklists (e.g., a sample where annotators confirm each checklist item is answerable and non-trivial for a knowledgeable human).","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong fit for the venue's interest in datasets, benchmarks, and agentic generation, and the released offline harness is a genuine contribution. However, the central 'knowledge boundary discovery' claim currently rests on evidence that does not measure the quantity defined in Definition 1, and the headline monotonicity is based on single-run point estimates with two observed NoSearch regressions. These are fixable within the scope of a revision: replace Figure 9b with a ΔQ distribution or per-prompt migration table, add variance/paired tests for Table 6, and include an independent or human evaluation on the co-trained outputs. I would also encourage the authors to be more cautious in the abstract about 'monotonic improvement' until these analyses are provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset and the co-training recipe are the real contributions here. SearchGen-20K/Bench with 20k prompts, a frozen million-item search corpus, and 283k cached generations is a genuinely useful resource: it lets people study search-augmented visual generation offline, reproducibly, without burning API credits. The main empirical claim also holds up in broad strokes: naive search degrades outputs, and a small 8B reasoner calibrated to a 4B generator gets close to a frontier oracle on the aggregate score. That is a nice, cheap result worth knowing about.\n\nThe soft spots are real but not fatal. The biggest one is the stress-test point, and it lands: Definition 1 defines the knowledge boundary in terms of the per-prompt quality gap ΔQ = Q(search) − Q(no-search), but Figure 9b plots only the no-search quality CDF before and after DPO. A rightward shift in that CDF does not imply ΔQ shrank. DPO could improve rendering generally, or improve search-augmented outputs just as much, leaving the boundary unchanged. The text even says the figure shows 'the per-prompt quality gap with and without search,' which it does not. That is a mismatch between definition and evidence, and it undercuts the 'discovery of the boundary' narrative as stated.\n\nThe judge circularity is a genuine secondary concern: the same Gemini-3-Flash judge supplies the DPO/RFT rewards and the final evaluation scores. The paper acknowledges judge noise but does not address the self-referentiality directly. The single-run point estimates without error bars are also annoying, especially when 'monotonic' is contradicted by small NoSearch drops (54.6→54.0 and 52.6→52.4). These are not disqualifying — the aggregate trend is probably right — but they are exactly the kind of thing a serious referee should force the authors to tighten.\n\nI disagree with the reader's skepticism on one point: the lack of error bars does not make the central empirical claim suspect; the naive-search failure and the co-training gains are large and consistent across two architectures. The load-bearing weakness is purely the boundary-shift evidence, which is weaker than the paper claims.\n\nThis paper should definitely go to peer review. The benchmark alone justifies referee time, and the co-training result is interesting enough to warrant a careful revision. A competent referee should ask the authors to either measure ΔQ directly or soften the 'boundary discovery' claim. I would bring this to reading group and would cite the dataset.","headline":"Useful benchmark and a plausible co-training recipe, but the central 'boundary discovery' evidence doesn't actually measure the boundary.","tokens_in":29830,"tokens_out":794,"would_cite":true,"duration_ms":11023,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that every image generator has a discoverable, generator-specific knowledge boundary, and that co-training the generator with a search reasoner expands what the generator can internalize while teaching the reasoner when to","keywords":["world-knowledge bottleneck","knowledge boundary","agentic visual generation","search-augmented generation","co-training","Diffusion-DPO","text-to-image evaluation","long-tail knowledge"],"falsifier":"Re-score the 751-prompt test set and the DPO preference pairs with human raters using the same checklists and rubrics, then check whether the Phase 0 to Phase 1 to Phase 2 monotonic progression and the rightward CDF shift survive at comparable magnitude; a significant shrink or reversal would indicate the knowledge boundary is an artifact of the automated judge.","tokens_in":28839,"feed_emoji":"🔍","tokens_out":4124,"duration_ms":42701,"temperature":0.7,"pith_summary":"The paper tries to establish that the failure of image generators on world-knowledge-grounded requests is not a rendering problem but a structural knowledge bottleneck, and that the divide between what a generator can internalize and what it must retrieve—the knowledge boundary—exists, is generator-specific, shifts under training, and is discoverable through a teach-then-search co-training procedure. It builds a 20,939-prompt bilingual benchmark spanning twelve failure categories, shows frontier open generators collapse to 21–28 out of 100 while search-integrated commercial systems stay near 71, and shows that naive search degrades prompts the generator already handles. The central result is that a minimal recipe—one DPO pass to teach the generator, one rejection-finetuning pass to recalibrate the reasoner—improves monotonically at every difficulty tier, and lets an 8B reasoner paired with a 4B generator match a frontier oracle reasoner on the same generator. A sympathetic reader would care because it turns 'search vs. memorize' from an engineering tradeoff into a measurable, trainable quantity, and it supplies a replayable offline harness for research.","feed_headline":"Co-training teaches image models what to look up","feed_subtitle":"A teach-then-search loop lifts open generators from the low 20s to 32 on world-knowledge prompts.","key_machinery":"The load-bearing object is the knowledge boundary (Definition 1), a generator-specific partition of world-knowledge units into internalizable and contextual sets, defined through the expected quality gain from search. The argument is carried by a three-stage gate–filter–integrate agentic reasoner (decide when to search, select noise-free references, integrate through grounded language rather than raw pixels) and a two-phase co-training loop: online iterative Diffusion-DPO to teach the generator what it can absorb, then rejection-sampling finetuning of the reasoner to the shifted boundary. The boundary is also the unit of evaluation: rightward shifts in the no-search quality CDF directly show","core_discovery":"The paper's central claim is that knowledge required for faithful visual generation splits along a structural axis: some units (a character's canonical appearance, a flag's geometry) can migrate into generator parameters once seen, while others (post-cutoff events, long-tail cultural specifics, per-instance compositions) resist parameterization and must stay in external context. This split, formalized as the knowledge boundary, is generator-specific and evolves as the generator improves. The paper further claims that the boundary need not be known in advance; it can be discovered by co-training: first use online iterative DPO on search-augmented samples to push the generator's boundary outwa","pith_inferences":["The knowledge-boundary formulation is a general tool-invocation principle: the same gate–filter–integrate and teach-then-search loop should apply to image editing, render-as-code, and 3D retrieval, not just web search.","If the automated judge is biased or exploitable, the monotonic gains are an artifact of optimizing the judge, not of expanding knowledge; a human re-evaluation of Phase 1 versus Phase 2 outputs would settle this.","The rightward CDF shift of no-search quality could be used as a stopping criterion for recursive co-training: when DPO stops shifting the boundary, further teach cycles should yield diminishing returns and the reasoner should take over.","With 93% of entities appearing in a single prompt, purely parametric scaling cannot close the gap; any path to frontier visual generation must pair large generators with external grounding, making the boundary a permanent architectural feature."],"forward_implications":["Open-weight generators lose up to 40 points on search-intensive world-knowledge prompts, a gap invisible to standard composition benchmarks; SearchGen-Bench makes it measurable.","Search must be selective: blind retrieval corrupts prompts the generator already handles, so the decision to search is a first-class design variable.","Co-training yields monotonic improvement at every difficulty tier for two architecturally distinct generators, so the benefits are not tied to one conditioning mechanism.","A generator-calibrated 8B reasoner matches or slightly exceeds a frontier oracle reasoner on the same 4B generator at a fraction of compute, indicating generator-specific calibration substitutes for scale.","The released pre-executed search corpus allows offline replay of the full pipeline, so search-augmented visual generation research no longer requires live search APIs."],"fun_headline_variants":["Teach-then-search lifts visual generators past knowledge wall","Co-training redraws what image models can learn to know","Search fills what training can't teach image generators","Knowledge boundary discovered by teach-then-search loop","Visual generators evolve by co-training with search"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the score produced by the automated judge (with its checklist, rubric, and reference-fidelity components) measures world-knowledge fidelity faithfully; if the judge can be gamed or is biased, the discovered boundary and the monotonic gains are artifacts of optimizing that judge.","fun_headline_variants_meta":{"raw":{"variants":["Teach-then-search lifts visual generators past knowledge wall","Co-training redraws what image models can learn to know","Search fills what training can't teach image generators","Knowledge boundary discovered by teach-then-search loop","Visual generators evolve by co-training with search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1077,"prompt_tokens":809,"completion_tokens":268,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":194}},"tokens_in":553,"tokens_out":268,"duration_ms":3203,"temperature":1.0,"reasoning_tokens":194,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:26:34.570196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the 751-prompt test set and the DPO preference pairs with human raters using the same checklists and rubrics, then check whether the Phase 0 to Phase 1 to Phase 2 monotonic progression and the rightward CDF shift survive at comparable magnitude; a significant shrink or reversal would indicate the knowledge boundary is an artifact of the automated judge.","supporting_citations":[],"review_version":3}