REVIEW 4 major objections 8 minor
Reviewed by Pith at T0; open to challenge.
T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →
T0 review · glm-5.2
Co-trained 8B reasoner beats frontier oracle on visual knowledge boundary
2026-07-07 13:07 UTC pith:DYO6ZH7I
load-bearing objection Solid benchmark and co-training recipe; headline comparison lacks significance testing and has a judge confound the 4 major comments →
Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The knowledge boundary between internalizable and contextual knowledge is generator-specific, shifts under training, and is discoverable through co-training without explicit boundary labels. The boundary emerges from the reward signal: Phase 1 DPO pushes the generator's boundary outward by internalizing stable visual knowledge and building noise robustness to imperfect search references; Phase 2 RFT recalibrates the reasoner to search only what the strengthened generator still cannot render. A calibrated 8B reasoner matched to a specific 4B generator slightly outperforms a frontier-scale VLM oracle on the same generator, demonstrating that generator-specific calibration can substitute forraw
What carries the argument
The knowledge boundary B(theta) is defined as a generator-specific partition of world-knowledge units K into internalizable (K_int) and contextual (K_ctx) sets, where internalizable knowledge satisfies the condition that search does not improve quality beyond a tolerance epsilon. The co-training framework discovers this boundary through two phases: Phase 1 applies online iterative Diffusion-DPO on search-augmented generations, scoring candidates by an automated VLM judge and constructing preference pairs from top- and worst-scored outputs; Phase 2 applies rejection-sampling finetuning to the reasoner, retaining only positive-advantage trajectories where search improved the strengthened gene
Load-bearing premise
The automated VLM judge providing reward signals for preference-pair construction achieves Spearman correlation of 0.87 with human ratings overall, but agreement weakens specifically on TextualSearch prompts where glyph-level correctness and fine layout dominate rubric variance. If the judge is noisiest on exactly the hardest prompts, the preference pairs on those prompts may be mislabeled, and observed monotonic improvement on the hardest tier could partly reflect the judge
What would settle it
The claim that co-training discovers a genuine knowledge boundary would be undermined if the automated judge's biases — rather than real knowledge internalization — drove the observed monotonic improvement, particularly on the hardest prompts (Set III) where judge-human agreement is weakest. A direct falsifier: replace the VLM judge with human raters for preference-pair construction and check whether the monotonic progression and Set III gains persist.
If this is right
- If the knowledge boundary framework generalizes, the same teach-then-search co-training principle could extend to video, 3D, and music generation, which face the same structural mismatch between finite training data and unbounded user requests.
- A recursive self-improvement flywheel becomes feasible: each co-training iteration pushes the boundary further outward and tightens the search policy further inward, converging toward a regime where only genuinely contextual knowledge triggers search.
- The finding that generator-specific calibration substitutes for frontier-scale reasoning suggests that small, well-calibrated generator-reasoner pairs could approach the performance of much larger monolithic systems on knowledge-intensive tasks.
- The released frozen search corpus (145,642 archived sessions, 559,973 URLs) enables reproducible research on tool-augmented visual generation without live API access or result drift, lowering the barrier for community participation.
- The gate-filter-integrate protocol and boundary-discovery mechanism may extend beyond search to other tools — image editing, render-as-code, 3D-asset retrieval — each supplying a different slice of contextual knowledge.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces SearchGen-20K/Bench, a large-scale bilingual dataset and benchmark for world-knowledge-grounded text-to-image generation, spanning 12 failure categories and 22 domains. The authors show that frontier open-weight generators collapse by up to 40 points on search-intensive prompts, and that naive (blind) search degrades performance on prompts generators already handle. To address this, the paper proposes a co-training framework: (1) online DPO teaches the generator to internalize stable knowledge and resist search noise, and (2) rejection-sampling fine-tuning (RFT) recalibrates an 8B agentic reasoner to the strengthened generator's shifted knowledge boundary. Experiments on two generator architectures (Klein-4B, Bagel-7B) show monotonic improvement across training phases, selectivity (recovery on NoSearch prompts), and generator-specificity. The co-trained 8B reasoner paired with a 4B generator (31.8) slightly exceeds a frontier VLM oracle (31.2) on the same generator. Full datasets, trajectories, and a frozen search corpus are released.
Significance. The paper makes a strong resource contribution: SearchGen-20K/Bench with 20,839 prompts, 90,452 reasoning traces, 281,925 generated images, and 145,642 archived search sessions (enabling offline reproducible research without live API drift) is substantial and well-motivated. The conceptual framing of a generator-specific, evolving knowledge boundary is productive and the three falsifiable predictions (monotonicity, selectivity, generator-specificity) are well-designed. The gate-filter-integrate protocol is a sensible response to the identified failure modes (concept corruption, copy effects). The finding that generator-specific calibration of an 8B reasoner can match a frontier oracle is the most striking claim and warrants careful scrutiny (see major comments). The release of a replayable harness is a significant community asset.
major comments (4)
- §4.1, Table 6: The headline claim that 'generator-specific calibration can substitute for frontier-scale reasoning' rests on a 0.6-point difference (GENERATOR-ADAPTIVESEARCH 31.8 vs ORACLE 31.2) on a 0-100 scale across 651 search-intensive prompts. No confidence intervals, bootstrap intervals, or significance tests are reported. Given likely per-prompt standard deviations of 15-20, the standard error of the mean is roughly 0.6-0.8, placing this difference at the boundary of statistical detectability. The claim should either be supported with bootstrap confidence intervals or softened to 'comparable to' rather than 'slightly exceeds.'
- §4, Evaluation: The ORACLE baseline uses Gemini-3-Flash as the reasoner, and the evaluation judge is also Gemini-3-Flash. This creates a self-preference confound: the oracle's reasoning traces are scored by the same model family that generated them, potentially inflating the oracle's 31.2. The direction of bias is ambiguous (it could make the co-trained model's relative advantage either more or less meaningful), but it must be discussed. The DPO training judge (Qwen3-VL-8B, Appendix E.2) is different from the eval judge, which is a methodological strength, but the scoring model used for RFT in Phase 2 (Appendix E.3) is unspecified. If RFT uses Gemini-3-Flash for scoring, the co-trained reasoner may have learned to produce outputs favored by the eval judge, creating a different confound. Please clarify which model scores RFT rollouts and discuss the self-preference issue.
- §3.2, Definition 1: The knowledge boundary is defined via the quality gap Q(G,p,SEARCH(k)) - Q(G,p,∅) < ε, and the paper then 'discovers' this boundary by training on exactly this quality gap signal (DPO preference pairs from scored generations, RFT advantages from scored rollouts). This is somewhat circular: the boundary is defined by the reward signal used to discover it. The paper should acknowledge this more explicitly and discuss whether the boundary has independent grounding (e.g., via the human-annotated knowledge gap references in §2.2) or whether it is purely operational — defined by what the reward signal can measure. This does not invalidate the empirical findings but affects the theoretical framing.
- Appendix B.2: The judge-human agreement (Spearman ρ=0.87) weakens on TextualSearch, 'where glyph-level correctness and fine layout dominate rubric variance.' Set III (hardest prompts) disproportionately contains TextualSearch-type prompts. If the judge is noisiest exactly on the hardest prompts, the DPO preference pairs and RFT advantages on those prompts may be mislabeled, and the observed 'monotonic improvement' on Set III (Figure 8a) could partly reflect the judge rewarding its own biases rather than genuine knowledge internalization. A stratified analysis of judge agreement by difficulty tier (not just by stratum type) would help assess this risk. At minimum, the paper should discuss whether Set III gains are robust to judge noise.
minor comments (8)
- §3.2, Definition 1: The claim that K_int(θ) ⊆ K_int(θ') when θ' results from DPO on search-augmented demonstrations is stated as an axiom but is not proven. It is an empirical assumption. Consider framing it as a conjecture or providing a brief argument for why DPO on search-augmented data monotonically expands the internalizable set (rather than, say, causing forgetting of some previously internalized knowledge).
- Algorithm 1, Line 8: The comment notes that the scoring function 'Score' is 'distinct from boundary-defining Q in Def. 1,' but the relationship between the two is unclear. If Score approximates Q, the circularity concern (Major Comment 3) is sharpened. Please clarify how Score relates to Q.
- Table 5: The strata sizes (NoSearch=100, VisualSearch=387, TextualSearch=264) sum to 751, matching the test set. However, the difficulty terciles in Table 6 (Set I-III) also sum to 651. Please clarify the relationship between the VisualSearch/TextualSearch partition and the Set I-III difficulty partition — are they orthogonal? A cross-tabulation would help readers understand the structure.
- Figure 8a: The y-axis range (17.5-35.0) is truncated, which visually exaggerates the magnitude of improvement across phases. Consider using a zero-baseline or explicitly noting the truncation.
- §4.1: The text states 'The 39-point gap to GPT-Image-2 (71.0) reflects generator capacity at the 4B scale rather than a framework limitation.' This is a strong claim that is not directly tested. Consider softening to 'likely reflects' or providing evidence (e.g., co-training results on a larger generator).
- Appendix E.2: DPO training uses only ~7K prompt-image preference pairs (Table 10) from the 20K dataset. Please clarify how the 7K subset was selected and whether results are sensitive to this choice.
- References: Several citations use future dates (e.g., [4] Black Forest Labs 2026, [22] Jiang et al. 2026, [42] Wang et al. 2026). Please verify these are correct and not placeholder dates.
- §2.2: The text states '20,188 rows (20,187 unique prompts)' but the abstract and elsewhere state '20,839 prompts.' Please reconcile these numbers.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive review. The comments are substantive and we address each below. We agree with most points and will revise accordingly; two points require partial disagreement with explanation.
read point-by-point responses
-
Referee: §4.1, Table 6: The headline claim that 'generator-specific calibration can substitute for frontier-scale reasoning' rests on a 0.6-point difference (31.8 vs 31.2) with no confidence intervals or significance tests. The claim should be supported with bootstrap CIs or softened.
Authors: The referee is correct that the 0.6-point difference is within the range of statistical noise, and we should not claim 'slightly exceeds' without supporting evidence. We will run bootstrap confidence intervals (10,000 resamples over the 651 search-intensive prompts) for both GENERATOR-ADAPTIVESEARCH and ORACLE and report them in the revised Table 6. If the intervals overlap substantially, we will soften the language to 'comparable to' rather than 'slightly exceeds,' as the referee suggests. We agree this is the right standard. That said, we note that the broader claim we wish to defend is not that the 8B reasoner strictly outperforms the frontier oracle, but rather that generator-specific calibration allows a small model to approach frontier-scale performance—a claim supported by the convergence (31.8 vs 31.2) regardless of which is nominally higher. We will make this framing explicit in the revision. revision: yes
-
Referee: §4, Evaluation: The ORACLE baseline uses Gemini-3-Flash as the reasoner, and the evaluation judge is also Gemini-3-Flash, creating a self-preference confound. The scoring model used for RFT in Phase 2 (Appendix E.3) is unspecified. Please clarify which model scores RFT rollouts and discuss the self-preference issue.
Authors: We thank the referee for identifying this important methodological issue. We address both parts. (1) RFT scoring model: The scoring model for Phase 2 RFT rollouts is Qwen3-VL-8B, the same model used for DPO preference scoring in Phase 1 (Appendix E.2), not Gemini-3-Flash. We will state this explicitly in the revised Appendix E.3. This means the co-trained reasoner's RFT signal comes from a different model family than the evaluation judge, which partially mitigates the self-preference concern for the co-trained model. (2) Self-preference confound for ORACLE: The referee is correct that the ORACLE baseline (Gemini-3-Flash as reasoner) and the evaluation judge (Gemini-3-Flash) share the same model family, which could inflate the oracle's score. We note that this confound, if present, would inflate the oracle's 31.2, making the co-trained model's relative performance more—not less—impressive. However, the direction of bias is indeed ambiguous in principle (e.g., the oracle's reasoning traces might be longer or more complex in ways the judge penalizes). We will add a dedicated paragraph in §4 discussing this confound, noting the RFT scoring model distinction, and acknowledging that the self-preference issue for the oracle means the comparison is conservative with respect to our claim. We will also note that the DPO training judge (Qwen3-VL-8B) being different from the eval judge (Gemini-3-Flash) is a methodological strength, as the referee acknowledges. revision: yes
-
Referee: §3.2, Definition 1: The knowledge boundary is defined via the quality gap Q(G,p,SEARCH(k)) - Q(G,p,∅) < ε, and the paper then 'discovers' this boundary by training on exactly this quality gap signal. This is somewhat circular. The paper should acknowledge this and discuss whether the boundary has independent grounding.
Authors: The referee raises a valid conceptual point. We agree that there is a degree of circularity: the boundary is defined by a quality gap, and the same quality gap signal (via scored generations) is used to discover it through co-training. We will add an explicit acknowledgment of this in the revised §3.2. However, we want to clarify two points that mitigate the circularity concern. First, the boundary does have independent grounding through the human-annotated knowledge gap references in §2.2: each prompt carries structured knowledge gap slots with severity labels, constructed via an answer-first strategy before any model training. These annotations provide an a priori characterization of where knowledge gaps are expected to lie, independent of the reward signal. Second, the boundary's generator-specificity and shift under training (Figure 8b) are empirical observations measured by the quality gap, not assumptions built into the definition. The definition is operational—it specifies what the boundary does (partitions knowledge into internalizable vs. contextual) rather than what it is in some ontological sense. We will revise the text to frame Definition 1 as an operational definition, acknowledge the circularity explicitly, and point to the human-annotated knowledge gap references as independent (if imperfect) grounding. revision: yes
-
Referee: Appendix B.2: Judge-human agreement (Spearman ρ=0.87) weakens on TextualSearch. Set III (hardest prompts) disproportionately contains TextualSearch-type prompts. If the judge is noisiest on the hardest prompts, the observed monotonic improvement on Set III could partly reflect judge bias. A stratified analysis of judge agreement by difficulty tier would help.
Authors: This is a well-taken concern. We agree that the interaction between judge noise and difficulty tier is important to assess. We will conduct a stratified analysis of judge-human agreement by difficulty tier (Set I/II/III) on the 500-pair human study set and report it in the revised Appendix B.2. If the stratified analysis confirms that agreement drops specifically on Set III TextualSearch prompts, we will discuss the implications for the monotonic improvement claim on Set III. We acknowledge that we cannot fully rule out the possibility that some Set III gains reflect judge bias rather than genuine knowledge internalization. However, we note two mitigating factors: (1) the DPO training judge is Qwen3-VL-8B, a different model from the evaluation judge (Gemini-3-Flash), so the generator is not trained to exploit the eval judge's specific biases; (2) the monotonic improvement on Set III is consistent across both Klein-4B and Bagel-7B architectures (Figure 8a), and the knowledge boundary shift in Figure 8b is measured on no-search quality (where the judge evaluates unaugmented generations, reducing the TextualSearch-specific noise channel). We will add a discussion of this risk in the revised text and present whatever the stratified analysis shows honestly. revision: yes
Circularity Check
No significant circularity: the knowledge boundary definition and its discovery via co-training are conceptually related but not tautological; the empirical predictions are independently falsifiable.
full rationale
The reader's take suggests circularity because Definition 1 defines the knowledge boundary via the quality gap Q(G,p,SEARCH(k)) - Q(G,p,∅) < ε, and then co-training 'discovers' this boundary using the same quality signal. However, on inspection this is not circular in the sense the framework targets. Definition 1 is a formal definition of a partition; the co-training algorithm (Algorithm 1) does not compute this partition directly. Phase 1 (DPO) trains the generator on search-augmented inputs using preference pairs ranked by image quality (line 8 explicitly notes the Score function is 'distinct from boundary-defining Q in Def. 1'). Phase 2 (RFT) trains the reasoner via group-relative advantage on image quality. The boundary 'emerges from the reward signal' (§3.3) as a consequence of training, not as a fitted parameter renamed as a prediction. The paper's three predictions — monotonicity, selectivity, and generator-specificity — are independently falsifiable: monotonicity could fail if DPO regressed some stratum; selectivity could fail if the calibrated reasoner still degraded NoSearch prompts; generator-specificity could fail if the cross-check (calibrated reasoner on base generator) matched the calibrated-on-DPO performance. None of these predictions reduce by construction to the definition. The 31.8-vs-31.2 comparison lacks confidence intervals (a correctness/statistical concern, not circularity), and the oracle/eval judge model overlap (Gemini-3-Flash) is a self-preference confound (also correctness, not circularity). The DPO training judge is explicitly Qwen3-VL-8B (Appendix E.2), distinct from the eval judge, which is a methodological strength. There is no self-citation chain where the central premise depends on the authors' own unverified prior work. The definition-to-discovery relationship is analogous to defining 'accuracy' and then training to improve it — the training signal is related to the metric, but the improvement is empirical, not tautological. Score 2: one minor conceptual proximity between definition and training signal, but no step where a prediction reduces to its inputs by construction.
Axiom & Free-Parameter Ledger
free parameters (7)
- DPO temperature β =
100
- EMA decay =
0.99
- SSIM penalty (dpo_penalize_condition_ssim) =
0.95
- Guidance scale =
4.0
- Tolerance ε =
not specified numerically
- M (candidates per prompt) =
5
- N_traj (RFT rollouts) =
not specified
axioms (4)
- domain assumption The VLM judge (Gemini-3-Flash) provides a sufficiently accurate proxy for human quality judgments to serve as reward signal for DPO and RFT training
- domain assumption The quality function Q(G_θ, p, c) is well-defined and bounded in [0,1] for all generators, prompts, and conditioning contexts
- ad hoc to paper K_int(θ) ⊆ K_int(θ') when θ' results from DPO on search-augmented demonstrations
- standard math Standard flow-matching DPO loss (Algorithm 1, line 11) correctly extends DPO to flow-matching velocity fields
invented entities (1)
-
Knowledge boundary B(θ)
no independent evidence
read the original abstract
Visual generators excel at rendering, but they confidently fabricate what they do not know. User requests are unbounded, evolving, and deeply long-tailed: new characters, trending entities, post-cutoff events, and more. This world-knowledge bottleneck is structural: generators are trained on fixed corpora, but the visual world is open-ended. We construct SearchGen-20K and SearchGen-Bench, with 20,839 prompts spanning twelve failure categories and twenty-two domains, paired with a pre-executed multimodal SearchGen-Corpus-1M to support offline, reproducible research. On SearchGen-Bench, frontier open generators score only 21 to 28 out of 100, a 40-point collapse invisible to existing benchmarks. The natural remedy is to employ search tools, enabling agentic visual generation. However, we find that naive search fails: it retrieves indiscriminately, injecting noise into prompts the generator already handles. We trace the root cause to a generator-specific, evolving knowledge boundary: the divide between what a generator can internalize through training and what must remain in external context. Although this boundary is hard to specify in advance, we show that it is discoverable through a teach-then-search co-training framework. Even a minimal version of this co-training recipe produces monotonic improvement, laying the foundation for recursive self-improvement in visual generation that can meet world-knowledge-grounded requests. We release the full dataset, co-training corpus, and search corpus as a replayable harness for tool-augmented, world-knowledge-grounded visual generation.
Figures
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.