{"id":"d986f644-6696-492b-848d-5634237d5086","arxiv_id":"2506.14753","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A cost-aware router selects per prompt the best among nine pre-trained text-to-image models, beating every single model on the quality-versus-cost frontier.","lead":"This paper routes each text-to-image prompt to the cheapest model from a pool of nine that still delivers the best predicted quality, letting easy prompts use small models while hard ones get expensive diffusion steps. The system reports higher average quality than any single model at the same compute cost, which matters for anyone serving image generation at scale.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-prompt quality predictability is the load-bearing assumption; Section M's noise analysis shows the learned router operates near the edge where its advantage over fixed baselines could vanish.","rationale":"The reader's weakest assumption correctly identifies per-prompt quality predictability as the pivotal condition. The paper's Section M is the most direct evidence on this point: the learned router sits between oracle-with-noise=1 and oracle-with-noise=2, meaning it captures only part of the oracle's gain. The margin over fixed baselines at several operating points is small, and the noisy-oracle analysis shows that a modest increase in estimation error would erase the advantage at the cheapest operating points. This makes the central claim sensitive to estimator quality, which is exactly the sort of condition that warrants a conditional verdict. I did not find a reason to move to REJECT: the actual experiments show consistent (if small) gains across two datasets and multiple metrics, there is a cross-dataset generalization check in Appendix C, and the noise analysis is a stress test rather than a failure demonstration. The concern reinforces the reader's CONDITIONAL verdict but does not change it. The proposed concrete test would settle the matter by measuring the estimator's actual noise level and its sensitivity to training-set size.","tokens_in":23069,"tokens_out":10044,"duration_ms":98631,"concrete_test":"Measure the actual per-prompt estimation error of gamma-hat on the held-out COCO test set: for each model m, compute the Pearson/Spearman correlation and RMSE between gamma-hat^{(m)}(x_i) and a high-accuracy ground-truth estimate obtained by averaging S=30 generated images per prompt. Then re-run the beta-noise analysis in Section M with the empirically measured noise level. If the empirical noise corresponds to beta>2, the router's advantage at low-cost operating points (e.g., Turbo, Lighting) would be expected to disappear, contradicting the central claim. Alternatively, retrain the KNN/Transformer estimators on 5%, 10%, and 50% of the training prompts and plot the deferral curves; if the advantage over fixed baselines vanishes with reduced training data, the claim is contingent on the estimator's accuracy rather than on a robust per-prompt quality signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that routing across nine models raises average quality above any single model at matched cost, requires the quality estimator gamma-hat^{(m)}(x) in Eq. (3) to approximate the per-prompt expected quality E[q(x,h^{(m)}(x))|x] accurately enough that the argmax in Eq. (3) selects the right model. The paper's own stress test (Section M, Table 6, Figure 12) shows that the learned router performs like an oracle whose per-model quality scores are corrupted by independent Gaussian noise with beta between 1 and 2 standard deviations. This is close to the failure boundary: at beta=2, the noisy oracle at the Turbo operating point (0.2780) falls below the fixed Turbo baseline (0.2798), and the reported actual gains at low-cost operating points are tiny (CLIPScore gain of 0.0002 at Turbo cost, Table 6). The paper does not report the direct accuracy of gamma-hat (e.g., correlation with held-out ground-truth quality), so it is unclear how much real signal the estimator captures and how robust the advantage is to estimator degradation. If the effective noise is at the high end of the beta in (1,2) range, the headline claim becomes fragile.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes CATImage, a prompt-level routing framework for text-to-image generation. Given a fixed pool of generation functions (SDXL at varying denoising steps plus Turbo, Lightning, DDIM, DeepCache, Infinity, and optionally FLUX), the router predicts the expected quality of each candidate for a prompt using a K-NN or Transformer estimator and selects the candidate maximizing predicted quality minus lambda times cost. The formulation is a constrained quality-cost optimization, and the optimal rule is Proposition 1 from Jitkrittum et al. (2025), applied here with a plug-in estimator. Deferral curves are produced by sweeping lambda. Experiments on COCO and DiffusionDB report quality-cost trade-offs under CLIPScore, Sharpness, Aesthetic, ImageReward, and HPSv2, with oracle and noisy-oracle controls, significance tests, cross-dataset generalization, and a small user study. The principal claim is that the trained router achieves higher average quality than any fixed model in the pool at matched cost.","tokens_in":23281,"tokens_out":10213,"duration_ms":105890,"significance":"If the claim holds, the paper makes a practical contribution: adaptive per-prompt computation that can be tuned at deployment time via lambda, with an extensive empirical evaluation across 9-12 models and multiple quality metrics, oracle upper bounds, cross-dataset checks, and a human-preference audit. The deferral-curve evaluation protocol, the noisy-oracle analysis as a calibration tool, and the released code are concrete strengths. The theoretical result is borrowed from prior work by co-authors, so the novelty lies in the application and empirical evaluation rather than in the routing rule itself. The main gaps are that the robustness of low-cost operating points is not fully established and the cost accounting of the router overhead is inconsistent and potentially omitted from the reported curves.","major_comments":[{"comment":"The paper's own robustness analysis places the learned router between the noisy-oracle curves with beta=1 and beta=2 (Figure 12). At the Turbo operating point, Table 6 reports Ours 0.2800 versus Fixed 0.2798, a gain of only 0.0002 CLIPScore, while the beta=2 noisy oracle falls below the fixed baseline (0.2780). Thus the low-cost margin is comparable to the estimator noise identified by the authors. Because the paper does not report direct diagnostics of gamma-hat accuracy (e.g., correlation between predicted and held-out per-model quality, or argmax accuracy on test prompts), the reader cannot assess how close the deployed router is to the beta=2 failure boundary. This gap matters because the abstract's headline claim is stated without cost-regime qualification.","section":"Section M, Eq. (5), Table 6, Figure 12"},{"comment":"The statement that both quality estimators incur less than 0.001 TFLOPs is inconsistent with the appendix numbers: Appendix E reports the Transformer estimator as 15.61 GFLOPs (0.0156 TFLOPs), and Appendix D reports about 13.087 GFLOPs for the Flan-T5 embedding when Infinity is selected (0.0131 TFLOPs). More importantly, the deferral curves and Table 6 costs appear to use base-model TFLOPs only, without adding these router overheads. Since the reported gain at the Turbo operating point is only 0.0002 CLIPScore, excluding router overhead can change the matched-cost comparison. The paper should state explicitly whether router overhead is included in all reported costs and correct the contradictory overhead figures.","section":"Section 5.1 vs Appendix D and Appendix E"},{"comment":"The external human-preference check shows that routers trained on CLIPScore, Sharpness, or ImageReward do not consistently beat the fixed Lightning baseline (e.g., Ours (CLIPScore) 0.287 versus Fixed 0.304; Ours (Sharpness) 0.293 versus Fixed 0.304). The abstract's phrasing, 'average quality that is higher than that achievable by any of these models alone,' is therefore stronger than the evidence unless it is explicitly qualified as quality measured by the chosen metric. The main text should add this qualification and discuss the transfer of metric-specific routing to human preference.","section":"Appendix A, HPSv2 table"}],"minor_comments":[{"comment":"The text refers to 'Stable Diffusion XL from Radford et al. (2021)'; Radford et al. is the CLIP paper, while SDXL is introduced in Podell et al. (2023). Please correct the citation.","section":"Introduction"},{"comment":"The appendix refers to 'Theorem 1' twice, but the paper's result is Proposition 1; the numbering should be aligned.","section":"Appendix M"},{"comment":"The column headers (INFI, TURB, LIGH, SDXL DEEP, etc.) should be expanded in the caption for readability, and the bold 'highest score' claims should be accompanied by pairwise significance tests since several values in adjacent columns overlap at one standard error.","section":"Table 2"},{"comment":"The sentence 'In Section C, we report...' should say 'In this appendix, we report...'.","section":"Appendix C"},{"comment":"The caption says 'presented in Table 5' but appears to refer to the K-NN (SDXL+) results from the main paper; please clarify which table is being referenced.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a solid application paper for TMLR. The main theoretical result is properly attributed to prior work by co-authors, so the contribution should be framed as an application and empirical study rather than a new optimality result. The robustness analysis in Section M is commendable but exposes a narrow margin at low-cost operating points; combined with the router-overhead accounting inconsistency, I would like to see these points resolved before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical story is the real contribution. Taking nine off-the-shelf text-to-image models and routing each prompt to the candidate that maximizes estimated quality minus lambda times cost gives a frontier that beats every fixed model on COCO and DiffusionDB across CLIPScore, Sharpness, Aesthetic, and ImageReward. Oracle and noisy-oracle controls, Welch t-tests, a sign test, cross-dataset transfer, and an HPSv2 check are all there, and they are the right controls. I believe the matched-cost claim.\n\nThe theory is not new. Proposition 1 is Proposition 1 of Jitkrittum et al. (2025), and although the text cites it immediately and says the result follows from that paper, the contribution list still frames it as 'we study the theoretically optimal routing rule.' That is overselling. The genuinely new piece is the application to heterogeneous text-to-image serving and the estimator design for per-prompt image quality, not the constrained-routing formulation.\n\nThe stress-test concern about Section M lands, but only partly. The learned router sits between an oracle corrupted by 1 and 2 standard deviations of noise, and at the Turbo operating point the gain over fixed Turbo is 0.0002 CLIPScore. If the estimator's effective noise is at the high end, the advantage at the cheap end looks fragile. However, the gains at mid-cost operating points are much larger and the matched-cost tables show consistent improvement across most of the frontier, so the headline claim survives. What is missing is a direct accuracy report for gamma-hat (correlation with ground-truth quality), which would resolve how much signal the estimator actually captures. That is a moderate gap, not a fatal one.\n\nMinor soft spots: AdaDiff is cited as the closest adaptive baseline but never run; the user study has 19 participants and yields only 52% average preference, so it should not carry preference-level claims; and the code/data link is given but the text alone does not allow verification of the generated images. None of these undercut the core empirical result.\n\nWho is this for: anyone serving text-to-image models at scale and anyone working on model routing. It deserves a serious referee even with the caveats; I would accept it for review and ask for the gamma-hat accuracy experiment and a fair AdaDiff comparison.","headline":"Routing across a heterogeneous pool of pre-trained T2I models is a genuine empirical win, but the theory is imported from prior work and the learned router's advantage is real but thin at the cheapest operating points.","tokens_in":23902,"tokens_out":2843,"would_cite":true,"duration_ms":28592,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that routing each text prompt to a pre-trained model selected by predicted quality minus cost yields higher average image quality than any single model in the pool at the same compute budget.","keywords":["cost-aware routing","text-to-image generation","diffusion models","model routing","quality-cost trade-off","prompt complexity","deferral curves","CLIPScore"],"falsifier":"Take a held-out set of prompts from a distribution unlike training; for each prompt measure each candidate's true average quality over many seeds and compute the rank correlation between the router's predicted qualities in Eq. (3) and the truth. If that correlation is near zero while the candidates' costs differ, the router's deferral curve will collapse toward the random-routing curve of Figure 12, and the central claim of beating every single model at matched cost would be refuted.","tokens_in":22823,"feed_emoji":"🎨","tokens_out":6991,"duration_ms":62288,"temperature":0.7,"pith_summary":"This paper tries to establish that the compute spent generating an image should depend on the prompt: simple prompts need few denoising steps, complex ones need many. The authors propose a router that, for each prompt, picks one of a fixed pool of already-trained text-to-image models (or step counts) by maximizing predicted image quality minus a cost penalty. They show on COCO and DiffusionDB across several quality metrics that this cost-adjusted routing delivers higher average quality than any single model in the pool at the same average cost, and can match the quality of a 100-step model at a small fraction of the compute. If true, this turns 'pick one model and serve everyone' into an adaptive allocation problem and lowers the average cost of high-quality generation.","feed_headline":"Prompt router beats any single text-to-image model","feed_subtitle":"Adaptive routing spends expensive denoising steps only on hard prompts, raising average quality at equal cost.","key_machinery":"The load-bearing object is the cost-adjusted Bayes optimal routing rule r*(x)=argmax_m E[q(x,h^(m)(x))|x] − λ c^(m), a Lagrange multiplier form of constrained quality maximization. The paper's contribution is to turn this rule into a trained plug-in estimator: a quality estimator γhat^(m)(x) (K-NN or Transformer over frozen CLIP text embeddings) predicts each candidate's expected quality per prompt, and at inference the router picks the candidate with maximum γhat^(m)(x) − λ c^(m). The sweep over λ generates a deferral curve (quality vs average TFLOPs), which is the paper's main evaluation device and the mechanism by which 'match any fixed model's quality at lower cost' is demonstrated.","core_discovery":"The central claim is that optimal per-prompt routing among M pre-trained text-to-image generators reduces to a simple score: route x to the model that maximizes E[q(x,h^(m)(x))|x] − λ c^(m), where λ is a cost-quality knob. The paper constructs a plug-in router by training a small quality estimator (K-nearest neighbors on CLIP embeddings, or a two-layer Transformer head) to predict each candidate's expected quality from the prompt, then sweeping λ to trace a quality-cost frontier. Empirically, with nine models spanning step counts, distilled SDXL variants, DeepCache, and the autoregressive Infinity model, the learned router's deferral curve lies above every fixed model's operating point on COCO and DiffusionDB, and Table 3 reports matching the quality of SDXL at 100 steps using as little as 0.13% of its cost on the sharpness metric. In the authors' words, the approach delivers an average quality higher than that achievable by any of these models alone, including when the pool is expanded with FLUX.1-dev at multiple step counts.","pith_inferences":["One could extend the same cost-adjusted rule to any stochastic generator with a per-call cost, such as video or audio diffusion models, provided an instance-level quality measure and a prompt embedding exist; the paper's formulation places no constraint on the generator class.","Because the router is prompt-only and the quality estimator is cheap (under 0.001 TFLOPs versus 1.5 TFLOPs for the smallest base model), the main cost of adoption is generating quality labels; for a new pool, reusing labels from an old pool is a plausible warm start if prompt-difficulty rankings are stable across model families.","A testable prediction follows from the paper's noise analysis: routing gains should scale with pool diversity; a pool of near-identical checkpoints should produce a frontier barely above a single model, while adding models with very different cost-quality profiles should widen the gap to every fixed baseline."],"forward_implications":["A serving system can raise average image quality without raising average cost by making the pool heterogeneous: adding better models such as FLUX moves the whole quality-cost frontier upward.","The same router can match the quality of the most expensive model in the pool at a fraction of its cost; on the sharpness metric the reported cost ratio versus SDXL-100 is 0.13%.","The routing decision transfers across quality metrics; routers trained on CLIPScore, sharpness, ImageReward, or aesthetic score each beat the fixed baselines when evaluated on their own metric, and the aesthetic-trained router tracks human preference scores best.","Routing remains effective on an unseen prompt distribution: a KNN estimator trained on DiffusionDB prompts and applied to COCO still beats fixed models, indicating prompt difficulty is learnable beyond keyword correlation.","At extreme budgets the router necessarily sends all prompts to one model, so the benefit is interior to the cost range rather than at the endpoints."],"supporting_citations":[{"why":"Supplies the optimal routing rule of Proposition 1, which the paper's plug-in estimator implements.","marker":"Jitkrittum et al. (2025)"},{"why":"Provides CLIP embeddings used for K-NN neighbor search, Transformer input, and the CLIPScore quality metric.","marker":"Radford et al. (2021)"},{"why":"Latent diffusion is the base architecture for the SDXL step-count candidates.","marker":"Rombach et al. (2022)"},{"why":"Defines the SDXL model whose varying denoising steps form the homogeneous candidate set.","marker":"Podell et al. (2023)"},{"why":"Turbo, a distilled one-step SDXL variant, is a heterogeneous candidate and a strong cheap baseline in the pool.","marker":"Sauer et al. (2024)"},{"why":"Lightning, a four-step distilled SDXL variant, is another distilled candidate the router can select.","marker":"Lin et al. (2024)"},{"why":"DeepCache contributes the caching-based candidate at intermediate cost, and is also compared as a dynamic baseline in Appendix J.","marker":"Ma et al. (2024b)"},{"why":"Infinity, the autoregressive model, is the cheapest candidate and the pool's non-diffusion entry.","marker":"Han et al. (2024)"},{"why":"Provides the COCO caption test distribution used for the main deferral curves and user study.","marker":"Lin et al. (2014)"},{"why":"Provides DiffusionDB, the large real-prompt dataset used for the multi-metric evaluation.","marker":"Wang et al. (2022b)"}],"fun_headline_variants":["Cost-aware routing beats every single model","Prompt router outguns fixed text-to-image models","Adaptive model pick tops all one-model schemes","Smarter diffusion routing trumps uniform generation","Quality-cost routing wins over each fixed generator"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The router's advantage rests on per-prompt image quality being predictable from the text prompt alone by a cheap estimator; if quality cannot be predicted from the prompt, routing degrades toward random model choice and can fall below fixed baselines.","fun_headline_variants_meta":{"raw":{"variants":["Cost-aware routing beats every single model","Prompt router outguns fixed text-to-image models","Adaptive model pick tops all one-model schemes","Smarter diffusion routing trumps uniform generation","Quality-cost routing wins over each fixed generator"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1392,"prompt_tokens":984,"completion_tokens":408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":341}},"tokens_in":600,"tokens_out":408,"duration_ms":5236,"temperature":1.0,"reasoning_tokens":341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:47:25.808866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of prompts from a distribution unlike training; for each prompt measure each candidate's true average quality over many seeds and compute the rank correlation between the router's predicted qualities in Eq. (3) and the truth. If that correlation is near zero while the candidates' costs differ, the router's deferral curve will collapse toward the random-routing curve of Figure 12, and the central claim of beating every single model at matched cost would be refuted.","supporting_citations":[{"cited_title":"Note that the produced image may be from a non-SDXL model (e.g., Turbo) since our approach may route to other baseline models described in Section 5.1","cited_arxiv_id":null,"evidence_quote":"Lightning, a four-step distilled SDXL variant, is another distilled candidate the router can select."}],"review_version":2}