{"id":"6509f5f3-8a37-4de9-a4d7-b51d3db04e7c","arxiv_id":"2411.15381","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DiffServe combines a discriminator-based diffusion model cascade with MILP-based resource allocation to serve text-to-image queries with higher quality and fewer SLO violations.","lead":"DiffServe is a serving system for text-to-image diffusion models that routes easy prompts to small, fast models and only sends hard prompts to large, slow models, using a learned quality checker. In experiments, it improves image quality by up to 24% and cuts missed latency deadlines by 19-70% compared with existing serving systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MILP maximizes confidence threshold t, but the paper's own Figure 1a shows FID is non-monotonic in the deferral rate, so the reported FID gains may not be produced by the stated optimization.","rationale":"The reader's weakest assumption flags the discriminator confidence as an unvalidated per-query quality proxy and notes the missing connection to FID. I agree that this is a real gap, but I see the more load-bearing internal issue as the objective mismatch: Eq. (5) maximizes t while the paper's own data in §2.1/Figure 1a show FID is non-monotonic in the deferral rate, i.e., in t. This means the central 'response quality' improvement is not demonstrably produced by the system's optimization; it could be a distributional artifact of mixing light and heavy outputs. That said, the paper has independent support: the artifact is public, the simulator matches the testbed to within ~1%, and the ablation in Figure 8 shows the allocation algorithm helps SLO violations. None of that resolves the monotonicity question, so the conditional verdict stands. I would not reject: the empirical gains are plausible and reproducible in principle, but the causal story needs the proposed FID-vs-t sweep before the quality claim can be accepted as the result of query-aware scaling.","tokens_in":20275,"tokens_out":8110,"duration_ms":76732,"concrete_test":"For Cascade 1 on the real Azure trace, fix the profiled execution times and enumerate the feasible combinations (x1,x2,b1,b2,t) that satisfy constraints (1)-(4), using the same 5K MS-COCO prompts to compute FID for each feasible configuration. Plot FID versus t (or versus deferral fraction f(t)) and compare the configuration chosen by objective (5) with the feasible configuration having the lowest FID. If the max-t choice is more than a few percent worse in FID than the best feasible configuration, the optimization is misaligned with the reported quality metric, and the headline quality gain is not attributable to the stated algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that DiffServe improves response quality (FID) through query-aware model scaling. The resource allocation component solves (5), max t, with constraints (1)-(4), based on the assertion in §3.3 that 'response quality is directly influenced by the confidence threshold' and the §2.2 intuition that higher thresholds defer more queries and hence improve quality. However, §2.2 also reports the opposite: 'FID gets worse as the latency increases in the end, implying that the overall system response quality can decrease as more queries are routed to the heavyweight model.' Because the deferral fraction f(t) in (3) increases with t, this is direct evidence that FID is non-monotonic in t, the very variable being maximized. If FID can degrade as t increases, the feasible max-t solution need not be anywhere near the FID-minimizing feasible solution. The up-to-24% FID improvements in Figures 5-6 are therefore not shown to be a consequence of the optimizer or of query-aware routing; they could be a byproduct of distributional mixing, which the paper itself offers as a hypothesis. No experiment isolates the effect of t on FID holding the rest of the allocation fixed, so the central quality claim is not yet causally supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DiffServe, a serving system for text-to-image diffusion models that combines a cascaded light-heavy model architecture with a learned discriminator that routes 'easy' queries to a lightweight model and defers the rest to a heavier, higher-quality model. A resource manager periodically solves a mixed-integer optimization to choose the confidence threshold, batch sizes, and model placement in response to fluctuating demand. The system is evaluated on three diffusion model pairs, using both a simulator and a 16-GPU testbed, with synthetic and real-world traces. The reported results claim up to 24% improvement in response quality (lower FID) and 19-70% lower SLO violation rates compared to state-of-the-art serving systems.","tokens_in":20545,"tokens_out":4756,"duration_ms":43510,"significance":"If the claims hold, the paper makes a useful contribution to ML serving systems by showing that query-aware routing and dynamic resource allocation can jointly improve quality and latency for expensive generative models. The work is backed by an artifact with public code (GitHub and Zenodo DOI), a real testbed evaluation, simulator validation, and ablations of both the discriminator design and the resource allocator. The observation that existing metrics like PickScore and CLIP Score underperform a random router for this cascading task is an interesting and non-obvious result. However, the central optimization claim—that maximizing the confidence threshold maximizes response quality—is in tension with the paper's own evidence of non-monotonic FID behavior, and the MILP formulation is not fully specified. These issues need to be resolved before the attributed quality improvements can be accepted as caused by the proposed optimizer.","major_comments":[{"comment":"The resource allocation problem is formulated as maximizing the confidence threshold t, with the claim that response quality is directly influenced by t. However, Figure 1a and the accompanying discussion in §2.2 explicitly state that 'FID gets worse as the latency increases in the end,' i.e., that response quality is non-monotonic in the deferral rate, which increases with t. Because f(t) in Eq. (3) is increasing in t, the feasible max-t solution may lie on the wrong side of the FID-quality curve, and the paper provides no experiment isolating the effect of t on FID while holding other allocation decisions fixed. As a result, the up-to-24% FID improvements in Figures 5 and 6 are not shown to be a consequence of the stated optimization objective. The authors should either replace the objective with one that directly models FID as a function of t (or the deferral rate) or provide evidence that, under the evaluated conditions, the feasible max-t solution indeed lies at or below the FID-minimizing deferral rate.","section":"§3.3, Eqs. (1)-(5)"},{"comment":"The optimization is described as a mixed integer linear program (MILP), but the formulation as given is not linear. The objective in Eq. (5) is linear in t, but constraint (3) contains f(t), a fitted and possibly non-linear function of the continuous variable t, and constraint (1) contains execution latency e(b) and queuing delay q(b), which are non-linear functions of integer batch sizes b1 and b2. The text does not describe the linearization, piecewise approximation, or any reformulation used to make the problem solvable by Gurobi. Without this detail, the claim that the problem is a MILP is not verifiable, and the reported ~10 ms solver time cannot be reproduced or trusted. Please provide the exact reformulation or specify the approximation method for f, e, and q.","section":"§3.3, Eqs. (1)-(5)"},{"comment":"The discriminator's softmax confidence is used as a per-query signal for whether the lightweight model's output meets the quality bar, but the evaluation metric is the distribution-level FID. The paper provides empirical evidence that the confidence score separates some queries (Figure 1b), but no calibration analysis (e.g., reliability diagrams or expected calibration error) and no held-out generalization study across datasets or prompt distributions are presented. The end-to-end quality gains could plausibly arise from distributional mixing of light and heavy model outputs rather than from accurate per-query deferral—a hypothesis the authors themselves raise in §2.2. To support the causal claim that confidence-based routing improves FID, the authors should provide direct per-query validation showing that high-confidence images from the light model are actually of comparable quality to heavy-model outputs, and low-confidence images are not, using a per-prompt quality metric on held-out data.","section":"§3.2, Figure 3"}],"minor_comments":[{"comment":"Typo: 'test-to-image' should be 'text-to-image'.","section":"§3.1"},{"comment":"In the Evaluation Metrics paragraph, 'We access system performance' should be 'We assess system performance'.","section":"§4.1"},{"comment":"The legend uses 'DiffServe' while the rest of the paper uses 'DIFFSERVE'; please keep the capitalization consistent.","section":"Figure 4"},{"comment":"The abstract reports 'up to 24% improvement' while §4.3 reports 'up to 23.4%' for the same trace; please reconcile the numbers or clarify that the abstract rounds the value.","section":"Abstract and §4.3"},{"comment":"The y-axis labeled 'FID' over time is not defined: it is unclear whether FID is computed on a rolling window of prompts, on the full 5K-prompt set for each time step, or on aggregated outputs per interval. Please specify the computation procedure.","section":"Figures 5 and 8"},{"comment":"The discriminator training details (number of epochs, learning rate, train/validation split, regularization) are not given, which makes the discriminator ablations difficult to reproduce; please add these to the artifact appendix.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for MLSys and the artifact is a significant strength. The main concern is the mismatch between the stated optimizer objective and the paper's own evidence of non-monotonic FID; this is fixable either by changing the objective to model FID directly or by adding experiments that justify the max-t choice. The MILP linearity issue is also fixable by adding the reformulation. The discriminator calibration concern is an additional load-bearing point that deserves a concrete experiment. I would not recommend rejection, but the authors need to address these points before the claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DiffServe is a legitimate addition to the ML serving literature. The core idea is simple: run a lightweight diffusion model first, use a discriminator trained to separate real from generated images as a quality gate, and only defer to a heavyweight model when the light model's output looks fake. The authors show that this beats both static baselines and Proteus (their own prior work) by a comfortable margin on a 16-GPU testbed with three model pairs and real traces. The finding that PickScore and CLIPScore are no better than random routing for this task is genuinely interesting and will save people time.\n\nThe paper has a real soft spot. The resource allocation MILP maximizes the confidence threshold t, on the theory that a higher threshold means higher quality. But the paper's own Figure 1a shows that FID is non-monotonic: routing too many queries to the heavy model actually makes FID worse. So the objective being maximized is not demonstrably the objective being reported. The up-to-24% FID improvements in Figures 5-6 could come from the distributional mixing of light and heavy outputs rather than from the optimizer's choice of t. No experiment isolates t's effect on FID holding the rest of the allocation fixed. That gap should be closed before publication.\n\nThere are a couple of smaller issues. The MILP is described as linear, but constraint (3) contains f(t), a fitted function of the decision variable t; the linearization is not specified. The stochastic metrics have no error bars. And the discriminator is trained on real images from the same datasets used for FID evaluation, so a small data-leakage concern is worth addressing. None of these are fatal; they are fixable in revision.\n\nWho should read this: anyone building serving infrastructure for generative image models. It is a solid systems paper with a practical result and a public artifact. Send it to a careful referee. The objective mismatch is the thing to push on.","headline":"Solid systems paper with a genuine empirical win, but the optimizer's objective (max t) is not the FID the paper reports, so the causal story needs work.","tokens_in":21063,"tokens_out":4786,"would_cite":true,"duration_ms":38958,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A serving system that routes easy text-to-image queries to lightweight models reports up to 24% better response quality and 19-70% fewer latency violations.","keywords":["diffusion model serving","model cascade","query-aware routing","text-to-image generation","resource allocation","latency SLO","mixed integer linear programming","discriminator"],"falsifier":"Take a held-out prompt set the discriminator never saw, have it score each light-model image, and compare those scores with an independent per-image quality judgment such as human preference or PickScore; if the scores are uncorrelated, or if the images the cascade keeps are no better than the ones it defers, then the routing signal—not the MILP—is what would fail, and the reported quality gains over random routing should disappear.","tokens_in":20087,"feed_emoji":"🖼️","tokens_out":8739,"duration_ms":75932,"temperature":0.7,"pith_summary":"This paper is trying to establish that a text-to-image serving system can get both faster responses and better output quality by treating queries as easy or hard. Its design sends every prompt first to a cheap, fast diffusion model; a learned judge decides whether that image is good enough, and only the images judged not good enough go to a slow, high-quality model. A controller then tunes the judge's threshold, batch sizes, and the number of machines assigned to each model so that latency deadlines are met while quality is as high as demand allows. The paper reports that this beats serving systems that either fix one model or switch models based only on load, improving response quality by up to 24% and cutting missed latency deadlines by 19-70%. If right, this changes the usual accuracy-versus-cost tradeoff for generative image serving, because it makes prompt difficulty a first-class resource-allocation signal.","feed_headline":"Easy prompts take the fast path; quality climbs 24%","feed_subtitle":"DiffServe's learned judge sends simple text-to-image queries to light models, cutting missed latency deadlines by 19-70%.","key_machinery":"The central mechanism is the diffusion model cascade with two components. First, the discriminator: a binary classifier trained offline to distinguish real images from images generated by both the light and heavy models; at serving time its softmax confidence measures how \"real\" the light model's output looks, and exceeding a threshold releases the image to the client. Second, the resource allocator: a mixed-integer linear program that maximizes the confidence threshold subject to per-query latency constraints (execution time plus a Little's-law queueing estimate) and throughput constraints (light-model workers must cover all demand, heavy-model workers must cover the deferred fraction $f(t)$), with $f(t)$ profiled as a function of the threshold. The MILP is re-solved periodically, so the threshold drops as demand rises and rises when the cluster is quiet.","core_discovery":"The central claim is that query-aware model scaling—building a cascade from available diffusion-model variants and routing each prompt by its difficulty—lets a serving system escape the standard quality-versus-throughput tradeoff. Concretely, the paper claims that for 20-40% of prompts a lightweight one-step model produces images equal to or better than a heavyweight model, and that a binary classifier trained to separate real photographs from generated images can identify those prompts through its softmax confidence at negligible runtime cost. On top of that, it claims that co-optimizing the confidence threshold, batch sizes, and model placement in a mixed-integer linear program, re-solved as demand changes, keeps latency under service-level objectives while maximizing quality; the reported consequence is up to 24% better response quality than load-only scaling and 19-70% fewer SLO violations than static allocation, with the cascade sometimes beating an all-heavy deployment on Frechet Inception Distance (FID).","pith_inferences":["Editorial inference: because the MILP objective is simply \"maximize the threshold,\" the same cascade machinery can be re-targeted at cost, energy, or fairness objectives by swapping the objective function without changing the routing or latency constraints.","Editorial inference: the reported result that mixing light-model outputs can improve FID over all-heavy output suggests response diversity is part of the quality story; a testable extension is whether that effect persists across more model pairs or holds under per-image quality metrics.","Editorial inference: a router that predicts difficulty from the text prompt alone would avoid running the light model as a probe; since the paper leaves this open, a natural next experiment is training a regressor from text embeddings to the discriminator's confidence.","Editorial inference: the confidence threshold could double as an admission-control knob, so under extreme overload the system might drop only the hardest queries rather than shedding load uniformly, preserving SLOs for the rest."],"forward_implications":["Under high demand, the controller can lower the confidence threshold, routing more queries to the light model and trading a little quality to keep latency within SLO; under low demand it raises the threshold and sends more work to the heavy model.","Operators can stop provisioning for peak demand alone, because the cascade absorbs load spikes by shifting work to lighter models instead of dropping or delaying queries.","The confidence threshold becomes a single tunable dial that converts an SLO requirement into an operating point on the quality-throughput frontier.","The same design extends to longer pipelines by placing a discriminator after each stage and adding each stage's threshold to the optimization, and the discriminator's overhead stays negligible relative to diffusion execution.","Since 20-40% of prompts are easy, a fixed heavy-only deployment wastes a large share of its compute on queries that a light model can serve as well or better."],"supporting_citations":[{"why":"The prior serving system that scales model variants by load but routes queries randomly; DiffServe's query-aware routing is measured against it and beats its quality by up to 20%.","marker":"(Ahmad et al., 2024)"},{"why":"The static Clipper serving system used as a baseline, and the additive-increase/multiplicative-decrease batching heuristic used as an ablation in the resource allocator.","marker":"(Crankshaw et al., 2017b)"},{"why":"Defines Frechet Inception Distance, the distribution-level metric used to measure response quality throughout the evaluation.","marker":"(Heusel et al., 2017)"},{"why":"Provides PickScore, the per-prompt quality metric used to establish that 20-40% of queries are easy for the lightweight model.","marker":"(Kirstain et al., 2023)"},{"why":"Supplies the real-world serverless workload trace used to drive dynamic demand in the end-to-end evaluation.","marker":"(Shahrad et al., 2020)"},{"why":"Gives the queueing-theory relation (Little's law) used inside the MILP to estimate per-worker queuing delay from queue length and arrival rate.","marker":"(Shortle et al., 2018)"},{"why":"The high-quality heavyweight model used in the third cascade and in the motivating quality-versus-speed comparison.","marker":"(Podell et al., 2023)"}],"fun_headline_variants":["Query-aware cascade lifts image quality 24% and cuts SLO misses","For 20-40% of prompts, a light model suffices—DiffServe tags them","DiffServe: easy prompts skip heavy compute, quality climbs 24%","Easy prompts go light, quality up 24%, latency misses down 70%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the discriminator's softmax confidence that a light-model image looks real is a trustworthy per-prompt signal of whether that image meets the quality bar, so easy queries can safely skip the heavy model.","fun_headline_variants_meta":{"raw":{"variants":["Query-aware cascade lifts image quality 24% and cuts SLO misses","For 20-40% of prompts, a light model suffices—DiffServe tags them","DiffServe: easy prompts skip heavy compute, quality climbs 24%","Easy prompts go light, quality up 24%, latency misses down 70%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001389,"raw_usage":{"total_tokens":5592,"prompt_tokens":887,"completion_tokens":4705,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":4619}},"tokens_in":503,"tokens_out":4705,"duration_ms":30036,"temperature":1.0,"reasoning_tokens":4619,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:21:52.078065+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out prompt set the discriminator never saw, have it score each light-model image, and compare those scores with an independent per-image quality judgment such as human preference or PickScore; if the scores are uncorrelated, or if the images the cascade keeps are no better than the ones it defers, then the routing signal—not the MILP—is what would fail, and the reported quality gains over random routing should disappear.","supporting_citations":[{"cited_title":"D., Williams, T., Sitaraman, R","cited_arxiv_id":null,"evidence_quote":"The prior serving system that scales model variants by load but routes queries randomly; DiffServe's query-aware routing is measured against it and beats its quality by up to 20%."},{"cited_title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","cited_arxiv_id":null,"evidence_quote":"Defines Frechet Inception Distance, the distribution-level metric used to measure response quality throughout the evaluation."},{"cited_title":"Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider","cited_arxiv_id":null,"evidence_quote":"Supplies the real-world serverless workload trace used to drive dynamic demand in the end-to-end evaluation."},{"cited_title":"F., Thompson, J","cited_arxiv_id":null,"evidence_quote":"Gives the queueing-theory relation (Little's law) used inside the MILP to estimate per-worker queuing delay from queue length and arrival rate."}],"review_version":1}