{"id":"02a1de25-2e6c-4678-940b-8ac0c82f690c","arxiv_id":"2507.10150","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A past-future scheduler predicts output lengths from historical requests and only admits new requests when the predicted future memory peak of the running batch fits in GPU memory.","lead":"This paper presents a scheduler for LLM serving that predicts request output lengths from history and checks whether the running batch will fit in GPU memory at each future step before admitting new requests. The authors report 2 to 3 times higher SLA-satisfying throughput than aggressive or conservative baselines in their open-source LightLLM framework.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scheduler's admission decision assumes output-length distributions are stationary across adjacent windows; the paper never isolates the transient right after an abrupt shift, where M* is systematically biased and the reserved-memory hedge may not compensate.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing premise: the immediate future output-length distribution must match the recent past. My stress-test agrees and sharpens it. The scheduler has no mechanism to detect a distribution shift inside the current window, and the paper's Figure 3 explicitly shows that hybrid/API workloads are not globally stable. The only evidence for adjacent-window stability is the diagonal pattern in Figure 3 and the aggregate concatenated-workload result in Figure 8, neither of which isolates the transient immediately after an abrupt shift. Under such a shift, the samples in Algorithm 1 lines 4 and 8 are drawn from a stale P(l), so M* from Eq. 4 is systematically wrong; the fixed reserved percentage cannot absorb a shift comparable to the mean output length. Because the paper's central claim is 'precisely estimates peak memory' and 'consistently achieving better goodput' across applications, this unaddressed failure mode is material. The verdict remains CONDITIONAL as the reader set it, since the concern is testable and the paper's empirical claims could still survive a controlled transient test. I therefore recommend no change to the reader's conditional verdict, with the addition of this specific shift-transient experiment as a condition.","tokens_in":23414,"tokens_out":12987,"duration_ms":165303,"concrete_test":"Run a controlled abrupt-shift trace: serve Distribution-3 (outputs roughly 32-4k) for several historical windows, then switch instantaneously to Distribution-1 (outputs roughly 2k-4k) with no warm-up. Log per-minute goodput, eviction rate, and the gap between the M* computed at admission and the observed peak memory. If, within the first 5-10 minutes after the switch, the eviction rate rises materially above the steady-state rate, or admitted M* underestimates the observed peak by more than the reserved fraction, the historical-window stationarity assumption is the load-bearing weakness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Eq. 4 (M* = max_i M_i) precisely estimates the future peak memory of the running batch, so admitting while M* <= M avoids harmful evictions. This only works if the predicted lengths sampled in Algorithm 1 lines 4 and 8 come from the same distribution that will generate the current requests' actual outputs. That stationarity is the load-bearing premise. The paper's own Figure 3 concedes that hybrid or API workloads have low global distribution similarity, and it relies entirely on adjacent-window similarity. If a shift occurs inside the current historical window, every sample for both running and queued requests is drawn from a stale P(l). The error is not just noise: it is a systematic bias in M* in the direction that causes over-admission. The reserved-memory parameter (3-10% in Table 1) is a fixed hedge; it cannot absorb a shift comparable to the mean output length. Figure 8 does not settle this because the concatenated workload is reported only as aggregate eviction/goodput, and the scheduler is allowed to refill its 1000-request history after each segment boundary. A transient collapse in the first few minutes after a cut would be hidden in the averages. Table 1's non-zero eviction rates even for stable synthetic distributions further show that the method is not literally precise, but the distribution-shift failure is more dangerous because it is systematic and has no detection or adaptation mechanism in the algorithm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Past-Future scheduler for continuous batching in LLM serving. The scheduler predicts each request's output length by sampling from the historical output-length distribution of recently completed requests, then estimates the future peak memory requirement of the running batch by computing memory occupancy at each future completion event (Eqs. 3-4). A queued request is admitted only when this predicted peak stays within a reserved fraction of the system memory capacity. The authors implement the scheduler in an open-source framework, LightLLM, and report goodput improvements of up to 2-3x over conservative and aggressive schedulers under SLA constraints on Llama-2 7B/13B/70B and multimodal models, together with trace-based evidence that output-length distributions are similar across adjacent time windows.","tokens_in":23705,"tokens_out":5548,"duration_ms":63360,"significance":"If the central claims hold, the paper makes a practical contribution to LLM serving: a lightweight, history-based scheduling rule that improves SLA-satisfying throughput without model-specific prediction models. The memory-accounting formulation in Eqs. 2-4 is coherent under a synchronous decode model, and the open-source release plus production deployment are tangible strengths. However, the paper's own Table 1 shows non-zero eviction rates for the Past-Future scheduler even with a reserved-memory hedge, and the evaluation does not isolate the transient behavior after an abrupt distribution shift. These issues undercut the abstract's 'precisely estimates' and 'consistently achieving better goodput' claims and need to be resolved before the paper can be accepted as written.","major_comments":[{"comment":"Table 1 reports that Past-Future with reserved=3% still evicts 6.86%, 7.42%, and 2.59% of requests on Distribution-1, Distribution-2, and Distribution-3, while the same rows show average Future Required Memory below 100%. This contradicts Section 5.2's statement that 'all scheduled requests are completed successfully' and the abstract's claim of 'precisely estimat[ing] the peak memory resources.' If M* is accurate and the admission test is M* <= M, evictions should be essentially zero even without reserved memory. The paper should explain the source of these residual evictions (prediction error, fragmentation, admission races, or other causes) and either strengthen the admission mechanism to actually prevent evictions or qualify the precision claims to match the measured behavior.","section":"Section 5.3, Table 1"},{"comment":"The scheduler's load-bearing premise is that the output-length distribution of the immediate future matches the recent past, so that samples from P(l) and P(l > l_t^i) in Algorithm 1 are reliable. Figure 3 itself shows that API/hybrid workloads have lower adjacent-window similarity, and the paper provides no mechanism to detect or adapt to a shift occurring within the current historical window. Figure 8 evaluates a concatenated workload only in aggregate, so a transient collapse in goodput or a burst of evictions immediately after each segment boundary would be hidden in the averages. Please add an experiment that reports goodput and evictions in short time intervals (e.g., every 100-200 requests) across a workload switch, and discuss whether the fixed reserved-memory hedge is sufficient or whether an adaptive mechanism is needed.","section":"Section 3.2 and Section 5.3, Figure 8"},{"comment":"The prediction method is described as 'parameter-free,' but Eq. 1 depends on the window size w, and the paper states in Section 4 that w=1000 was chosen based on Figure 4. The reserved memory ratio is also a free parameter, with 3%, 5%, and 10% variants explored in Table 1 and Figure 8. These parameters are tuned on the same workloads used for evaluation, which weakens the claim that the method generalizes across workloads without configuration. Please remove the 'parameter-free' description or provide a principled, workload-independent way to set w and the reserved ratio, and clearly state which results are sensitivity analyses of tuned parameters.","section":"Section 3.2 and Section 4"},{"comment":"The implementation initializes the output-length distribution with the preset maximum output length at service startup and only updates it after a few minutes of traffic. During that cold-start period, the scheduler is effectively conservative and will underutilize memory. The paper does not quantify the duration of this transient or its impact on goodput, yet the abstract and Section 5 claim consistent goodput improvements. Please report the cold-start behavior or explicitly scope the claims to steady-state operation after the history window is filled.","section":"Section 4, startup behavior"}],"minor_comments":[{"comment":"The notation L_h = {l_h^0, l_h^1, ..., l_h^w} contains w+1 elements while Eq. 1 divides the count by w; use a consistent indexing scheme, e.g., L_h = {l_h^1, ..., l_h^w} or divide by w+1.","section":"Section 3.2, Eq. 1"},{"comment":"The sentence 'ensuring that all scheduled requests are completed successfully' is too strong given the eviction rates in Table 1; please replace it with a hedged formulation such as 'aims to ensure' or adjust the reported eviction data accordingly.","section":"Section 5.2"},{"comment":"The text refers to the 'Future-Past scheduler' in the paragraph on effect on request eviction; this should be 'Past-Future scheduler' for consistency.","section":"Section 5.3"},{"comment":"The algorithm is introduced as 'PageAttention' here but referred to as 'PagedAttention' elsewhere; please use one spelling consistently.","section":"Section 2.2"},{"comment":"The claim that output-length distributions are 'stable within a short time period (minutes)' is qualitative; please quantify the timescale using the trace data in Figure 3 or a similar analysis so that the stationarity assumption can be checked.","section":"Section 3.2"},{"comment":"The parameter labels in Figure 8 (overcommit, watermark, reserved) are dense and the marker styles for the three scheduler families are difficult to distinguish; add a clearer legend or separate subplots for each scheduler.","section":"Section 5.3, Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a useful, practical scheduler and includes an open-source implementation, which is a strength. My main concern is that the empirical results in Table 1 contradict the paper's precision and zero-eviction claims, and the distribution-shift scenario that is most dangerous for a history-based predictor is not directly evaluated. These are fixable with additional analysis and revised claims rather than being fundamental flaws in the approach, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core scheduling rule is the one thing worth remembering: instead of summing worst-case memory or ignoring output memory entirely, it sorts the running batch by predicted remaining length and computes the peak memory at each request-completion point (Eq 3), admitting a queued request only if that peak fits (Eq 4). That is a clean, sensible heuristic, and the math is coherent under synchronous decode. The paper is honest about the baseline age and ships code, and the observed goodput gains are plausible if the memory estimate holds.\n\nWhat is not new: predicting output lengths from a recent history window. S3 already did that, and it isn't cited. The window size and reserved ratio are hand-tuned rather than derived.\n\nThe soft spots are in proportion: one internal contradiction and one unaddressed assumption. The text says 'all scheduled requests are completed successfully,' but Table 1 shows Past-Future evicting 6.86% of requests at reserved=3% on Distribution-1. That doesn't kill the idea, but it directly contradicts the precision claim and needs explaining. The larger issue is stationarity: the admission decision treats the recent output-length distribution as the distribution for the current batch. The paper's own Figure 3 shows hybrid/API workloads have low global similarity, and there is no detection or adaptation for a shift inside the window. Figure 8 concatenates workloads but reports only aggregate eviction/goodput, so a transient collapse after each cut would be hidden. That concern lands.\n\nEnd-to-end comparisons are against December 2023 versions of vLLM, TGI, and DeepSpeed; they disclose this, but it limits what the 2-3x number means today.\n\nWho gets value: systems researchers working on LLM serving scheduling and anyone building continuous batching. It deserves a serious referee. I'd send it out with a request to reconcile Table 1, add a distribution-shift experiment that isolates the transient, and rerun against current frameworks. The core idea is worth one round of revision.","headline":"Plausible scheduling heuristic with a sound memory model, but the paper overclaims precision and misses uncited prior work on output-length prediction.","tokens_in":24266,"tokens_out":2387,"would_cite":true,"duration_ms":26210,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Admitting LLM requests based on projected peak future memory, forecast from historical output lengths, lets a serving system approach theoretical memory utilization while keeping SLA-satisfying throughput under heavy load.","keywords":["LLM serving","continuous batching","request scheduling","KV cache memory management","SLA guarantees","goodput","output length prediction","Past-Future scheduler"],"falsifier":"Run a serving workload with a known, abrupt change in output-length distribution halfway through a history window—for example, the first half of requests finish in 20–50 tokens and the second half in 2000–4000 tokens—and compare the scheduler's predicted $M^*$ against the actual peak memory needed to finish the batch. If $M^*$ stays under capacity while actual memory demand exceeds it, the central claim fails at its load-bearing assumption.","tokens_in":23227,"feed_emoji":"⚡","tokens_out":11345,"duration_ms":107719,"temperature":0.7,"pith_summary":"This paper claims that an LLM serving system can admit incoming requests much more aggressively without causing evictions if it forecasts the peak memory the running batch will need at every future decoding step. The forecast comes from two moves: sample each request's likely remaining output length from the recent history of finished requests, then compute the maximum memory occupancy the batch would hit at any future completion point, admitting a queued request only when that projected peak fits in memory. The authors implemented this Past-Future scheduler in a serving framework called LightLLM and report that under heavy load it meets SLA constraints while delivering up to 2–3 times the goodput of conservative schedulers, which over-reserve memory, and aggressive schedulers, which under-reserve memory and evict requests. If correct, the paper turns request scheduling from a static reservation problem into a dynamic peak-memory forecast, which matters for any deployment where output lengths vary widely.","feed_headline":"History-based scheduler lifts SLA-meeting LLM throughput up to 3x","feed_subtitle":"Predicting output lengths from recent requests lets LightLLM run memory near peak without evictions.","key_machinery":"The Past-Future scheduler is the central object: a parameter-free admission-control rule combining two estimators. The first is $P(l) = C(l, L_h)/w$ (Equation 1), the empirical output-length distribution of the $w$ most recently finished requests. The second is the future-memory profile: requests are sorted by predicted remaining length, and $M_i = (\\sum_{j=1}^{i}(l_j^p + l_j^t)) + (\\hat{l}_i^t - l_i^t) \\cdot i$ (Equation 3) gives the batch's memory occupancy at the moment the $i$-th request finishes, with $M^* = \\max_i M_i$ (Equation 4) the peak needed to complete the batch. The admission rule is simply $M^* \\le M$ (Algorithm 1), while running requests are resampled from $P(l > l_i^t)$ at each step so the projection tightens as requests make progress.","core_discovery":"On the paper's own terms, the central claim is that the future memory requirement of a running batch, not its current memory usage, is the right admission criterion, and that this future demand can be estimated to near-practical accuracy from the empirical distribution of recent output lengths. The scheduler sorts requests by predicted remaining length, computes at each completion point the occupancy $M_i$ as the sum of the memory of requests not yet finished at that point, and takes the maximum $M^* = \\max_i M_i$; it admits a queued request exactly when $M^* \\le M$ (Equations 2–4). It re-samples the predicted remaining length of every running request each step from the conditional distribution $P(l > l_i^t)$, so the projection tracks partial progress rather than assuming a fixed final length. The paper reports that this yields memory utilization close to the theoretical optimum (the case where true output lengths would be known) with far fewer evictions than aggressive scheduling, and better time-to-first-token and max-time-per-output-token SLA compliance under rising concurrency.","pith_inferences":["The paper leaves implicit that the same $M^*$ projection could drive proactive load balancing: a node whose projected peak is about to exceed capacity could direct new requests to a less-loaded peer before eviction becomes necessary.","A natural testable extension is to add a distribution-shift detector that resets the historical window; the paper's own trace analysis shows hybrid API workloads drift over hours, and the scheduler currently has no explicit mechanism to react within a window.","The authors' future-work suggestion about dynamic instance scaling follows directly: if $M^*$ reliably predicts when a node will saturate, it can trigger scaling decisions earlier than utilization metrics.","The paper itself notes that its head-to-head framework comparison reflects December 2023 versions of all frameworks, so the 2–3x gain is tied to those baselines; re-running the comparison against current releases is the natural check."],"forward_implications":["Admission control becomes a forecast of peak future memory rather than a static check of current usage, so batches can run closer to capacity without inviting later evictions.","The scheduler removes the need to hand-tune a memory watermark for aggressive scheduling or an overcommit ratio for conservative scheduling; a single reserved-memory percentage suffices across decode-heavy, balanced, and prefill-heavy workloads.","Because the prediction is parameter-free and model-independent, the same scheduling rule applies to any autoregressive model and adds negligible overhead.","Under rising client concurrency, goodput stabilizes near the hardware's SLA-limited maximum instead of degrading, which is the regime where current schedulers lose the most throughput."],"supporting_citations":[{"why":"Introduces continuous batching, the iteration-level serving model that the Past-Future scheduler controls.","marker":"[40]"},{"why":"Provides the paged KV-cache memory-management baseline and the aggressive scheduler that the evaluation compares against.","marker":"[13]"},{"why":"Supplies the real-world request traces whose adjacent-window output-length similarity motivates the history-based prediction.","marker":"[34]"},{"why":"Represents the conservative scheduler baseline that over-reserves memory in the end-to-end comparison.","marker":"[12]"},{"why":"Represents the conservative scheduler baseline with a split-fuse strategy used in the end-to-end comparison.","marker":"[17]"},{"why":"Supplies the open-weights models used for the main scheduler evaluations.","marker":"[33]"},{"why":"Provides an additional online serving trace used to validate the output-length distribution similarity observation.","marker":"[25]"}],"fun_headline_variants":["Predicting LLM output lengths from history lifts SLA goodput 3x","Past-future scheduler predicts memory needs, triples LLM serving goodput","LightLLM scheduler uses output-length history to triple SLA goodput","Memory estimation from past outputs cuts LLM evictions, raises goodput 3x","History-aware scheduler for LLM serving triples SLA-compliant goodput"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the output lengths of requests arriving in the near future resemble the output lengths of recently finished requests; if that distribution shifts while the scheduler is still using old history, its projected peak memory will be wrong and it will either admit too many requests or waste memory.","fun_headline_variants_meta":{"raw":{"variants":["Predicting LLM output lengths from history lifts SLA goodput 3x","Past-future scheduler predicts memory needs, triples LLM serving goodput","LightLLM scheduler uses output-length history to triple SLA goodput","Memory estimation from past outputs cuts LLM evictions, raises goodput 3x","History-aware scheduler for LLM serving triples SLA-compliant goodput"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00115,"raw_usage":{"total_tokens":4819,"prompt_tokens":1049,"completion_tokens":3770,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":3669}},"tokens_in":665,"tokens_out":3770,"duration_ms":27588,"temperature":1.0,"reasoning_tokens":3669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:38:47.633146+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a serving workload with a known, abrupt change in output-length distribution halfway through a history window—for example, the first half of requests finish in 20–50 tokens and the second half in 2000–4000 tokens—and compare the scheduler's predicted $M^*$ against the actual peak memory needed to finish the batch. If $M^*$ stays under capacity while actual memory demand exceeds it, the central claim fails at its load-bearing assumption.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the conservative scheduler baseline that over-reserves memory in the end-to-end comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the open-weights models used for the main scheduler evaluations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides an additional online serving trace used to validate the output-length distribution similarity observation."}],"review_version":1}