{"id":"611f9901-9c3d-4466-b863-2cec839b694e","arxiv_id":"2506.12204","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A semantic scheduler for LLM inference uses urgency labels and estimated remaining compute to cut waiting times for urgent requests, tested on emergency medical data.","lead":"This paper introduces a scheduler for LLM-serving systems that reads the content of incoming requests, assigns each an emergency level, and prioritizes urgent ones while tracking remaining compute time. It reports large waiting-time reductions for high-urgency requests in an emergency-medicine workload, which matters because AI systems are being used for time-critical response tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6) is not the minimizer of Eq. (5); the cache-or-recompute rule in Section 3.3 is internally inconsistent and is not validated in isolation.","rationale":"The reader's weakest-assumption diagnosis is correct: the cache-or-recompute derivation in Section 3.3 is the least secure load-bearing component. The paper has genuine independent value—code and data are released, the simulated experiments vary predictor accuracy and load, and the priority-scheduling direction is plausible and supported by the real-dataset comparison for urgency-0 requests. However, the mathematical error in Eqs. (5)-(6) is not cosmetic: it determines whether Algorithm 4 offloads or discards KV cache, and with the reported constants it would always choose discard. Since the real-dataset experiments do not isolate cache management, the claim that the full system including adaptive cache management drives the speedups is unsupported. The correct response is to keep the CONDITIONAL verdict: the scheduling contribution can be separated and may survive, but a corrected derivation and an ablation are required before the full system claim can be accepted. The paper should not be rejected outright, because the concern is localized and fixable.","tokens_in":16186,"tokens_out":9732,"duration_ms":117260,"concrete_test":"Independently minimize C(m'_*)=βm'_*+γ1[0.5(m'-m'_*)^2+n(m'-m'_*)+0.5(m'-m'_*)]+γ2(m'-m'_*) over 0≤m'_*≤m' using the Appendix B coefficients and the n,m' ranges used in the experiments. Compare the true optimum to Eq. (6). Then run a memory-pressure ablation with three configurations: always discard, always offload, and the Eq. (6) rule; if the true optimum is always 0 or m' while Eq. (6) predicts an interior value, replace the rule and re-test to see whether cache management changes waiting time at all.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.3, the paper derives the total cost of saving m'_* of m' already-decoded tokens as C(m'_*)=βm'_*+γ1[0.5(m'-m'_*)^2+n(m'-m'_*)+0.5(m'-m'_*)]+γ2(m'-m'_*) (Eq. 5). The continuation cost for the remaining m-m' tokens is independent of m'_*, so it does not affect the optimum. Differentiating C with respect to m'_* gives dC/dm'_*=β-γ1[(m'-m'_*)+n+0.5]-γ2, which is monotone in m'_*; hence the optimum is at a boundary (save all or save none), not the interior formula claimed in Eq. (6). Eq. (6) also has no m' dependence, and with the Appendix B profiled coefficients (β≈1e-4, γ1≈1e-8, γ2≈1e-2) it yields m'_*=0 for all practical n, making the offload-to-CPU branch of Algorithm 4 dead code. Because adaptive cache management is one of the paper's four claimed contributions, and the real-dataset experiments do not ablate this component, the contribution of the cache-or-recompute strategy to the reported speedups is not established. This does not invalidate the priority-scheduling part of the system, but it is a load-bearing gap in the full claim as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM inference scheduler that assigns priorities to requests based on semantic urgency, then uses priority-ordered scheduling, stage-aware batching, KV-cache eviction, and a cache-or-recompute decision to minimize average waiting time while respecting priority constraints. The system is evaluated in simulations and on an emergency medical services dataset, reporting large speedups for the most urgent requests compared with FCFS, SJF, and HPJF baselines. The authors also release code and data.","tokens_in":16456,"tokens_out":16734,"duration_ms":289520,"significance":"The direction is timely and practically motivated: semantic awareness in LLM serving could materially improve emergency and high-stakes applications. The paper is commendable for shipping code and data, and the empirical trend—that prioritizing urgent requests reduces their waiting time—is plausible and demonstrated across several model/GPU configurations. However, the central theoretical claim of optimality is unproven, and the cache-or-recompute derivation in Section 3.3 is mathematically inconsistent. Since the experiments do not ablate the cache-management components, the contribution of those components to the reported speedups is not established. With revision, the priority-scheduling contribution could be salvageable, but the paper in its current form overclaims.","major_comments":[{"comment":"The cache-or-recompute derivation is internally inconsistent. In Eq. (5), the recomputation term sums gamma_1 * i + gamma_2 over i = n+1 to n+(m'-m'_*), which is the cost of recomputing the first m'-m'_* decoded tokens, not the tokens that remain after saving m'_* caches; the correct sum should run from n+m'_*+1 to n+m'. Moreover, Eq. (6) is not obtained by differentiating Eq. (5) with respect to the correct variable: the stationary point of Eq. (5) is m'_* = m' - (beta - gamma_1*n - 0.5*gamma_1 - gamma_2)/gamma_1, which depends on m', whereas Eq. (6) has no m' dependence and is dimensionally inconsistent because gamma_1 and gamma_2 have incompatible units. With the profiled coefficients in Appendix B, the correct threshold is negative (beta << gamma_2), so the optimum is to save all decoded tokens, but Eq. (6) returns m'_* = 0 for typical n, making the OFFLOADTOCPU branch in Algorithm 4 dead code and causing the system to recompute tokens that are far cheaper to save. Since the experiments in Section 4 do not ablate this component, its contribution to the reported speedups is not established.","section":"Section 3.3, Eqs. (5)-(6)"},{"comment":"The paper claims that the proposed scheduling policy minimizes the average waiting time in Eqs. (1)-(2), but no proof of optimality is provided. The lexicographic ordering by (f_e, f_t) is asserted as the scheduling rule; there is no theorem showing that this policy attains the minimum of Eq. (1) subject to Eq. (2), nor any analysis of suboptimality. The phrase 'designed to minimize' in the abstract is weaker, but the optimization setup in Section 3.2 invites a formal claim. Please either provide a proof for the single-server preemptive-resume setting or explicitly state that the algorithm is a heuristic.","section":"Section 3.2 and Algorithm 2"},{"comment":"The system only re-schedules at iteration boundaries; it does not preempt a running batch. Consequently, a low-urgency request that completes an ongoing decoding iteration after a high-urgency request has arrived will finish before the high-urgency request, violating Eq. (2) if the high-urgency request has lower rank. The paper should state the preemption granularity explicitly and either modify the algorithm to support true preemption or weaken the constraint to hold only at iteration boundaries.","section":"Algorithm 2, Section 3.3"},{"comment":"The experiments compare the full system against FCFS, SJF, and HPJF, but no ablation disables stage-aware batching, priority-based eviction, or cache-or-recompute. The real-dataset results (Section 4.2) are only versus FCFS, so the large speedups cannot be attributed to the four claimed components individually. Adding ablations (e.g., semantic priority only, without cache management or stage-aware batching) would clarify which components are responsible for the observed gains and would also expose the effect of the defective cache-or-recompute formula.","section":"Section 4"}],"minor_comments":[{"comment":"\"Reloading that cache on resumption takes beta m' units of time\" should read beta m'_*, since only the saved tokens are reloaded.","section":"Section 3.3, sentence before Eq. (5)"},{"comment":"The phrase \"the waiting time between the time interval between the arrival time and the completion time\" is garbled; Eq. (1) defines it as completion time minus arrival time, which is more precisely the flow time or response time.","section":"Section 1, Introduction"},{"comment":"There is a typo in \"asynchronoulsy\" (should be \"asynchronously\").","section":"Section 3.3"},{"comment":"The phrase \"unde memory eviction\" contains a typo; it should likely be \"under memory eviction.\"","section":"Section 1"},{"comment":"The \"Avg.\" row renders as \"0.210.61\" without a separator; it should be \"0.21 0.61\" or similar.","section":"Table 1"},{"comment":"The coefficients are listed without identifying which number corresponds to alpha_1, alpha_2, gamma_1, gamma_2, and beta for each model/GPU setting; please specify the mapping (e.g., \"prefill quadratic coefficient alpha_1 = ...\").","section":"Appendix B"},{"comment":"The dataset from (Yu et al., 2024) is titled \"AIPatient: Simulating patients with EHRs and LLM powered agentic workflow,\" which suggests the conversations may be synthetic; calling it a \"real-world dataset\" is misleading unless the conversations are actual hospital records. Please clarify the provenance.","section":"Section 4.2 and Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible empirical core but the theoretical and cache-management parts need substantial work. The cache-or-recompute error is fixable, but it requires re-deriving the optimal saving rule and either re-running experiments or adding ablations that clearly separate the contributions. The 'real-dataset' label should also be reconciled with the cited AIPatient source. I recommend major revision rather than rejection, as the priority-scheduling direction is valuable and the implementation is shared."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea is worth a look: use content-derived urgency as a priority signal for LLM request scheduling, with SJF tie-breaking and stage-aware batching. That is new in the serving literature, and the experiments show large waiting-time reductions for the most urgent requests. The paper is not as clean as it presents itself.\n\nWhat is genuinely useful: the semantic urgency dimension, the dual-heap design, and the concrete EMS testbed. The simulation studies predictor error honestly, and the real-dataset results show a strong effect in the right direction. The speed-up numbers are large but in a plausible range for a priority scheduler.\n\nThe soft spots are real. First, 'optimal' is claimed without proof. The schedule is essentially highest-priority-job-first with shortest-job-first tie-breaking, and there is no comparison to an optimal scheduler; the empirical claim would survive a proof or a benchmark. Second, the cache-or-recompute derivation in Section 3.3 is wrong. Eq. (5) is convex in m'_*, so its minimizer is at a boundary, not the interior formula of Eq. (6). With the profiled coefficients in Appendix B, the derivative is negative over the whole interval, so the formula returns zero and the offload branch of Algorithm 4 never runs. The contribution of adaptive memory management to the reported speedups is therefore unquantified. Third, the real experiments compare only with FCFS and report only urgency-0 waiting times, with no error bars and no check on whether lower-priority requests are starved. It is also not stated whether the real experiments use groundtruth urgency labels or the predictor.\n\nNone of this is fatal to the main idea. The priority-scheduling direction is sound, and the empirical core holds up. But the optimality claim and the cache-save analysis need to be either fixed or removed. This paper deserves a serious referee. I would send it out and expect major revision, mainly for the derivation and for a broader evaluation.","headline":"A genuinely new priority-scheduling idea for LLM serving with a strong empirical core, but the cache-or-recompute math has a real error and the optimality claim is unproven.","tokens_in":17022,"tokens_out":5385,"would_cite":true,"duration_ms":69154,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantic scheduling prioritizes LLM requests by urgency and estimated remaining time, minimizing waiting time under priority constraints and outperforming FCFS on emergency data.","keywords":["semantic scheduling","LLM inference serving","priority-aware scheduling","KV cache eviction","preemptive scheduling","emergency severity index","graded poset","waiting time minimization"],"falsifier":"Compute the optimal $m'_*$ by numerically minimizing the total-time expression in Eq. (5) for the paper's profiled coefficients and compare with the closed form in Eq. (6); the printed summation in Eq. (4) and the expression in Eq. (5) do not match term-for-term, so a direct minimization would settle whether the threshold is correct. Separately, run an ablation on the real urgency-labeled dataset that replaces the adaptive cache-or-recompute choice with always-save and always-recompute policies, and check whether the adaptive rule still reduces urgency-0 waiting time.","tokens_in":15932,"feed_emoji":"🚨","tokens_out":8996,"duration_ms":87351,"temperature":0.7,"pith_summary":"This paper tries to establish that scheduling decisions for LLM inference should be guided by the semantics of the request, not just arrival time or job length. It proposes a semantic scheduling system in which a small model assigns each incoming prompt an urgency level, and an output-length predictor estimates remaining compute time; a priority queue then selects the highest-urgency, shortest-remaining requests, preempting lower-urgency work when needed. The system also manages GPU memory by evicting the KV cache of low-priority, high-remaining-work requests, choosing between offloading and recomputation via a cost model. If correct, this means emergency and other time-critical LLM requests can bypass ordinary traffic, and the paper reports that on a real emergency-medical dataset the approach outperforms FCFS in all settings for the highest-urgency requests, with speed-ups up to 270x.","feed_headline":"Semantic scheduling speeds urgent LLM requests by up to 270x","feed_subtitle":"Priority-aware batching and KV-cache eviction let critical queries skip routine traffic.","key_machinery":"The load-bearing object is the scheduling priority tuple $(f_e(p), f_t(p))$: the semantic emergency level assigned by a small model, and the estimated remaining computation time from an output-length bucket predictor. The paper formalizes request priorities as a graded poset with a ranking function $\\rho$ that maps urgency levels to ranks, so that all maximal chains between two requests have equal length. The same priority tuple, with signs flipped, orders the eviction heap, so the system always executes the most urgent and shortest-remaining requests and evicts the least urgent and longest-remaining KV caches in $O(\\log n)$ time. The cache-or-recompute policy is carried by a threshold formula $m'_* = \\max(0, \\frac{\\beta - \\gamma_1 n - (\\gamma_1 + 2\\gamma_2)/2}{2\\gamma_1})$, obtained from a quadratic prefill/decode cost model, which decides how many computed tokens' KV cache to keep when a preempted request resumes.","core_discovery":"The paper's central claim is that a content-aware priority queue with preemption can minimize average waiting time under a relative priority constraint in LLM serving. User requests are modeled as a graded poset with urgency rank $\\rho(p)$, and the scheduling objective is to minimize average finish time minus arrival time while respecting that a request may finish before another only if it arrived earlier or has rank at least as high. The proposed algorithm realizes this with a min-heap keyed by urgency and estimated remaining time, a max-heap for eviction keyed by the inverse tuple, stage-aware batching that keeps high-priority decoding from being blocked by lower-priority prefilling, and an adaptive cache-or-recompute threshold that decides how much of an evicted request's KV cache to save. The experiments, both simulated and on a real emergency hospital dataset, support the claim that the approach reduces waiting time for urgency-0 requests relative to FCFS, SJF, and HPJF, with the largest gains under request spikes.","pith_inferences":["An immediate testable extension is to ablate the adaptive cache-or-recompute rule against always-save and always-recompute policies; the real-dataset experiments report only the full system, so the marginal contribution of this component is not yet isolated.","The same dual-heap structure transfers to other semantic dimensions, such as safety-critical alerts, fraud reports, or deadline-bound legal queries, by replacing the ESI urgency scale with any graded priority ordering.","The optimization objective is average waiting time; a variant that minimizes the tail or maximum waiting time for the highest urgency class would match emergency-service service-level guarantees more directly, and the same machinery likely applies."],"forward_implications":["If the central claim holds, urgency-aware scheduling can be added to LLM serving without giving up overall efficiency: low-priority requests are preempted and their KV caches evicted, then re-enter the heap with updated remaining times.","Stage-aware batching removes a specific failure mode where a high-urgency decoding request waits behind a low-urgency prefilling request in the same batch.","The adaptive cache-or-recompute rule means eviction does not always discard all progress on a request; when reloading is cheaper than recomputing, the cache is offloaded, which reduces the latency cost of preemption.","The paper's simulation results imply that predictor quality is a first-order lever: as semantic-predictor error rises from 0.1 to 0.9, the normalized waiting time for urgency-0 requests grows roughly 3.5x, so deployment would require a reliable urgency classifier.","On the real emergency-dataset experiments, urgency-0 requests see up to 270x lower normalized waiting time than FCFS, so the benefit is concentrated exactly where delays are most consequential."],"supporting_citations":[{"why":"Supplies the real EMS dataset with ESI urgency annotations used for the real-dataset waiting-time experiments.","marker":"Yu et al., 2024"},{"why":"Supplies the S3 output-length bucket predictor that estimates each request's remaining computation time for scheduling.","marker":"Jin et al., 2023"},{"why":"Provides the paged-attention serving engine on which the real-dataset implementation runs and whose memory model motivates KV-cache management.","marker":"Kwon et al., 2023"},{"why":"Earlier KV-cache management work that the paper extends to priority-based eviction and cache-or-recompute choices under memory pressure.","marker":"Lee et al., 2024"},{"why":"Shows the cost of conflating prefill and decode stages, informing the stage-aware batching that prevents priority inversions.","marker":"Zhong et al., 2024"},{"why":"Provides the motivating emergency-medical-services application with ESI urgency levels that define the semantic ranks.","marker":"Fan et al., 2025"},{"why":"Provides the second motivating scenario, 911 dispatch with streaming emergency messages that require content-aware prioritization.","marker":"Otal et al., 2024"}],"fun_headline_variants":["Semantic scheduling speeds urgent LLM requests by up to 270x","Urgent LLM queries skip the queue via semantic scheduling","Meaning-aware batching cuts wait for critical LLM prompts","Content-based priority slashes latency for emergency LLM tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The adaptive cache-or-recompute step depends on the paper's quadratic cost model for prefill, decode, and cache load/save times being accurate; if the model or the threshold formula is wrong, eviction can increase latency instead of decreasing it.","fun_headline_variants_meta":{"raw":{"variants":["Semantic scheduling speeds urgent LLM requests by up to 270x","Urgent LLM queries skip the queue via semantic scheduling","Meaning-aware batching cuts wait for critical LLM prompts","Content-based priority slashes latency for emergency LLM tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000772,"raw_usage":{"total_tokens":3398,"prompt_tokens":904,"completion_tokens":2494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":2424}},"tokens_in":520,"tokens_out":2494,"duration_ms":25303,"temperature":1.0,"reasoning_tokens":2424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:57:18.651211+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the optimal $m'_*$ by numerically minimizing the total-time expression in Eq. (5) for the paper's profiled coefficients and compare with the closed form in Eq. (6); the printed summation in Eq. (4) and the expression in Eq. (5) do not match term-for-term, so a direct minimization would settle whether the threshold is correct. Separately, run an ablation on the real urgency-labeled dataset that replaces the adaptive cache-or-recompute choice with always-save and always-recompute policies, and check whether the adaptive rule still reduces urgency-0 waiting time.","supporting_citations":[],"review_version":1}