{"id":"f774b38a-b240-4964-ba4a-c4cf6c34aca9","arxiv_id":"2604.17633","paper_version":2,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"SLO-Guard, a crash-aware two-phase autotuner for vLLM serving, achieves no best-latency improvement over random search but demonstrates more consistent budget allocation across 150 trials on Qwen2-1.5B/A100.","lead":"The paper presents SLO-Guard, a two-phase autotuner for vLLM LLM serving that treats configuration crashes as first-class observations. It finds that while SLO-Guard doesn't achieve lower latency than random search, it allocates a fixed tuning budget more consistently—more trials in the fast-serving regime and tighter cross-seed variance.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The consistency advantage is well-measured but causally unattributed: with zero crashes across 300 trials, the crash-aware mechanisms are never exercised, and the observed advantage likely stems from the TBA→TPE handoff structure on a single-knob bimodal landscape rather than from crash-awareness pe","rationale":"The reader's verdict of CONDITIONAL is appropriate. The empirical consistency result is real, replicated across two harnesses, transparently reported with per-seed data, and supported by public code. The paper is commendably honest about its limitations — it explicitly states it does not claim a best-latency win, acknowledges the bimodal single-knob structure, and lists the missing ablations. These are not the marks of an overclaiming paper; they are the marks of a careful case study whose framing slightly exceeds its evidence. The CONDITIONAL verdict captures this precisely: the consistency finding stands, but the 'crash-aware' attribution is unverified. The metadata mismatch (abstract describes a different paper about multilingual pretraining) is likely an arXiv submission error rather than a substantive concern about the argument, though it should be corrected. My concern aligns exactly with the reader's weakest_assumption: the crash-aware mechanisms are never exercised, and the consistency advantage could be entirely attributable to the TBA→TPE handoff structure. The concrete test I propose (stripped ablation without crash-awareness components) would settle whether the framing is justified or should be revised to 'two-phase warm-started search.' Until that test is run, CONDITIONAL remains the correct verdict — the consistency result is sound, but the causal story connecting it to crash-awareness is incomplete.","tokens_in":12790,"tokens_out":2407,"duration_ms":57407,"concrete_test":"Run the TBA-only ablation (no TPE handoff) and a stripped variant (TBA→TPE handoff with crash-awareness mechanisms disabled: no bad-region tracker, no crash encoding, no taxonomy-conditioned tracking) at the same 5-seed × 15-trial scale under the concurrent harness. If the stripped variant achieves statistically equivalent fast-cluster counts and post-hit consistency to full SLO-Guard (Mann-Whitney p > 0.05 on both metrics), then the crash-awareness components are not load-bearing and the title/framing should be revised to 'two-phase warm-started search.' If TBA-only matches SLO-Guard, then the TPE handoff itself is not load-bearing either, narrowing the contribution further.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim — that SLO-Guard allocates budget more consistently than random search — is supported by replicated, honestly-reported statistics. However, the causal attribution of this advantage to crash-awareness is unsupported by the evidence. Zero crashes occurred across all 300 trials (Table 1, Table 2). The three crash-aware mechanisms — the bad-region tracker (Algorithm 1, line 12), crash encoding as extreme constraint violations (Section 3.4), and the crash taxonomy (Section 3.5) — are never empirically exercised. The consistency advantage is almost certainly explained by a simpler mechanism: TBA-explore runs for 6 trials, then hands off to warm-started TPE at trial 7 (Section 3.4, confirmed in all runs). TPE, once warm-started with feasible-region history, concentrates sampling in the fast regime — a well-known property of TPE density-ratio optimization. Random search, by contrast, continues to sample uniformly across the full space, including the slow regime. This explanation requires no crash-awareness whatsoever. The search space structure compounds this: Section 5.5 shows that enforce_eager (a single binary knob) explains >95% of latency variance, making the problem essentially binary regime discovery. The ablations that would disentangle crash-awareness from the handoff structure — TBA-only vs TBA-TPE (Section 5.7, item 1), cold-start TPE vs warm-started TPE (item 2), and repair map on/off (item 3) — are listed but explicitly not run. Without these, the paper cannot distinguish 'crash-aware search helps' from 'TPE with warm-start concentrates on good regions,' which is already well-established. The paper is honest about these limitations (Sections 5.5, 5.7, 7), but the title and framing ('Crash-Aware') still outpace what the evidence supports.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"The paper presents SLO-Guard, a crash-aware two-phase autotuner for vLLM serving that combines a Thermal Budget Annealing (TBA) exploration phase with a warm-started TPE exploitation phase. The central empirical claim is that SLO-Guard allocates a fixed 15-trial tuning budget more consistently than uniform random search—more fast-regime trials (10.20 vs 7.40/15, one-sided p=0.014), higher post-hit consistency (0.876 vs 0.539, p=0.010), and 4.4× tighter cross-seed best-latency variance under concurrent load—while being statistically tied on best-achieved latency (p=0.84). The study uses a pre-specified five-seed protocol with Mann-Whitney tests and Holm-Bonferroni correction, and replicates across sequential and concurrent harness conditions.","tokens_in":13073,"tokens_out":3045,"duration_ms":88768,"significance":"The empirical methodology is a genuine strength: pre-specified seeds, non-parametric tests with multiple-comparison correction, explicit effect-size reporting, honest non-claiming of best-latency superiority, and a harness-replication analysis that corrects a measurement defect. The public code repository with raw JSONL logs and commit hashes is a real reproducibility contribution. However, a critical submission error (abstract/title mismatch) and a framing-evidence gap substantially undermine the manuscript as it stands.","major_comments":[{"comment":"Abstract/title mismatch. The submitted abstract and title ('Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining') describe a completely different paper about multilingual pretraining. The full text is about SLO-Guard for LLM serving autotuning. This is a fundamental submission error that must be corrected before content assessment can proceed.","section":null},{"comment":"Crash-awareness framing is unsupported by the experimental evidence. The paper's title, abstract, and §1 frame SLO-Guard as 'crash-aware,' and three of four contributions reference crash-awareness. However, zero crashes occurred across all 300 trials (Table 1, Table 2). The three crash-aware mechanisms—the bad-region tracker (Algorithm 1, line 12), crash encoding as extreme constraint violations (§3.4), and the crash taxonomy (§3.5)—are never empirically exercised. The observed consistency advantage is more parsimoniously explained by the TBA→TPE handoff structure: TBA explores for 6 trials, then hands off to warm-started TPE at trial 7 (§3.4, confirmed in all runs), and TPE with feasible-region history concentrates sampling in the fast regime—a standard property of TPE density-ratio optimization. The ablations that would disentangle crash-awareness from the handoff structure—TBA-only vs","section":null},{"comment":"TBA-TPE (§5.7, item 1), cold-start TPE vs warm-started TPE (item 2), and repair map on/off (item 3)—are listed but explicitly not run. Without at least one of these ablations, the causal attribution to crash-awareness is unsupported. The paper should either run a minimal ablation or reframe the contribution around two-phase warm-started search rather than crash-awareness.","section":null},{"comment":"Single-knob landscape limits generalizability. §5.5 shows that enforce_eager (a single binary knob) explains >95% of latency variance, making the problem essentially binary regime discovery. The §5.7 item 4 ablation (fix enforce_eager) would test whether the consistency advantage persists as a within-cluster fine-tuning effect; its absence leaves the practical scope of the contribution unclear.","section":null}],"minor_comments":[{"comment":"Reference [12] (the author's own TBA preprint) is cited as 'arXiv preprint pending; update citation details after public release.' A load-bearing methodological dependency on an unavailable reference is problematic for review; the key TBA details should be self-contained in the present manuscript.","section":null},{"comment":"§3.4, Algorithm 2: the handoff parameters t_min, t_max, n_min_f, n_min_b are listed as free parameters but their values for the 15-trial budget are not explicitly stated (only the empirical result that handoff occurs at trial 7).","section":null},{"comment":"§5.4: the sequential-harness best-latency variance ratio is 0.84× (Table 2), meaning random search was slightly tighter under sequential dispatch. This is mentioned but not discussed; a sentence explaining why would help.","section":null},{"comment":"Table 3: seed 242 for SLO-Guard shows first fast trial at trial 3, not trial 7 as the §5.6 narrative suggests. The representative trajectory (Figure 6) should be clarified as seed-specific rather than general.","section":null},{"comment":"References [1] and [21] have 'Verify final bibliographic details before submission' placeholders.","section":null}],"recommendation":"major_revision","confidential_remarks":"The abstract/title mismatch suggests either a submission error or a template reuse issue. The author should be asked to confirm the correct manuscript is being reviewed. Separately, the paper's framing as 'crash-aware' when zero crashes occurred across 300 trials is a significant overclaim relative to the evidence. The underlying consistency result is sound and honestly reported; the paper could be publishable if reframed around two-phase warm-started search or if a minimal ablation isolating the crash-awareness contribution is added. The dependency on the author's own unavailable preprint [12] for the TBA method is also a concern for independent assessment."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and substantive review. The referee correctly identifies a submission error (abstract/title mismatch) and raises a serious framing concern about the crash-awareness label. We address each point below.","responses":[{"response":"The referee is correct. This is a submission error: the wrong abstract and title were uploaded. The full text, keywords, and all content are the SLO-Guard paper. We will replace the abstract and title with the correct ones (already present in the full text) in the next version. We apologize for the error.","revision_made":"yes","referee_comment":"Abstract/title mismatch: the submitted abstract and title describe a completely different paper about multilingual pretraining, while the full text is about SLO-Guard for LLM serving autotuning."},{"response":"The referee raises a fair and important point. We concede that the crash-awareness framing is not supported by the current experimental evidence: with zero crashes across 300 trials, the crash-specific mechanisms (bad-region tracker, crash encoding, crash taxonomy) are never exercised, and the observed consistency advantage is more parsimoniously explained by the TBA-to-TPE warm-start handoff. We will reframe the contribution around two-phase warm-started search rather than crash-awareness. Specifically: (1) the title and abstract will be revised to remove crash-awareness as the central framing; (2) the crash-aware mechanisms will be repositioned as design features for crash-prone settings rather than empirically validated contributions; (3) the contributions list will be rewritten to center on the two-phase warm-started search structure; (4) we will run at least one of the listed ablations—specifically, cold-start TPE vs warm-started TPE (item 2)—to provide causal evidence for the warm-start handoff as the source of the consistency advantage. We agree that without an ablation, attribution to crash-awareness is unsupported, and we will not make that attribution in the revision.","revision_made":"yes","referee_comment":"Crash-awareness framing is unsupported by the experimental evidence: zero crashes occurred across all 300 trials, the three crash-aware mechanisms are never empirically exercised, and the consistency advantage is more parsimoniously explained by the TBA-to-TPE handoff structure. The ablations that would disentangle crash-awareness from the handoff structure are listed but not run."},{"response":"The referee is correct that the single-knob structure limits generalizability and that the enforce_eager-fixed ablation would clarify whether the consistency advantage persists as a within-cluster fine-tuning effect. We acknowledge this limitation and will add it explicitly to the threats-to-validity section. We will also run the enforce_eager-fixed ablation (item 4) in the revision. If the consistency advantage collapses under this ablation, we will state this plainly and scope the contribution to regime-discovery consistency on bimodal landscapes. We agree that the current scope claim is incomplete without this test.","revision_made":"yes","referee_comment":"Single-knob landscape limits generalizability: enforce_eager explains over 95% of latency variance, making the problem essentially binary regime discovery. The ablation fixing enforce_eager (item 4) is absent, leaving the practical scope unclear."}],"tokens_in":12608,"tokens_out":683,"duration_ms":113851,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Here's my read on the SLO-Guard paper (the actual text — note the arXiv metadata lists a completely different paper by Körner et al. on multilingual pretraining; that's a submission error worth flagging to the authors, but the full text is clearly the LLM serving autotuner by Lysenstøen).","headline":"Paper is honest and methodologically careful, but its crash-aware framing outpaces the evidence — zero crashes occurred across 300 trials, and the consistency advantage likely comes from the TBA→TPE handoff, not crash-awareness. Still a solid empirical contribution worth a serious referee.","tokens_in":13706,"tokens_out":844,"would_cite":false,"duration_ms":28805,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Tuning vLLM: Crash-Aware Search Spends Budget Better","keywords":[],"falsifier":"If, in a search space where crashes actually occur, SLO-Guard's consistency advantage disappears or reverses, or if the TBA-to-TPE handoff without crash encoding achieves the same consistency, the central framing of crash-awareness as the operative mechanism would be falsified.","tokens_in":12904,"feed_emoji":"","tokens_out":913,"duration_ms":84427,"temperature":0.7,"pith_summary":"This paper presents SLO-Guard, a two-phase autotuner for vLLM serving configurations that combines a feasibility-first annealing exploration phase with a warm-started Tree-structured Parzen Estimator exploitation phase. The central claim is not that SLO-Guard finds a better final configuration than random search, but that it spends a fixed 15-trial tuning budget more consistently: it allocates more trials to the fast-serving regime (10.20 vs 7.40 out of 15, p=0.014), maintains higher post-discovery consistency (0.876 vs 0.539, p=0.010), and yields 4.4x tighter cross-seed best-latency variance under concurrent load, while being statistically tied on best-achieved latency (p=0.84). The method treats crashes as first-class observations encoded as extreme constraint violations, includes a configuration-repair pass and GPU-aware KV-cache memory guard, and introduces a four-category crash taxonomy. The search space is sharply bimodal, dominated by a single binary knob (enforce_eager), which the paper acknowledges limits the generality of the claim.","feed_headline":"Crash-Aware Tuner Spends Budget More Consistently for LLM Serving","feed_subtitle":"SLO-Guard matches random search on best latency but allocates 38% more trials to the fast regime and cuts variance 4.4x, suggesting budget-p","key_machinery":"SLO-Guard combines a Thermal Budget Annealing (TBA) exploration phase that searches for feasible configurations while tracking bad regions, with a warm-started Optuna TPE exploitation phase that replays all exploration history including crashes encoded as extreme constraint violations. A configuration-repair map and GPU-aware KV-cache memory guard remove implementation-level impossibilities before the optimizer sees them. The handoff from TBA to TPE occurs empirically at trial 7 of 15, after sufficient feasible and crash history accumulates.","core_discovery":"The paper's central discovery is that under a small tuning budget with a bimodal, regime-switching search space, a two-phase optimizer (annealing exploration followed by warm-started TPE exploitation) does not beat random search on peak performance but does allocate the fixed budget more predictably: more trials land in the fast-serving regime, the optimizer stays in that regime more reliably after finding it, and cross-seed variance on best latency is 4.4x tighter. This consistency advantage survives replication across two independent measurement harnesses (sequential and concurrent request dispatch), both yielding statistically equivalent consistency metrics. The bimodal structure of the v","pith_inferences":[],"forward_implications":["Operators running repeated tuning jobs on vLLM serving configurations can expect more predictable budget usage with SLO-Guard, even though the best configuration found is no better than random search's.","The bimodal structure dominated by enforce_eager suggests that for this specific model and hardware, the first-order tuning task is regime discovery, not fine-grained multi-knob optimization.","The finding that sequential dispatch masks variance structure that concurrent dispatch reveals has implications for how LLM serving benchmarks should be designed: serialized load generators can hide consistency differences that matter in production.","If the consistency advantage is attributable to the TBA-to-TPE handoff structure rather than crash-awareness per se (since zero crashes occurred), the approach may generalize to other regime-switching optimization problems even without crash-prone search spaces."],"fun_headline_variants":["Copying Precedes Translation in Multilingual Pretraining","Translation Emerges in Two Phases During Multilingual Pretraining","Models Learn Token Copying Before Cross-Lingual Translation","Cross-Lingual Generalization Develops Through Copying First","Two-Phase Translation Development in Multilingual LLM Pretraining"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper frames itself around crash-awareness, but zero crashes occurred across all 150 concurrent-harness trials and 150 sequential-harness trials, meaning the crash-aware mechanism was never empirically exercised. The consistency advantage could be entirely attributable to the TBA-to-TPE handoff structure rather than to crash-awareness, and the ablation that would disentangle these (TBA-only vs TBA-TPE, repair map on/off) is listed but not run.","fun_headline_variants_meta":{"raw":{"variants":["Copying Precedes Translation in Multilingual Pretraining","Translation Emerges in Two Phases During Multilingual Pretraining","Models Learn Token Copying Before Cross-Lingual Translation","Cross-Lingual Generalization Develops Through Copying First","Two-Phase Translation Development in Multilingual LLM Pretraining"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":608,"prompt_tokens":543,"completion_tokens":65,"prompt_tokens_details":null},"tokens_in":543,"tokens_out":65,"duration_ms":18531,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-05T16:22:52.010121+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If, in a search space where crashes actually occur, SLO-Guard's consistency advantage disappears or reverses, or if the TBA-to-TPE handoff without crash encoding achieves the same consistency, the central framing of crash-awareness as the operative mechanism would be falsified.","supporting_citations":[],"review_version":2}