The reviewed record of science sign in
Pith

arxiv: 2604.17633 · v2 · pith:WQ3MBGKQ · submitted 2026-04-19 · cs.CL

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

Felicia K\"orner , Maria Matveev , Florian Eichin , Gitta Kutyniok , Barbara Plank , Michael A. Hedderich This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-05 16:22 UTCglm-5.2pith:WQ3MBGKQrecord.jsonopen to challenge →

classification cs.CL
keywords translationcapabilitiescopyingcross-lingualdevelopsmultilingualduringearly
0
0 comments X

The pith

Tuning vLLM: Crash-Aware Search Spends Budget Better

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents SLO-Guard, a two-phase autotuner for vLLM serving configurations that combines a feasibility-first annealing exploration phase with a warm-started Tree-structured Parzen Estimator exploitation phase. The central claim is not that SLO-Guard finds a better final configuration than random search, but that it spends a fixed 15-trial tuning budget more consistently: it allocates more trials to the fast-serving regime (10.20 vs 7.40 out of 15, p=0.014), maintains higher post-discovery consistency (0.876 vs 0.539, p=0.010), and yields 4.4x tighter cross-seed best-latency variance under concurrent load, while being statistically tied on best-achieved latency (p=0.84). The method treats crashes as first-class observations encoded as extreme constraint violations, includes a configuration-repair pass and GPU-aware KV-cache memory guard, and introduces a four-category crash taxonomy. The search space is sharply bimodal, dominated by a single binary knob (enforce_eager), which the paper acknowledges limits the generality of the claim.

Core claim

The paper's central discovery is that under a small tuning budget with a bimodal, regime-switching search space, a two-phase optimizer (annealing exploration followed by warm-started TPE exploitation) does not beat random search on peak performance but does allocate the fixed budget more predictably: more trials land in the fast-serving regime, the optimizer stays in that regime more reliably after finding it, and cross-seed variance on best latency is 4.4x tighter. This consistency advantage survives replication across two independent measurement harnesses (sequential and concurrent request dispatch), both yielding statistically equivalent consistency metrics. The bimodal structure of the v

What carries the argument

SLO-Guard combines a Thermal Budget Annealing (TBA) exploration phase that searches for feasible configurations while tracking bad regions, with a warm-started Optuna TPE exploitation phase that replays all exploration history including crashes encoded as extreme constraint violations. A configuration-repair map and GPU-aware KV-cache memory guard remove implementation-level impossibilities before the optimizer sees them. The handoff from TBA to TPE occurs empirically at trial 7 of 15, after sufficient feasible and crash history accumulates.

If this is right

  • Operators running repeated tuning jobs on vLLM serving configurations can expect more predictable budget usage with SLO-Guard, even though the best configuration found is no better than random search's.
  • The bimodal structure dominated by enforce_eager suggests that for this specific model and hardware, the first-order tuning task is regime discovery, not fine-grained multi-knob optimization.
  • The finding that sequential dispatch masks variance structure that concurrent dispatch reveals has implications for how LLM serving benchmarks should be designed: serialized load generators can hide consistency differences that matter in production.
  • If the consistency advantage is attributable to the TBA-to-TPE handoff structure rather than crash-awareness per se (since zero crashes occurred), the approach may generalize to other regime-switching optimization problems even without crash-prone search spaces.

Load-bearing premise

The paper frames itself around crash-awareness, but zero crashes occurred across all 150 concurrent-harness trials and 150 sequential-harness trials, meaning the crash-aware mechanism was never empirically exercised. The consistency advantage could be entirely attributable to the TBA-to-TPE handoff structure rather than to crash-awareness, and the ablation that would disentangle these (TBA-only vs TBA-TPE, repair map on/off) is listed but not run.

What would settle it

If, in a search space where crashes actually occur, SLO-Guard's consistency advantage disappears or reverses, or if the TBA-to-TPE handoff without crash encoding achieves the same consistency, the central framing of crash-awareness as the operative mechanism would be falsified.

Figures

Figures reproduced from arXiv: 2604.17633 by Barbara Plank, Felicia K\"orner, Florian Eichin, Gitta Kutyniok, Maria Matveev, Michael A. Hedderich.

Figure 1
Figure 1. Figure 1: Training dynamics of word-level transla￾tion over the course of pretraining. The stacked areas decompose outputs into correct translations and error types, and the black line tracks overall translation ac￾curacy. Early training (Phase I) is characterized by frequent copying behavior, whereas in later training (Phase II) translation is developed. We argue that studying the final model alone cannot explain w… view at source ↗
Figure 2
Figure 2. Figure 2: Evolution of model performance across diverse linguistic tasks over training. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Multilingual logit lens analysis over training. We aggregate the results across all 72 language pairs. and more weakly 15 → 17 (Fig. 3a). Through￾out Phase II, the layer-wise dynamics of copying change little; the copy-promoting layer transitions established during Phase I persist. However, the target candidate rises steadily while the source can￾didate is progressively down-ranked in Phase II, indicating … view at source ↗
Figure 4
Figure 4. Figure 4: Loss decomposition of WLT samples over time. We decompose the loss of a subsample of WLT predictions onto groups of three consecutive layers Θ(i,i+2) over the training steps. Details on the methodology can be found in Section 6.2; data is detailed in Appendix D. final checkpoint supports this—candidates across languages only separate in the top block ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Parameter swapping over time. We swap different parameter blocks of the final checkpoint into earlier checkpoints and compare WLT accuracy of the resulting model to the (unchanged) original checkpoint. Other combinations are shown in Appendix [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: We report the (smoothed) loss-curve of the [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: System prompt for LLM-based candidate filtering (Claude Sonnet, [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Analysis of directional asymmetry in word￾level translation. We observe that translation per￾formance is directional, favoring English, and, to a lesser degree, Chinese and Japanese as target languages throughout the developmental trajectory of pretraining. C.2 Evaluation We complement the metrics reported in Figs. 1 and 2, with additional statistics about WLT perfor￾mance, alongside further noteworthy obs… view at source ↗
Figure 10
Figure 10. Figure 10: We report the fraction of incorrect translations attributable to source word copying, measured at the [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Decomposition of outputs onto copying types. Context copying is more prevalent for repetition than for WLT. Note that the left plot is restricted to the Latin-script languages, as source word copying is less frequent between scripts (see [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Break-down of accuracy and copying variant, per token overlap bucket. For word pairs with partial token overlap, erroneous source word copying is a more frequent error mode than for pairs with no overlap. For pairs with no token overlap, context copying is more prevalent and decays more slowly throughout training. Translation accuracy is higher for pairs with higher token overlap. 18 [PITH_FULL_IMAGE:fig… view at source ↗
Figure 13
Figure 13. Figure 13: Evolution of layer-wise log probabilities of true translation across token-overlap categories. Fig. 13a shows all translation tasks averaged across language-pair means. Figs. 13b to 13d separate the same data into token-overlap buckets between the source and target word (no overlap, partial overlap, and identical tokens). For the full translation task, the bottom and intermediate layer blocks remain flat … view at source ↗
Figure 14
Figure 14. Figure 14: Evolution of layer-wise log probabilities for different language categories. Evolution of log probabilities across training steps for different language categories. For each language pair, we evaluate the model under teacher forcing and track the logits assigned to candidate translations of the same concept in all nine languages. These candidates are grouped according to their relationship to the translat… view at source ↗
Figure 15
Figure 15. Figure 15: Evolution of layer-wise log probabilities during training, grouped by language. Aggregated across all language pairs, we visualize the progression of log probability across model layers, grouped by their relationship to the translation task. In the bottom and intermediate layers, all languages are close to each other. From layer 15 onward, the source word and, later in training, also the correct translati… view at source ↗
Figure 16
Figure 16. Figure 16: Evolution of layer-wise log probabilities during training, grouped by token overlap. We analyze the translation task and group the highest-ranking target-language candidate according to its token overlap with the source word. During training, all groups show increasing log probability with layers depths. Around layer 14, the groups begin to separate: candidates with higher token overlap on average receive… view at source ↗
Figure 17
Figure 17. Figure 17: Exciting and inhibiting copy behavior in bottom layers. We scale the parameters of the copy promoting parameter groups as identified by ExPLAIND in Phase I and detailed in Section 6.2. The corresponding parameter groups are the self attention value projections in layer 0, 1 and 2. To investigate their effect on the copy behavior of the model, we perform a naive ablation by doubling and halving their param… view at source ↗
Figure 18
Figure 18. Figure 18: Exciting and inhibiting copy behavior in upper bottom block. We scale the parameters suppressing copy as identified by ExPLAIND in the upper bottom block in Phase II and detailed in Section 6.2. The corresponding parameter groups are the attention key and query projections in layer 9, the attention key and value projection in layer 8, and the MLP down projections of layer 6 and 7, i.e. a total of six para… view at source ↗
Figure 19
Figure 19. Figure 19: For each of the layer swapping experiments shown in Fig. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Parameter swapping over time. We swap different parameter blocks of the final checkpoint into earlier checkpoints and compare WLT accuracy of the resulting model to the (unchanged) original checkpoint. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_20.png] view at source ↗
read the original abstract

Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We use word-level translation as a testbed, introducing a novel dataset to trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 5 minor

Summary. The paper presents SLO-Guard, a crash-aware two-phase autotuner for vLLM serving that combines a Thermal Budget Annealing (TBA) exploration phase with a warm-started TPE exploitation phase. The central empirical claim is that SLO-Guard allocates a fixed 15-trial tuning budget more consistently than uniform random search—more fast-regime trials (10.20 vs 7.40/15, one-sided p=0.014), higher post-hit consistency (0.876 vs 0.539, p=0.010), and 4.4× tighter cross-seed best-latency variance under concurrent load—while being statistically tied on best-achieved latency (p=0.84). The study uses a pre-specified five-seed protocol with Mann-Whitney tests and Holm-Bonferroni correction, and replicates across sequential and concurrent harness conditions.

Significance. The empirical methodology is a genuine strength: pre-specified seeds, non-parametric tests with multiple-comparison correction, explicit effect-size reporting, honest non-claiming of best-latency superiority, and a harness-replication analysis that corrects a measurement defect. The public code repository with raw JSONL logs and commit hashes is a real reproducibility contribution. However, a critical submission error (abstract/title mismatch) and a framing-evidence gap substantially undermine the manuscript as it stands.

major comments (4)
  1. Abstract/title mismatch. The submitted abstract and title ('Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining') describe a completely different paper about multilingual pretraining. The full text is about SLO-Guard for LLM serving autotuning. This is a fundamental submission error that must be corrected before content assessment can proceed.
  2. Crash-awareness framing is unsupported by the experimental evidence. The paper's title, abstract, and §1 frame SLO-Guard as 'crash-aware,' and three of four contributions reference crash-awareness. However, zero crashes occurred across all 300 trials (Table 1, Table 2). The three crash-aware mechanisms—the bad-region tracker (Algorithm 1, line 12), crash encoding as extreme constraint violations (§3.4), and the crash taxonomy (§3.5)—are never empirically exercised. The observed consistency advantage is more parsimoniously explained by the TBA→TPE handoff structure: TBA explores for 6 trials, then hands off to warm-started TPE at trial 7 (§3.4, confirmed in all runs), and TPE with feasible-region history concentrates sampling in the fast regime—a standard property of TPE density-ratio optimization. The ablations that would disentangle crash-awareness from the handoff structure—TBA-only vs
  3. TBA-TPE (§5.7, item 1), cold-start TPE vs warm-started TPE (item 2), and repair map on/off (item 3)—are listed but explicitly not run. Without at least one of these ablations, the causal attribution to crash-awareness is unsupported. The paper should either run a minimal ablation or reframe the contribution around two-phase warm-started search rather than crash-awareness.
  4. Single-knob landscape limits generalizability. §5.5 shows that enforce_eager (a single binary knob) explains >95% of latency variance, making the problem essentially binary regime discovery. The §5.7 item 4 ablation (fix enforce_eager) would test whether the consistency advantage persists as a within-cluster fine-tuning effect; its absence leaves the practical scope of the contribution unclear.
minor comments (5)
  1. Reference [12] (the author's own TBA preprint) is cited as 'arXiv preprint pending; update citation details after public release.' A load-bearing methodological dependency on an unavailable reference is problematic for review; the key TBA details should be self-contained in the present manuscript.
  2. §3.4, Algorithm 2: the handoff parameters t_min, t_max, n_min_f, n_min_b are listed as free parameters but their values for the 15-trial budget are not explicitly stated (only the empirical result that handoff occurs at trial 7).
  3. §5.4: the sequential-harness best-latency variance ratio is 0.84× (Table 2), meaning random search was slightly tighter under sequential dispatch. This is mentioned but not discussed; a sentence explaining why would help.
  4. Table 3: seed 242 for SLO-Guard shows first fast trial at trial 3, not trial 7 as the §5.6 narrative suggests. The representative trajectory (Figure 6) should be clarified as seed-specific rather than general.
  5. References [1] and [21] have 'Verify final bibliographic details before submission' placeholders.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and substantive review. The referee correctly identifies a submission error (abstract/title mismatch) and raises a serious framing concern about the crash-awareness label. We address each point below.

read point-by-point responses

  1. Referee: Abstract/title mismatch: the submitted abstract and title describe a completely different paper about multilingual pretraining, while the full text is about SLO-Guard for LLM serving autotuning.

    Authors: The referee is correct. This is a submission error: the wrong abstract and title were uploaded. The full text, keywords, and all content are the SLO-Guard paper. We will replace the abstract and title with the correct ones (already present in the full text) in the next version. We apologize for the error. revision: yes

  2. Referee: Crash-awareness framing is unsupported by the experimental evidence: zero crashes occurred across all 300 trials, the three crash-aware mechanisms are never empirically exercised, and the consistency advantage is more parsimoniously explained by the TBA-to-TPE handoff structure. The ablations that would disentangle crash-awareness from the handoff structure are listed but not run.

    Authors: The referee raises a fair and important point. We concede that the crash-awareness framing is not supported by the current experimental evidence: with zero crashes across 300 trials, the crash-specific mechanisms (bad-region tracker, crash encoding, crash taxonomy) are never exercised, and the observed consistency advantage is more parsimoniously explained by the TBA-to-TPE warm-start handoff. We will reframe the contribution around two-phase warm-started search rather than crash-awareness. Specifically: (1) the title and abstract will be revised to remove crash-awareness as the central framing; (2) the crash-aware mechanisms will be repositioned as design features for crash-prone settings rather than empirically validated contributions; (3) the contributions list will be rewritten to center on the two-phase warm-started search structure; (4) we will run at least one of the listed ablations—specifically, cold-start TPE vs warm-started TPE (item 2)—to provide causal evidence for the warm-start handoff as the source of the consistency advantage. We agree that without an ablation, attribution to crash-awareness is unsupported, and we will not make that attribution in the revision. revision: yes

  3. Referee: Single-knob landscape limits generalizability: enforce_eager explains over 95% of latency variance, making the problem essentially binary regime discovery. The ablation fixing enforce_eager (item 4) is absent, leaving the practical scope unclear.

    Authors: The referee is correct that the single-knob structure limits generalizability and that the enforce_eager-fixed ablation would clarify whether the consistency advantage persists as a within-cluster fine-tuning effect. We acknowledge this limitation and will add it explicitly to the threats-to-validity section. We will also run the enforce_eager-fixed ablation (item 4) in the revision. If the consistency advantage collapses under this ablation, we will state this plainly and scope the contribution to regime-discovery consistency on bimodal landscapes. We agree that the current scope claim is incomplete without this test. revision: yes

Circularity Check

0 steps flagged

No significant circularity: the central empirical claim is tested against an external baseline with pre-specified seeds and non-parametric statistics.

full rationale

The paper's central claim—that SLO-Guard allocates a fixed 15-trial budget more consistently than uniform random search—is evaluated against an external baseline (uniform random search) using pre-specified seeds, non-parametric Mann-Whitney U tests, and Holm-Bonferroni correction. The metrics (fast-cluster trials, post-hit consistency, best latency) are defined independently of the method's internal parameters. The one self-citation to the author's pending Thermal Budget Annealing preprint [12] extends TBA-explore to the LLM setting, but the empirical results do not depend on claims from that preprint: the TBA-explore algorithm is fully specified in Algorithm 1, the handoff condition in Algorithm 2, and the TPE exploitation in Section 3.4. The crash taxonomy, repair map, and GPU-aware memory guard are new contributions defined within this paper. No 'prediction' or 'first-principles result' reduces to a fitted input by construction. The paper explicitly acknowledges that zero crashes occurred and that the crash-aware mechanisms were not exercised, and it lists but does not run the ablations that would disentangle crash-awareness from the TBA→TPE handoff structure—these are correctness/attribution concerns, not circularity. The derivation chain from problem formulation (Section 3.1) through method (Sections 3.2–3.5) to experimental evaluation (Section 5) is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 2 invented entities

The paper introduces several engineering components (repair map, GPU guard, crash taxonomy, bad-region tracker) whose crash-related functionality is never exercised because the repair map eliminated all crashes. The consistency advantage may be entirely due to the TBA→TPE handoff structure.

free parameters (5)
  • SLO thresholds (TTFT p99, ITL p99, memory) = 500ms, 100ms, auto-detected
    Set by the authors based on MLPerf conventions; not fitted to data but chosen for the experiment.
  • Fast-cluster latency threshold = 1000 ms
    Chosen to separate two visible latency bands in Figure 5; acknowledged as structural rather than optimized.
  • TBA temperature schedule = Not specified in detail
    Annealing schedule parameters are referenced but not fully specified in the paper text.
  • Handoff parameters (t_min, t_max, n_min_f, n_min_b) = t_min=3, t_max=6 for B=15
    Set by formula in Algorithm 2; n_min_f and n_min_b values not explicitly stated.
  • Safety margin alpha = Not specified
    The KV-cache safety margin in Equation 4 is introduced but its value is not stated.
axioms (3)
  • domain assumption Crash-aware search improves budget consistency when the feasible set is ragged and regime switches exist
    Section 6 states this as the testable answer to when crash-awareness helps, but it is assumed rather than proven since no crashes were observed.
  • domain assumption Tail latencies over short benchmark runs generalize to sustained serving behavior
    Section 7 (Threats to Validity) explicitly flags this as a construct validity concern.
  • ad hoc to paper The TBA exploration method from [12] is sound for constrained optimization
    The paper extends the author's own pending preprint [12] as the exploration phase, but that work is not yet publicly available for independent verification.
invented entities (2)
  • Crash taxonomy (4 categories: Healthy, Startup failure, Preflight failure, Runtime failure) no independent evidence
    purpose: Classify trial outcomes to condition the bad-region tracker
    The taxonomy is a useful engineering contribution but is never empirically exercised because zero crashes occurred. Its value for crash-aware search is asserted, not demonstrated.
  • Bad-region tracker (per-category) no independent evidence
    purpose: Discourage repeated sampling of structurally-similar crashed configurations
    No crashes occurred, so the tracker's influence on search behavior is untested.

pith-pipeline@v1.1.0-glm · 14811 in / 3858 out tokens · 295375 ms · 2026-07-05T16:22:52.010121+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · 3 internal anchors

  1. [1]

    Sarathi-serve: Efficient LLM inference by piggybacking decodes with chunked prefills.arXiv preprint, 2024

    Amey Agrawal et al. Sarathi-serve: Efficient LLM inference by piggybacking decodes with chunked prefills.arXiv preprint, 2024. Verify final bibliographic details before submission

    work page 2024

  2. [2]

    Optuna: A next-generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2623–2631, 2019

    work page 2019

  3. [3]

    Amazon SageMaker automatic model tuning

    Amazon Web Services. Amazon SageMaker automatic model tuning. https://docs. aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning.html, 2025. Doc- umentation page

    work page 2025

  4. [4]

    Algorithms for hyper-parameter optimization

    James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. InAdvances in Neural Information Processing Systems 24, pages 2546–2554, 2011

    work page 2011

  5. [5]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems 35, pages 16344–16359, 2022

    work page 2022

  6. [6]

    Bayesian Optimization with Unknown Constraints

    Michael A. Gelbart, Jasper Snoek, and Ryan P . Adams. Bayesian optimization with unknown constraints.arXiv preprint arXiv:1403.5607, 2014. 17

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  7. [7]

    Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, David Kochanski, John Karro, and D. Sculley. Google vizier: A service for black-box optimization. InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1487–1495, 2017

    work page 2017

  8. [8]

    Hoos, and Kevin Leyton-Brown

    Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. InProceedings of the 5th International Conference on Learning and Intelligent Optimization, pages 507–523, 2011

    work page 2011

  9. [9]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woojin Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Jinyang Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023

    work page 2023

  10. [10]

    Constrained bayesian optimization with noisy experiments.Bayesian Analysis, 14(2):495–519, 2019

    Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. Constrained bayesian optimization with noisy experiments.Bayesian Analysis, 14(2):495–519, 2019

    work page 2019

  11. [11]

    SLO-Guard: Crash-aware autotuning for llm serving, 2026

    Christian Lysen. SLO-Guard: Crash-aware autotuning for llm serving, 2026. URL https://github.com/Chrislysen/SLO-Guard

    work page 2026

  12. [12]

    Thermal budget annealing: Feasible-first exploration for con- strained ml deployment, 2026

    Christian Lysenstøen. Thermal budget annealing: Feasible-first exploration for con- strained ml deployment, 2026. arXiv preprint pending; update citation details after public release

    work page 2026

  13. [13]

    GenAI-Perf: Generative AI performance measurement for Triton and OpenAI-compatible apis

    NVIDIA. GenAI-Perf: Generative AI performance measurement for Triton and OpenAI-compatible apis. https://docs.nvidia.com/ deeplearning/triton-inference-server/user-guide/docs/perf _benchmark/ genai-perf-README.html, 2025. Documentation page

    work page 2025

  14. [14]

    MLPerf Inference Benchmark

    Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Max Breughe, Maximilien Charlebois, William Chou, et al. MLPerf inference benchmark.arXiv preprint arXiv:1911.02549, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  15. [15]

    Gonzalez, Cho-Jui Hsieh, and Ion Stoica

    Ying Sheng, Lianmin Zheng, Binhang Yu, Siyuan Zhuang, Zhuohan Li, Max Ryabinin, Alex Touzalin, Joseph E. Gonzalez, Cho-Jui Hsieh, and Ion Stoica. Flexgen: High- throughput generative inference of large language models with a single GPU or CPU. InProceedings of the 40th International Conference on Machine Learning, pages 31094–31116, 2023

    work page 2023

  16. [16]

    Local Bayesian Optimization for Controller Tuning with Crash Constraints

    David Stenger, Dominik Scheurenberg, and Sebastian Trimpe. Local bayesian opti- mization for controller tuning with crash constraints.arXiv preprint arXiv:2411.16267, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    Morphling: Fast, near-optimal auto-configuration for cloud-native model serving

    Luping Wang, Lingyun Yang, Yinghao Yu, Wei Wang, Bo Li, Xianchao Sun, Jian He, and Liping Zhang. Morphling: Fast, near-optimal auto-configuration for cloud-native model serving. InProceedings of the 12th ACM Symposium on Cloud Computing, pages 639–653, 2021

    work page 2021

  18. [18]

    Revisiting SLO and goodput metrics in LLM serving.arXiv preprint arXiv:2410.14257, 2024

    Zhibin Wang, Shipeng Li, Yuhang Zhou, Xue Li, Rong Gu, Cam-Tu Nguyen, Chen Tian, and Sheng Zhong. Revisiting SLO and goodput metrics in LLM serving.arXiv preprint arXiv:2410.14257, 2024. 18

    work page arXiv 2024

  19. [19]

    Orca: A distributed serving system for transformer-based generative models

    Gyeong-In Yu, Jeongmin Jeong, Gunho Kim, Soojeong Shin, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation, pages 521–538, 2022

    work page 2022

  20. [20]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Zhen Jia, Minmin Sun, Sheng Wu, Juncen Yu, Ameer Haj-Ali, Yida Wang, Jie Yang, Danyang Zhuo, Kaushik Sen, Joseph E. Gonzalez, and Ion Stoica. Alpa: Automating inter- and intra-operator parallelism for distributed deep learning. In16th USENIX Symposium on Operating Systems Design and Implementation, pages 559–578, 2022

    work page 2022

  21. [21]

    Distserve: Disaggregating prefill and decoding for goodput- optimized large language model serving.arXiv preprint, 2024

    Yongkang Zhong et al. Distserve: Disaggregating prefill and decoding for goodput- optimized large language model serving.arXiv preprint, 2024. Verify final bibliographic details before submission. A Per-seed summaries Tables 3 and 4 expand the aggregate statistics to the per-seed level. With only five seeds, the reader should be able to see every point th...

    work page 2024