Pith. sign in

REVIEW 4 major objections 6 minor 46 references

LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes a lightweight extension of the WAIT LLM scheduler, estimating arrival rates online from interarrival times and adapting batching thresholds to bursty traffic, and reports higher throughput than Sarathi-Serve, ORCA, and…

desk verdict The adaptive threshold rule inverts Eq. (1), so the paper's central mechanism cannot explain the reported gains—desk reject. read the letter →

arxiv 2608.06135 v1 pith:7X36HFZG submitted 2026-08-06 cs.LG

Anjali Gangadhar Katageria , Shobha Rani , Raghu Nandan Sengupta This is my paper
classification cs.LG
keywords LLMinferenceschedulingWAITalgorithmarrivalrateestimationburstyworkloadMarkovModulatedPoissonProcessthroughputlatency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the WAIT batching thresholds, designed for known constant Poisson arrival rates, lose their edge when LLM request traffic is bursty and time-varying, and that a lightweight online rate estimator can recover most of that edge without seeing the future. To test this, it replaces WAIT's fixed threshold with an adaptive threshold updated as $n_j(t) \propto 1/\hat\lambda_j(t)$, where $\hat\lambda_j(t)$ is an exponentially smoothed estimate of the current arrival rate for prompt type $j$. In simulations with two-state Markov-modulated Poisson arrivals, the modified scheduler matches the full-knowledge WAIT baseline and reports higher throughput than Sarathi-Serve, ORCA, and vLLM in low-shift burst scenarios, with comparable latency. If the results hold, latency-sensitive LLM serving can adapt to real bursts using only observed arrival timestamps.

What carries the argument

The mechanism is the adaptive threshold update $n_j(t) \propto 1/\hat\lambda_j(t)$ applied to WAIT's inventory-triggered batching: a prompt type $j$ is batched only when its waiting inventory reaches $n_j(t)$, and the threshold now moves with a smoothed estimate of the observed arrival rate, obtained by averaging recent interarrival times, feeding an exponential moving average, and then applying a Savitzky–Golay filter. Each request type keeps its own estimator and threshold, so types with different prompt lengths and decode lengths adapt independently. The paper presents this as a feedback loop that shortens waiting during traffic spikes and lengthens it during lulls, keeping the batch size near the memory-constrained optimum.

What would settle it

Run the same MMPP-2 burst traces through two variants of the modified scheduler: one with $n_j(t) \propto 1/\hat\lambda_j(t)$ and one with $n_j(t) \propto \hat\lambda_j(t)$, plus a WAIT whose thresholds are reset to the true per-state rates at each state change. If the inverse rule does not beat the proportional rule, or if true-state WAIT clearly beats both, the central claim that threshold updates should shrink with rising arrivals is refuted.

Watch

Extended reading notes

Core claim

The central claim is that WAIT's threshold rule can be made workload-aware online. The paper argues that when the arrival rate $\lambda_j$ rises, the batching threshold $n_j$ should fall (since in its reading of the WAIT optimality condition the threshold is inversely related to the rate), so batches form sooner during surges and wait longer during lulls; an exponential moving average over interarrival times, refined by a Savitzky–Golay filter, supplies the rate estimate. Running this modified WAIT in a simulation of an NVIDIA A100 GPU, the paper reports it achieves throughput close to the original WAIT that knows the full trace, and higher throughput than Sarathi-Serve, ORCA, and vLLM in the evaluated low-shift scenarios, while keeping latency comparable. The authors conclude that an online estimator is enough to adapt threshold-based scheduling to bursty, non-stationary workloads.

Load-bearing premise

The paper's adaptation rests on reading WAIT's optimality equation as saying the batching threshold $n_j$ is inversely related to arrival rate $\lambda_j$; if that reading is wrong, the update rule $n_j(t) \propto 1/\hat\lambda_j(t)$ is not a valid WAIT instantiation and the throughput gains are not explained by the stated mechanism.

Editorial extensions

If this is right

  • Production schedulers could operate without pre-observed arrival traces and still track the performance of a scheduler that knows the full distribution.
  • The threshold-based WAIT policy remains the right skeleton; only the threshold-setting rule needs to be online, lowering the barrier to deployment.
  • Under high-shift burst conditions both WAIT variants underperform, pointing to a need for a faster or state-aware rate estimator.
  • Per-type adaptive thresholds let heterogeneous prompt classes batch at different cadences, which is directly relevant to multi-tenant serving.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A useful next test is to compare $n_j(t) \propto 1/\hat\lambda_j(t)$ against $n_j(t) \propto \hat\lambda_j(t)$ on identical traces; that isolates whether the throughput gain comes from the inverse relation or from the act of adapting thresholds at all.
  • The same per-type online rate estimator could be grafted onto other threshold-based serving policies, for example the token and prompt limits used in Sarathi-Serve or vLLM, giving them burst resilience without replacing their core scheduler.
  • Because the evaluation uses synthetic MMPP-2 traces, the most direct follow-up is to rerun the same scheduler on recorded production request traces to see whether the low-shift advantage persists when the burst structure is not generated by the assumed model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a modification to the WAIT LLM scheduling algorithm, called Modified WAIT, which uses an online arrival-rate estimator to adapt batching thresholds under bursty, non-stationary workloads. The workloads are modeled as two-state Markov-Modulated Poisson Processes (MMPP-2). The authors evaluate Modified WAIT against WAIT, Sarathi, ORCA, and vLLM using the Vidur simulator, reporting throughput and latency under four scenarios (Low/High Demand with Low/High Shift). The abstract claims higher throughput than Sarathi-Serve, ORCA, and vLLM in low arrival-rate shift scenarios while maintaining comparable latency, and the conclusion emphasizes competitiveness with ideal WAIT and outperformance of Sarathi and vLLM under Low Shift conditions.

Significance. If the central mechanism were sound, the paper would offer a lightweight, practical extension to a theoretically grounded scheduler, with a reproducible simulation framework and open-source code—all valuable for the LLM-systems community. The use of MMPP-2 to model burstiness is appropriate, and the comparison against strong baselines (Sarathi, ORCA, vLLM) in a public simulator is a positive feature. However, the core adaptive-threshold rule is algebraically inconsistent with the WAIT optimality condition, and the empirical claims lack statistical support, so the significance of the contribution, as stated, cannot be accepted.

major comments (4)
  1. [§5.2.1, §5.3, Eq. (1)] The adaptive rule n_j(t) ∝ 1/λhat_t contradicts the WAIT optimality condition in Eq. (1). Eq. (1) requires ΔT(n_1:m) ≤ n_j/λ_j for all j, which implies n_j ≥ λ_j · ΔT(n_1:m) (up to the coupling through M^π). For a fixed target processing time ΔT, the threshold n_j must therefore grow with λ_j, not shrink. The statement in §5.2.1 that 'the batching threshold n_j is inversely related to the mean arrival rate λ_j' is algebraically backwards. Consequently, the modified scheduler that sets n_j(t) ∝ 1/λhat_t is not a valid WAIT instantiation: it violates the optimality condition precisely in high-rate states, making thresholds smallest during bursts and largest in lulls. The throughput and latency results in §6 therefore cannot be attributed to the stated adaptive-threshold mechanism.
  2. [Abstract and §6] The abstract claims that the proposed method 'achieves higher throughput than Sarathi-Serve, ORCA, and vLLM in the evaluated low arrival-rate shift scenarios', but the numerical-experiments summary in §6 only claims outperformance of 'Sarathi and vLLM' under Low Shift scenarios, and the Conclusion (Section 7) similarly lists only Sarathi and vLLM. ORCA is neither mentioned in the results summary nor shown to be outperformed. The abstract overstates the evidence presented in the paper.
  3. [§6] All throughput and latency comparisons are reported as point values without confidence intervals, standard errors, or significance tests. Since the traces are generated from stochastic MMPP-2 processes, single-run or mean-only comparisons do not establish that Modified WAIT's apparent advantages over baselines are statistically meaningful. The authors should report results across multiple independent trace seeds and perform appropriate significance tests (e.g., paired tests or confidence-interval comparisons).
  4. [§5.2.1 and §6] The estimator hyperparameters (EMA α, Savitzky–Golay window length and polynomial order, moving-average window) are selected via sensitivity analysis on MMPP-2 traces with state rates (50,100), (75,175), and (100,250), and the chosen configuration is then used for evaluation on the same MMPP-2 model family (with different rate values, e.g., Table 3 and Table 4). Because the tuning and evaluation distributions share the same parametric family, the reported performance may reflect overfitting to MMPP-2 assumptions rather than genuine robustness. The paper should demonstrate generalization, for example by evaluating on a held-out real-world trace (e.g., BurstGPT) or on a different arrival model.
minor comments (6)
  1. [§5.2.1] Typo: 'appying' should be 'applying'.
  2. [§5.3] The notation n_j(t) is used without a formal definition; please define it as the time-dependent threshold for request type j.
  3. [§6] The sentence 'In both figures, the red line denotes the performance of the WAIT algorithm' is ambiguous because Figures 6 and 7 each contain multiple subfigures; specify which subfigure corresponds to which scenario.
  4. [§6.1 and §6.2] The text says Modified WAIT 'does not outperform all four baselines' but lists only three baselines in the evaluation; clarify whether ORCA is included and what its relative performance is.
  5. [§5.1, Algorithm 2] Algorithm 2 generates uniform timestamps within each state interval, which does not produce true Poisson arrivals within the interval; this is a discrepancy between the stated MMPP model and the implemented trace generator that should be explained or corrected.
  6. [References] References [12] and [18] duplicate the same BurstGPT paper, and [13] and [19] duplicate the same ELIS paper; consolidate these to avoid redundancy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: throughput claims rest on external simulation benchmarks; the adaptive threshold rule is an asserted control law, not a fitted prediction.

full rationale

The paper's central claim is an empirical simulation result: Modified WAIT is compared in Section 6, via Microsoft Vidur, against external baselines (Sarathi-Serve, ORCA, vLLM, and original WAIT). No parameter is fitted to the reported throughput/latency outcomes and then renamed as a prediction. The Section 5.2.1 sensitivity analysis tunes the EMA/Savitzky-Golay preprocessing hyperparameters to minimize MAE/MAPE of the arrival-rate estimator against ground-truth MMPP-2 rates on traces with rates (50,100), (75,175), and (100,250); the evaluation scenarios use different parameters (e.g., low demand: state rates 1000/1000 and 500/200; high demand: 5000/4000/2000 and 4000/3000/1000), so the tuning does not by construction determine the benchmark outcomes. All load-bearing references (WAIT [1], Sarathi [2], ORCA [3], vLLM [4], Vidur [46]) are external prior work; the paper contains no self-citation chain or imported uniqueness theorem. The threshold rule n_j(t) proportional to 1/lambda-hat_t in Section 5.2.1 is asserted as a design choice; whether it is a valid reading of Eq. (1) is a correctness concern (Eq. (1) actually gives n_j >= lambda_j * DeltaT, so the proportionality direction appears reversed), but it is not circular because the rule is an input to the simulator, not a quantity derived from the measured throughput. Section 7's deferred real-world validation is a stated limitation, not evidence of circularity. The abstract's ORCA throughput claim is not substantiated by the Section 6 text, but an unsupported claim is not a circular derivation. No equation in the paper reduces to another by construction, and no fitted value is relabeled as a prediction; score 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the MMPP-2 workload model, the WAIT optimality condition from prior work, and an unproved inverse relationship between threshold and arrival rate. The estimator hyperparameters are fitted to synthetic traces. No new physical or mathematical entities are introduced.

free parameters (5)
  • EMA smoothing factor α = 0.05
    Chosen by sensitivity analysis in Section 5.2.1 to minimize average MAE/MAPE across three MMPP-2 scenarios; this tuning is used for all scheduling experiments.
  • Savitzky-Golay window length = 151
    Selected in the same sensitivity analysis; affects smoothness of the rate estimate.
  • Savitzky-Golay polynomial order = 2
    Selected in the same sensitivity analysis.
  • Moving-average window for global rate = 5
    Selected in the same sensitivity analysis.
  • Threshold scaling/proportionality constant for n_j(t) = unspecified
    The paper sets n_j(t) ∝ 1/λhat_j(t) but does not specify the absolute level or how it relates to Eq. (1), so the batching thresholds are not fully determined by the text.
assumptions (6)
  • standard math Interarrival times from a Poisson process are exponentially distributed; MMPP follows standard definitions.
    Used throughout Sections 2.3, 2.4, and Algorithm 2.
  • domain assumption Real-world LLM request traffic is adequately represented by a two-state Markov Modulated Poisson Process (MMPP-2).
    Section 2.4 and Table 1 motivate MMPP-2 via analogies to teletraffic, web, IoT, and two LLM trace papers; the evaluation uses only MMPP-2 synthetic traces, so external validity depends on this assumption.
  • domain assumption WAIT thresholds must satisfy ΔT(n) ≤ n_j/λ_j for all j (Eq. 1).
    Section 4.2 invokes Eq. (1) as the optimality condition for WAIT; the modification is supposed to preserve this framework.
  • ad hoc to paper The batching threshold n_j is inversely related to arrival rate λ_j, so n_j(t) ∝ 1/λhat_j(t).
    Section 5.2.1 states this relation, but it contradicts Eq. (1), where n_j/λ_j appears, implying n_j should scale with λ_j. This unproved assumption is the core of the modification.
  • domain assumption The batch size limit is the same across all compared algorithms, ensuring fairness.
    Section 6 states 'we enforce the same batch size limit across all algorithms' but the limit value is not given.
  • domain assumption Vidur simulation of an NVIDIA A100 GPU faithfully reproduces real LLM serving performance.
    Section 6 uses Microsoft Vidur [46] and cites prior studies [47]; no validation against real hardware is reported in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm." pith.science (2026). https://pith.science/paper/7X36HFZG

@misc{pith2026260806135,
  author       = {Pith},
  title        = {Pith review of: LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7X36HFZG}},
  note         = {Machine review of arXiv:2608.06135}
}
read the original abstract

Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving. Recent work has focused on improving scheduling algorithms to boost throughput while maintaining low latency. However, these approaches often assume Poisson request arrivals with constant rates - an assumption that fails to reflect the inherently bursty and dynamic nature of real-world traffic. We propose a lightweight extension to the state-of-the-art WAIT algorithm [1], which adapts to time-varying arrival rates without prior traffic knowledge. The proposed algorithm performs online estimation of request intensity based on observed interarrival times. Using Markov Modulated Poisson Process (MMPP)-based synthetic workloads with diverse request types, we conduct a simulation-based evaluation demonstrating that the proposed method achieves higher throughput than Sarathi-Serve [2], ORCA [3], and vLLM [4] in the evaluated low arrival-rate shift scenarios while maintaining comparable latency.

Figures

Figures reproduced from arXiv: 2608.06135 by the authors.

Figure 1
Figure 1. Flowchart representing prefill and decode phases of LLM inference [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Request Arrival: Fixed rate Poisson vs. MMPP (2-State) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Preprocessing of Request Arrival Trace 18 [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Pipeline for Estimating Arrival Rates and Updating Thresholds [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Pipeline of the Modified WAIT Algorithm. The grey action blocks represent our [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Average throughput and latency on datasets with Low Demand [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Average throughput and latency on datasets with High Demand [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 30 canonical work pages

  1. [1]

    R. Ao, G. Luo, D. Simchi-Levi, X. Wang, Optimizing llm inference: Fluid-guided online scheduling with memory constraints, arXiv preprint arXiv:2504.11320 (2025)

    arXiv 2025

  2. [2]

    Agrawal, A

    A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, R. Ram- jee, Sarathi: Efficient llm inference by piggybacking decodes with chun- ked prefills, arXiv preprint arXiv:2308.16369 (2023)

    arXiv 2023

  3. [3]

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, B.-G. Chun, Orca: A dis- tributed serving system for{Transformer-Based}generative models, in: Proc. 16th USENIX Symp. Oper. Syst. Des. Implement. (OSDI ’22), 2022, pp. 521–538

    work page 2022

  4. [4]

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, Efficient memory management for large language model serving with pagedattention, in: Proc. ACM SIGOPS Symp. Oper. Syst. Principles, 2023

    work page 2023

  5. [5]

    L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, A. Krishnamurthy, Punica: Multi-tenant lora serving, Proc. Mach. Learn. Syst. 6 (2024) 1–13. 26

    work page 2024

  6. [6]

    Zhong, S

    Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, H. Zhang, Dist- serve: Disaggregating prefill and decoding for goodput-optimized large language model serving, in: 18th USENIX Symp. Oper. Syst. Design Implement. (OSDI ’24), 2024, pp. 193–210

    work page 2024

  7. [7]

    S. Shen, K. Tilwani, Z. Tan, T. Dao, Z. Liu, S. Kaur, E. Wallace, T. Jun, C. Zhou, A. Wan, Y. Tian, I. Stoica, J. E. Gonzalez, vllm: Easy, fast, and cheap llm serving with open-source infrastructure, arXiv preprint arXiv:2309.06180 (2023)

    arXiv 2023

  8. [8]

    S. Shen, S. Kaur, Z. Tan, T. Dao, J. E. Gonzalez, I. Stoica, Faster and cheaper llm serving with kv cache optimization, arXiv preprint arXiv:2306.07179 (2023)

    arXiv 2023

Show all 46 references
  1. [9]

    Zheng, H

    S. Zheng, H. Wu, J. Chen, Z. Yao, M. W. Mahoney, K. Keutzer, Smoothquant: Accurateandefficientpost-trainingquantizationforlarge language models, arXiv preprint arXiv:2211.10438 (2024)

    2024 arXiv

  2. [10]

    Rajbhandari, J

    S. Rajbhandari, J. Rasley, O. Ruwase, Y. He, Deepspeed-mii: Multi- turninferenceoflargelanguagemodelsmadeeasyandefficient, in: Proc. ACM Symp. Cloud Comput. (SoCC), 2022

    2022

  3. [11]

    R. Anil, X. Garcia, Y. Tay, M. Dehghani, S. Gehrmann, N. Constant, et al., Palm 2 technical report, arXiv preprint arXiv:2305.10403 (2023)

    2023 arXiv

  4. [12]

    Wang, et al., BurstGPT: A real-world workload dataset to optimize llm serving systems, in: Proc

    Y. Wang, et al., BurstGPT: A real-world workload dataset to optimize llm serving systems, in: Proc. 31st ACM SIGKDD Conf. Knowl. Dis- covery Data Mining (KDD), vol. 2, ACM, Toronto, ON, Canada, 2025. doi:https://doi.org/10.1145/3711896.3737413

    2025

  5. [14]

    F. Dong, K. Wu, V. Srinivasan, Copula analysis of temporal depen- dence structure in markov modulated poisson process and its applica- tions, ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS) 2 (3) (2017) 1–28. 27

    2017

  6. [15]

    Casale, N

    G. Casale, N. Zhang, E. Smirni, Dealing with burstiness in multi-tier ap- plications, ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS) 1 (1) (2016) 1–31

    2016

  7. [16]

    El-Fawal, et al., Markov modulated poisson process modeling for iot traffic, IEEE Internet of Things Journal (2024)

    A. El-Fawal, et al., Markov modulated poisson process modeling for iot traffic, IEEE Internet of Things Journal (2024)

    2024

  8. [17]

    Yoshihara, S

    K. Yoshihara, S. Kasahara, Y. Takahashi, A practical time-scale fitting method for traffic with long-range dependence, Performance Evaluation 43 (3) (2001) 169–186

    2001

  9. [18]

    Y. Wang, Y. Chen, Z. Li, X. Kang, Y. Fang, Y. Zhou, Y. Zheng, Z. Tang, X. He, R. Guo, et al., Burstgpt: A real-world workload dataset to op- timize llm serving systems, in: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, 2025, pp. 5831–5841

    2025

  10. [19]

    S. Choi, J. Goo, E. Jeon, M. Yang, M. Jang, Elis: Efficient llm iter- ative scheduling system with response length predictor, arXiv preprint arXiv:2505.09142 (2025)

    2025 arXiv

  11. [20]

    Y. Tay, M. Dehghani, D. Bahri, D. Metzler, Efficient transformers: A survey, ACM Comput. Surv. 55 (6) (2022) 1–28

    2022

  12. [21]

    H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, T. Zhao, Gear: An efficient kv cache compression recipefor near-lossless genera- tive inference of llm, arXiv e-prints (2024) arXiv–2403

    2024

  13. [22]

    Hooper, S

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, A. Gholami, Kvquant: Towards 10 million context length llm inference with kv cache quantization, Adv. Neural Inf. Process. Syst. 37 (2024) 1270–1303

    2024

  14. [23]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Adv. Neural Inf. Process. Syst. 33 (2020) 1877– 1901

    2020

  15. [24]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., Palm: Scaling 28 language modeling with pathways, J. Mach. Learn. Res. 24 (240) (2023) 1–113

    2023

  16. [25]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  17. [26]

    OpenAI Achiam, S

    J. OpenAI Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report. arxiv, arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  18. [27]

    A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, et al., Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, arXiv preprint arXiv:2405.04434 (2024)

    2024 arXiv

  19. [28]

    Y. Fu, S. Zhu, R. Su, A. Qiao, I. Stoica, H. Zhang, Efficient llm schedul- ing by learning to rank, arXiv preprint arXiv:2408.15792 (2024)

    2024 arXiv

  20. [29]

    R. Cole, T. Roughgarden, The sample complexity of revenue maximiza- tion, in: Proc. 46th Annu. ACM Symp. Theory Comput. (STOC), 2014, pp. 243–252

    2014

  21. [30]

    Balkanski, A

    E. Balkanski, A. Rubinstein, Y. Singer, The power of optimization from samples, Adv. Neural Inf. Process. Syst. 29 (2016)

    2016

  22. [31]

    Lattanzi, T

    S. Lattanzi, T. Lavastida, B. Moseley, S. Vassilvitskii, Online scheduling via learned weights, in: Proc. 14th Annu. ACM–SIAM Symp. Discrete Algorithms (SODA), SIAM, 2020, pp. 1859–1877

    2020

  23. [32]

    W. Li, L. Wang, X. Chai, H. Yuan, Online batch scheduling of simple linear deteriorating jobs with incompatible families, Math. 8 (2) (2020) 170

    2020

  24. [33]

    N. Mi, Q. Zhang, A. Riska, E. Smirni, E. Riedel, Performance impacts of autocorrelated flows in multi-tiered systems, Perform. Eval. 64 (9-12) (2007) 1082–1101

    2007

  25. [34]

    Fischer, K

    W. Fischer, K. Meier-Hellstern, The markov-modulated poisson process (mmpp) cookbook, Perform. eval. 18 (2) (1993) 149–171. 29

    1993

  26. [35]

    Horváth, M

    A. Horváth, M. Telek, Markovian modeling of real data traffic: Heuristic phase type and map fitting of heavy tailed and fractal like samples, in: IFIP Int. Symp. Comput. Perform. Model. Meas. Eval., Springer, 2002, pp. 405–434

    2002

  27. [36]

    Okamura, T

    H. Okamura, T. Dohi, Faster maximum likelihood estimation algorithms for markovian arrival processes, in: 6th Int. Conf. Quantitative Eval. Syst., IEEE, 2009, pp. 73–82

    2009

  28. [37]

    Perez-Palacin, J

    D. Perez-Palacin, J. Merseguer, Performance sensitive self-adaptive service-oriented software using hidden markov models, in: Proc. 2nd ACM/SPEC Int. Conf. Perform. Eng., 2011, pp. 201–206

    2011

  29. [38]

    Perez-Palacin, J

    D. Perez-Palacin, J. Merseguer, R. Mirandola, Analysis of bursty workload-aware self-adaptive systems, in: Proc. 3rd ACM/SPEC Int. Conf. Perform. Eng., 2012, pp. 75–84

    2012

  30. [39]

    Mitzenmacher, R

    M. Mitzenmacher, R. Shahout, Queueing, predictions, and llms: Chal- lenges and open problems, arXiv preprint arXiv:2503.07545 (2025)

    2025 arXiv

  31. [40]

    Patel, A

    D. Patel, A. Ahmad, Peeling the onion’s layers-large language models search architecture and cost, Blog Post (2023)

    2023

  32. [41]

    Y. Li, J. Dai, T. Peng, Throughput-optimal scheduling algorithms for llm inference and ai agents (2025).arXiv:2504.07347

    2025 arXiv

  33. [42]

    Rydén, Parameter estimation for markov modulated poisson pro- cesses, Commun

    T. Rydén, Parameter estimation for markov modulated poisson pro- cesses, Commun. Stat. Stoch. Models 10 (4) (1994) 795–829.doi: 10.1080/15326349408807323

    1994 doi

  34. [43]

    Y. G. Yera, R. E. Lillo, P. Ramírez-Cobo, Fitting procedure for the two- state batch markov modulated poisson process (2024).arXiv:arXiv: 2401.14561,doi:10.1016/j.ejor.2019.04.018

    2024 arXiv

  35. [44]

    A. H. El Fawal, A. Mansour, A. Nasser, Markov-modulated poisson process modeling for machine-to-machine heterogeneous traffic, Appl. Sci. 14 (18) (2024).doi:10.3390/app14188561

    2024 doi

  36. [45]

    Yoshihara, S

    T. Yoshihara, S. Kasahara, Y. Takahashi, Practical time-scale fitting of self-similar traffic with markov-modulated poisson process, Telecom- mun. Syst. 17 (1) (2001) 185–211.doi:10.1023/A:1016616406118. 30

    2001 doi

  37. [46]

    Agrawal, N

    A. Agrawal, N. Kedia, J. Mohan, A. Panwar, N. Kwatra, B. Gulavani, R. Ramjee, A. Tumanov, Vidur: A large-scale simulation framework for llm inference (2024).arXiv:arXiv:2405.05465

    2024 arXiv

  38. [47]

    Agrawal, et al., Taming throughput-latency tradeoff in llm inference with sarathi-serve, in: Proc

    A. Agrawal, et al., Taming throughput-latency tradeoff in llm inference with sarathi-serve, in: Proc. 18th USENIX Symp. Oper. Syst. Des. Im- plement. (OSDI), Santa Clara, CA, USA, 2024, pp. 117–134. 31

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.