Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Past-Future Scheduler for LLM Serving under SLA Guarantees

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Admitting LLM requests based on projected peak future memory, forecast from historical output lengths, lets a serving system approach theoretical memory utilization while keeping SLA-satisfying throughput under heavy load.

desk verdict Plausible scheduling heuristic with a sound memory model, but the paper overclaims precision and misses uncited prior work on output-length prediction. read the letter →

arxiv 2507.10150 v1 pith:DSL5SYE7 submitted 2025-07-14 cs.DC

Ruihao Gong , Shihao Bai , Siyu Wu , Yunqian Fan , Zaijun Wang , Xiuhong Li , Hailong Yang , Xianglong Liu This is my paper
classification cs.DC
keywords LLMservingcontinuousbatchingrequestschedulingKVcachememorymanagementSLAguaranteesgoodputoutputlengthpredictionPast-Futurescheduler
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an LLM serving system can admit incoming requests much more aggressively without causing evictions if it forecasts the peak memory the running batch will need at every future decoding step. The forecast comes from two moves: sample each request's likely remaining output length from the recent history of finished requests, then compute the maximum memory occupancy the batch would hit at any future completion point, admitting a queued request only when that projected peak fits in memory. The authors implemented this Past-Future scheduler in a serving framework called LightLLM and report that under heavy load it meets SLA constraints while delivering up to 2–3 times the goodput of conservative schedulers, which over-reserve memory, and aggressive schedulers, which under-reserve memory and evict requests. If correct, the paper turns request scheduling from a static reservation problem into a dynamic peak-memory forecast, which matters for any deployment where output lengths vary widely.

What carries the argument

The Past-Future scheduler is the central object: a parameter-free admission-control rule combining two estimators. The first is $P(l) = C(l, L_h)/w$ (Equation 1), the empirical output-length distribution of the $w$ most recently finished requests. The second is the future-memory profile: requests are sorted by predicted remaining length, and $M_i = (\sum_{j=1}^{i}(l_j^p + l_j^t)) + (\hat{l}_i^t - l_i^t) \cdot i$ (Equation 3) gives the batch's memory occupancy at the moment the $i$-th request finishes, with $M^* = \max_i M_i$ (Equation 4) the peak needed to complete the batch. The admission rule is simply $M^* \le M$ (Algorithm 1), while running requests are resampled from $P(l > l_i^t)$ at each step so the projection tightens as requests make progress.

What would settle it

Run a serving workload with a known, abrupt change in output-length distribution halfway through a history window—for example, the first half of requests finish in 20–50 tokens and the second half in 2000–4000 tokens—and compare the scheduler's predicted $M^*$ against the actual peak memory needed to finish the batch. If $M^*$ stays under capacity while actual memory demand exceeds it, the central claim fails at its load-bearing assumption.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the future memory requirement of a running batch, not its current memory usage, is the right admission criterion, and that this future demand can be estimated to near-practical accuracy from the empirical distribution of recent output lengths. The scheduler sorts requests by predicted remaining length, computes at each completion point the occupancy $M_i$ as the sum of the memory of requests not yet finished at that point, and takes the maximum $M^* = \max_i M_i$; it admits a queued request exactly when $M^* \le M$ (Equations 2–4). It re-samples the predicted remaining length of every running request each step from the conditional distribution $P(l > l_i^t)$, so the projection tracks partial progress rather than assuming a fixed final length. The paper reports that this yields memory utilization close to the theoretical optimum (the case where true output lengths would be known) with far fewer evictions than aggressive scheduling, and better time-to-first-token and max-time-per-output-token SLA compliance under rising concurrency.

Load-bearing premise

The load-bearing premise is that the output lengths of requests arriving in the near future resemble the output lengths of recently finished requests; if that distribution shifts while the scheduler is still using old history, its projected peak memory will be wrong and it will either admit too many requests or waste memory.

Editorial extensions

If this is right

  • Admission control becomes a forecast of peak future memory rather than a static check of current usage, so batches can run closer to capacity without inviting later evictions.
  • The scheduler removes the need to hand-tune a memory watermark for aggressive scheduling or an overcommit ratio for conservative scheduling; a single reserved-memory percentage suffices across decode-heavy, balanced, and prefill-heavy workloads.
  • Because the prediction is parameter-free and model-independent, the same scheduling rule applies to any autoregressive model and adds negligible overhead.
  • Under rising client concurrency, goodput stabilizes near the hardware's SLA-limited maximum instead of degrading, which is the regime where current schedulers lose the most throughput.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same $M^*$ projection could drive proactive load balancing: a node whose projected peak is about to exceed capacity could direct new requests to a less-loaded peer before eviction becomes necessary.
  • A natural testable extension is to add a distribution-shift detector that resets the historical window; the paper's own trace analysis shows hybrid API workloads drift over hours, and the scheduler currently has no explicit mechanism to react within a window.
  • The authors' future-work suggestion about dynamic instance scaling follows directly: if $M^*$ reliably predicts when a node will saturate, it can trigger scaling decisions earlier than utilization metrics.
  • The paper itself notes that its head-to-head framework comparison reflects December 2023 versions of all frameworks, so the 2–3x gain is tied to those baselines; re-running the comparison against current releases is the natural check.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a Past-Future scheduler for continuous batching in LLM serving. The scheduler predicts each request's output length by sampling from the historical output-length distribution of recently completed requests, then estimates the future peak memory requirement of the running batch by computing memory occupancy at each future completion event (Eqs. 3-4). A queued request is admitted only when this predicted peak stays within a reserved fraction of the system memory capacity. The authors implement the scheduler in an open-source framework, LightLLM, and report goodput improvements of up to 2-3x over conservative and aggressive schedulers under SLA constraints on Llama-2 7B/13B/70B and multimodal models, together with trace-based evidence that output-length distributions are similar across adjacent time windows.

Significance. If the central claims hold, the paper makes a practical contribution to LLM serving: a lightweight, history-based scheduling rule that improves SLA-satisfying throughput without model-specific prediction models. The memory-accounting formulation in Eqs. 2-4 is coherent under a synchronous decode model, and the open-source release plus production deployment are tangible strengths. However, the paper's own Table 1 shows non-zero eviction rates for the Past-Future scheduler even with a reserved-memory hedge, and the evaluation does not isolate the transient behavior after an abrupt distribution shift. These issues undercut the abstract's 'precisely estimates' and 'consistently achieving better goodput' claims and need to be resolved before the paper can be accepted as written.

major comments (4)
  1. [Section 5.3, Table 1] Table 1 reports that Past-Future with reserved=3% still evicts 6.86%, 7.42%, and 2.59% of requests on Distribution-1, Distribution-2, and Distribution-3, while the same rows show average Future Required Memory below 100%. This contradicts Section 5.2's statement that 'all scheduled requests are completed successfully' and the abstract's claim of 'precisely estimat[ing] the peak memory resources.' If M* is accurate and the admission test is M* <= M, evictions should be essentially zero even without reserved memory. The paper should explain the source of these residual evictions (prediction error, fragmentation, admission races, or other causes) and either strengthen the admission mechanism to actually prevent evictions or qualify the precision claims to match the measured behavior.
  2. [Section 3.2 and Section 5.3, Figure 8] The scheduler's load-bearing premise is that the output-length distribution of the immediate future matches the recent past, so that samples from P(l) and P(l > l_t^i) in Algorithm 1 are reliable. Figure 3 itself shows that API/hybrid workloads have lower adjacent-window similarity, and the paper provides no mechanism to detect or adapt to a shift occurring within the current historical window. Figure 8 evaluates a concatenated workload only in aggregate, so a transient collapse in goodput or a burst of evictions immediately after each segment boundary would be hidden in the averages. Please add an experiment that reports goodput and evictions in short time intervals (e.g., every 100-200 requests) across a workload switch, and discuss whether the fixed reserved-memory hedge is sufficient or whether an adaptive mechanism is needed.
  3. [Section 3.2 and Section 4] The prediction method is described as 'parameter-free,' but Eq. 1 depends on the window size w, and the paper states in Section 4 that w=1000 was chosen based on Figure 4. The reserved memory ratio is also a free parameter, with 3%, 5%, and 10% variants explored in Table 1 and Figure 8. These parameters are tuned on the same workloads used for evaluation, which weakens the claim that the method generalizes across workloads without configuration. Please remove the 'parameter-free' description or provide a principled, workload-independent way to set w and the reserved ratio, and clearly state which results are sensitivity analyses of tuned parameters.
  4. [Section 4, startup behavior] The implementation initializes the output-length distribution with the preset maximum output length at service startup and only updates it after a few minutes of traffic. During that cold-start period, the scheduler is effectively conservative and will underutilize memory. The paper does not quantify the duration of this transient or its impact on goodput, yet the abstract and Section 5 claim consistent goodput improvements. Please report the cold-start behavior or explicitly scope the claims to steady-state operation after the history window is filled.
minor comments (6)
  1. [Section 3.2, Eq. 1] The notation L_h = {l_h^0, l_h^1, ..., l_h^w} contains w+1 elements while Eq. 1 divides the count by w; use a consistent indexing scheme, e.g., L_h = {l_h^1, ..., l_h^w} or divide by w+1.
  2. [Section 5.2] The sentence 'ensuring that all scheduled requests are completed successfully' is too strong given the eviction rates in Table 1; please replace it with a hedged formulation such as 'aims to ensure' or adjust the reported eviction data accordingly.
  3. [Section 5.3] The text refers to the 'Future-Past scheduler' in the paragraph on effect on request eviction; this should be 'Past-Future scheduler' for consistency.
  4. [Section 2.2] The algorithm is introduced as 'PageAttention' here but referred to as 'PagedAttention' elsewhere; please use one spelling consistently.
  5. [Section 3.2] The claim that output-length distributions are 'stable within a short time period (minutes)' is qualitative; please quantify the timescale using the trace data in Figure 3 or a similar analysis so that the stationarity assumption can be checked.
  6. [Section 5.3, Figure 8] The parameter labels in Figure 8 (overcommit, watermark, reserved) are dense and the marker styles for the three scheduler families are difficult to distinguish; add a clearer legend or separate subplots for each scheduler.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scheduler's memory-peak estimate is a genuine forward computation from an external historical distribution, not a restatement of its inputs.

full rationale

I find no circular step. The prediction pipeline is: (1) build P(l) from finished requests (Eq. 1); (2) sample or resample predicted final lengths for queued and running requests (Algorithm 1 lines 4 and 8); (3) sort by remaining length (Eq. 2) and compute the maximum intermediate memory M* (Eqs. 3-4); (4) admit while M* <= M. Each step is a forward calculation from observed history and current state, and the goodput/eviction results are measured against real decoding, not derived from the scheduler's own equations. The stationarity of P(l) across adjacent windows is an empirical premise validated in Figures 3-4; if it fails, the estimate is inaccurate, but the inaccuracy is not a circular definition. The only author self-citations ([11], [37]) appear in related work and are not load-bearing. The window size of 1,000 and the reserved-memory ratios are hyperparameters reported with sensitivity analysis (Figure 8, Table 1), not outputs of the derivation, and they are not used to define the predicted memory peak. The claim of 2-3x goodput improvement is an empirical benchmark outcome rather than a consequence of the scheduler's equations. Therefore the derivation chain is self-contained and the paper is not circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The scheduler rests on two engineering knobs (window size, reserved ratio) and four domain assumptions about workload stationarity, synchronous decoding, conditional sampling, and token-budget memory. No new physical entities or conservation laws are introduced.

free parameters (3)
  • Historical window size w = 1000 requests
    Used to build P(l) in Eq 1; chosen as a balance in Figure 4 and 'adopted this setting uniformly' in Section 3.2; affects all admission decisions.
  • Reserved memory ratio = 3%, 5%, 10%
    Extra capacity margin added in scheduling (Table 1, Figure 8); controls the tradeoff between evictions and memory utilization, with no workload-independent selection rule.
  • Initial output length distribution = preset max_new_tokens
    At service startup, the distribution is initialized to the preset maximum output length and updated within minutes (Section 4), so early decisions use a worst-case assumption.
assumptions (4)
  • domain assumption Output length distribution is stable across adjacent time windows.
    Central to Section 3.2; Figure 3 and 4 show cosine similarity, but for API/hybrid services the long-term distribution varies, and stability is only asserted, not guaranteed.
  • domain assumption All running requests decode synchronously at one token per step, so memory at future completion points follows Eq 3.
    Used in Section 3.3 Eq 2-3; ignores variable decode rates, prefill/decode interleaving and block-level allocation.
  • domain assumption Each request's remaining output length is independent of input content and can be sampled from the historical distribution conditioned on current length.
    Algorithm 1 lines 3-8; if output lengths depend on prompt category, the sampling is biased.
  • domain assumption Memory capacity can be modeled as a token budget; KV cache size equals input tokens plus generated tokens.
    Eq 2-4; does not account for activation memory, fragmentation or non-token-aligned block allocation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Past-Future Scheduler for LLM Serving under SLA Guarantees." pith.science (2026). https://pith.science/paper/DSL5SYE7

@misc{pith2026250710150,
  author       = {Pith},
  title        = {Pith review of: Past-Future Scheduler for LLM Serving under SLA Guarantees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DSL5SYE7}},
  note         = {Machine review of arXiv:2507.10150}
}
abstract

The exploration and application of Large Language Models (LLMs) is thriving. To reduce deployment costs, continuous batching has become an essential feature in current service frameworks. The effectiveness of continuous batching relies on an accurate estimate of the memory requirements of requests. However, due to the diversity in request output lengths, existing frameworks tend to adopt aggressive or conservative schedulers, which often result in significant overestimation or underestimation of memory consumption. Consequently, they suffer from harmful request evictions or prolonged queuing times, failing to achieve satisfactory throughput under strict Service Level Agreement (SLA) guarantees (a.k.a. goodput), across various LLM application scenarios with differing input-output length distributions. To address this issue, we propose a novel Past-Future scheduler that precisely estimates the peak memory resources required by the running batch via considering the historical distribution of request output lengths and calculating memory occupancy at each future time point. It adapts to applications with all types of input-output length distributions, balancing the trade-off between request queuing and harmful evictions, thereby consistently achieving better goodput. Furthermore, to validate the effectiveness of the proposed scheduler, we developed a high-performance LLM serving framework, LightLLM, that implements the Past-Future scheduler. Compared to existing aggressive or conservative schedulers, LightLLM demonstrates superior goodput, achieving up to 2-3$\times$ higher goodput than other schedulers under heavy loads. LightLLM is open source to boost the research in such direction (https://github.com/ModelTC/lightllm).

Figures

Figures reproduced from arXiv: 2507.10150 by the authors.

Figure 1
Figure 1. Comparison of current consumed memory (solid), future required memory (dashed), and request eviction rate (>100% means one request is evicted multiple times on av￾erage) for different schedulers under various input-output length distributions. Lower memory utilization indicates lower efficiency, while future required memory exceeds the memory capacity will lead to request evictions afterward. processing units. Other… view at source ↗
Figure 2
Figure 2. The overview of the proposed Past-Future scheduler. At time t, the service includes a running batch with 𝑘 requests and a request queue with 𝑛 − 𝑘 requests. The proposed scheduler will first predict the most probable output length for queued requests and update the predicted output length of each request in the running batch based on the historical distribution of request output lengths and their output lengths at t… view at source ↗
Figure 3
Figure 3. Cosine similarity of output length distributions (higher values indicate higher similarity) between the different partitioned time windows (1000 requests, no overlap) of multiple request trace dataset, points indicating the overlapped request window are excluded. The adjacent time windows (the diagonal pattern) have similar distributions (some dataset has similar distributions globally). 100 200 500 1000 2000 5000 H… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Average cosine similarity on global or diagonal line under different historical (x-axis) and running request win￾dows size (in different brightnesses of a color, corresponding to the number markers on the right) in the BurstGPT dataset. adopted this setting uniformly i…
Figure 5
Figure 5. Figure 5: Memory demand of a running batch when the queuing request is scheduled at different time steps [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Behavior comparison of different schedulers when handling new requests at time t. It shows that our method can introduce requests at the appropriate time, maintaining high memory utilization while avoiding harmful request evictions. the shortest output completes and re…
Figure 7
Figure 7. Figure 7: Detailed goodput under different numbers of clients for different schedulers. Conservative schedulers can only achieve low goodput at all times, while aggressive schedulers, although they can achieve equally goodput under light loads, have a rapid degradation of goodpu…
Figure 8
Figure 8. Figure 8: Comparison of different scheduler parameters on a varying load consisting of ShareGPT-o1 and Distribution-1 to 3. utilized when the request output length is known, which is impossible in real-world service. Effect on Request Eviction. Although an aggressive sched￾uler …
Figure 9
Figure 9. Figure 9: Throughput (dashed) and SLA-guarantee goodput (solid) of different frameworks across various hardware. SLA constraints marked as P99 TTFT 10s, P99 MTPOT 1.5s refers to the 99% of the requests’ first token latency must be < 10s, and the requests’ maximum per-token laten…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 25 canonical work pages

  1. [1]

    Anthropic. 2023. Anthropic Claude. https://claude.ai/

    work page 2023

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, et al. 2023. Qwen Technical Report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities. arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  4. [4]

    Baichuan. 2023. Baichuan 2: Open Large-scale Language Models.arXiv preprint arXiv:2309.10305 (2023). https://arxiv.org/abs/2309.10305

    arXiv 2023

  5. [5]

    Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Paral- lelism and Work Partitioning. (2023)

    work page 2023

  6. [6]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

  7. [7]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 320–335

    work page 2022

  8. [8]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers. arXiv preprint arXiv:2210.17323 (2022)

    arXiv 2022

Show all 45 references
  1. [9]

    Github. 2022. Github Copilot. https://github.com/features/copilot Ruihao Gong et al

    2022

  2. [10]

    Google. 2023. Google Bard. https://bard.google.com/

    2023

  3. [11]

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Hanyu Dong, and Yu Wang. 2023. FlashDecoding++: Faster Large Language Model Inference on GPUs. arXiv preprint arXiv:2311.01282 (2023)

    2023 arXiv

  4. [12]

    HuggingFace. 2023. Text Generation Inference. https://github.com/ huggingface/text-generation-inference

    2023

  5. [13]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  6. [14]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, et al

  7. [15]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee

  8. [16]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In NeurIPS

    2023

  9. [17]

    StarCoder: may the source be with you! arXiv:2305.06161 [cs.CL]

    arXiv

  10. [18]

    Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gul- cehre, and Bing Xiang. 2016. Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. Association for Computa- tional Linguistics, Berlin, Germany, 280–290. https://doi.org/10.18653/ v1/K16-1028

    2016

  11. [19]

    arXiv:2310.03744 [cs.CV]

    Improved Baselines with Visual Instruction Tuning. arXiv:2310.03744 [cs.CV]

    arXiv

  12. [20]

    NVIDIA. 2023. TensorRT-LLM. https://github.com/NVIDIA/TensorRT- LLM/

    2023

  13. [21]

    Microsoft. 2023. DeepSpeed-FastGen: High-throughput Text Genera- tion for LLMs via MII and DeepSpeed-Inference. https://github.com/ microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen

    2023

  14. [22]

    OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]

    2023 arXiv

  15. [23]

    NVIDIA. 2021. FasterTransformer. https://github.com/NVIDIA/ FasterTransformer/

    2021

  16. [24]

    Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A Deep Reinforced Model for Abstractive Summarization. InInternational Con- ference on Learning Representations. https://openreview.net/forum? id=HkAClQgA-

    2018

  17. [25]

    OpenAI. 2022. OpenAI ChatGPT. https://openai.com/blog/chatgpt

    2022

  18. [26]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. Association for Computational Linguistics, Vancouver, Canada, 1073–1083. https: //doi.org/10.18653/v1/P17-1099

    2017 doi

  19. [27]

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017)

    2017

  20. [28]

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA Models That Can Read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8317–8326

    2019

  21. [29]

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, , and Xinran Xu. 2024. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. (2024). https://arxiv. org/abs/2407.00079

    2024 arXiv

  22. [30]

    InternLM Team. 2023. InternLM: A Multilingual Language Model with Progressively Enhanced Capabilities. https://github.com/InternLM/ InternLM-techreport

    2023

  23. [31]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. 2023. FlexGen: high-throughput generative inference of large language models with a single GPU. In International Conference on Machine Learning...

    2023

  24. [32]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  25. [33]

    01-AI Team. 2023. Yi LLM. https://github.com/01-ai/Yi

    2023

  26. [34]

    Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2024. To- wards Efficient and Reliable LLM Serving: A Real-World Workload Study. arXiv:2401.17644 [cs.DC]

    2024 arXiv

  27. [35]

    Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: An Intermedi- ate Language and Compiler for Tiled Neural Network Computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (Phoenix, AZ, USA) (MAPL 2019). A...

    2019

  28. [36]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems , Alice H. Oh, Alekh Agarwal, ...

    2022

  29. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL]

    2023 arXiv

  30. [38]

    Wikipedia. 2023. Service-level agreement. (2023). https://en. wikipedia.org/wiki/Service-level_agreement [Online; accessed 24- October-2023]

    2023

  31. [39]

    Jason Wei, Yi Tay, Rishi Bommasani, Barret Zoph Colin Raffel, Sebas- tian Borgeaud, Dani Yogatama, et al. 2022. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research (2022). https://openreview.net/forum?id=yzkSU5zdwD Survey Certification

    2022

  32. [40]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 521–538

    2022

  33. [41]

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. 2023. Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference on Empirical M...

    2023

  34. [43]

    BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, et al. 2023. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv:2211.05100 [cs.CL]

    2023 arXiv

  35. [45]

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al . 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 (2022)

    2022 arXiv

  36. [2022]

    In Advances in Neural Information Processing Systems

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems

  37. [2023]

    In Proceedings of the 29th Symposium on Operating Systems Principles

    Efficient memory management for large language model serv- ing with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.