Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Multi-Bin Batching for Increasing LLM Inference Throughput

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Batching requests by predicted length provably raises LLM inference throughput.

desk verdict Solid queueing analysis of multi-bin batching for LLM serving, but the provable result lives under a uniform-service-time assumption that the real experiments don't satisfy; worth refereeing with a required scope fix. read the letter →

arxiv 2412.04504 v1 pith:D3JCBPZ5 submitted 2024-12-03 cs.CL cs.DCcs.LGcs.SYeess.SY

Ozgur Guldogan , Jackson Kunde , Kangwook Lee , Ramtin Pedarsani This is my paper · ORCID
classification cs.CLcs.DCcs.LGcs.SYeess.SY MSC 60K2568M2090B22
keywords LLMinferencethroughputbatchingqueueingtheoryschedulingoutputlengthpredictionmulti-binorderstatistics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-Bin Batching sorts incoming LLM requests into k bins by their predicted generation length before forming batches, so requests with similar execution times are served together. The paper's central claim is that this simple scheduler provably increases expected throughput: under a Poisson arrival model with uniformly distributed service times, the expected throughput of k-bin batching is an explicitly given formula that increases with k and converges to the system's theoretical maximum as k grows. The practical motivation is that standard batching wastes computation because every batch waits for its longest request, and LLM output lengths vary widely. The paper supports the claim with queueing-theoretic proofs and with LLM inference experiments reporting throughput gains of up to 70 percent when true output lengths are used for binning.

What carries the argument

The workhorse is the order-statistic formula for the expected maximum of $B$ i.i.d. uniform random variables on an interval $[a,b]$, which is a weighted average of the endpoints: $E[\max] = (B b + a)/(B+1)$. Multi-bin batching partitions $[l_{\min}, l_{\max}]$ into k bins and uses this identity to write the expected batch service time as a convex function of the k-1 interior boundaries. Minimizing that function yields the optimal equal-probability bin boundaries of Lemma 4.1; substituting them back gives the closed-form throughput expression of Theorem 4.2. Thus the entire argument reduces to making each batch's service time as close as possible to a single request's service time by shrinking the spread within each bin.

What would settle it

Simulate k-bin batching with a fixed batch size and a service-time distribution that is visibly non-uniform, such as a log-normal fitted to real LLM output lengths. If equal-probability boundaries do not maximize throughput, or if throughput does not increase monotonically with k, the paper's central claim fails outside its model. A more direct check: for uniform traffic, measure the average batch service time for k = 1, 2, 4, 8 and compare it to the denominator of Theorem 4.2.

Watch

Extended reading notes

Core claim

The paper claims that the throughput ceiling of a batched LLM server is set by the mean request service time, and that k-bin batching approaches that ceiling from below. Concretely, Theorem 4.2 states that under i.i.d. uniform service times $l\sim U(l_{\min},l_{\max})$ and batch size $B$, the expected throughput is $$\text{Throughput}_k = \frac{B}{\frac{l_{\max}+l_{\min}}{2} + \frac{1}{k}\left(\frac{B}{B+1}l_{\max} + \frac{1}{B+1}l_{\min} - \frac{l_{\max}+l_{\min}}{2}\right)},$$ and this quantity is increasing in $k$. As $k$ grows, the extra term in the denominator vanishes, so throughput converges to $B / ((l_{\max}+l_{\min})/2)$, which the paper identifies as the maximum capacity under perfect parallelism (Remark 4.3). The paper also proves that a target throughput within $\epsilon$ of the ceiling is reachable with $k = O(1/\epsilon)$ bins, making the asymptotic optimality practically accessible with a small number of bins.

Load-bearing premise

The load-bearing premise is that every request's execution time is drawn independently and evenly across the whole possible range from shortest to longest; if real output lengths cluster at certain values, the paper's optimal bin boundaries and throughput formula no longer apply.

Editorial extensions

If this is right

  • Under the uniform service-time model, throughput with k bins is strictly increasing in k and converges to $B/( (l_{\max}+l_{\min})/2 )$, the ceiling set by the mean request service time.
  • Reaching within $\epsilon$ of that ceiling requires only $k = O(1/\epsilon)$ bins, so most of the theoretical gain is available with a small number of bins.
  • Standard single-queue batching is exactly the $k=1$ case, so multi-bin batching dominates it whenever requests are placed in the correct bins.
  • The latency cost of binning is concrete: the expected time spent waiting to fill a batch grows linearly with k (Lemma 5.1), so choosing k is a throughput-latency trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves open is to re-derive optimal bin boundaries from an empirical service-time distribution; the same minimize-the-expected-maximum objective applies, but equal-probability boundaries are only optimal under uniformity.
  • An implicit consequence of the latency formula is that for any fixed quality-of-service budget there is a finite optimal k that maximizes throughput; the paper gives the two closed forms needed to compute it but does not solve the combined optimization.
  • A testable prediction of the asymptotic result is that the ceiling depends only on mean service time, so heavy-tailed output lengths mainly change how many bins are needed, not the limiting throughput; this could be checked by simulating log-normal or Pareto length distributions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Multi-Bin Batching, a scheduling policy for LLM inference that groups requests with similar (predicted) execution times into k bins before forming batches of size B, with the goal of reducing the idle time caused by the longest-running request in a batch. The theoretical core (Section 4) assumes i.i.d. uniform service times in [lmin, lmax] (Assumption 3.2) and derives the optimal bin boundaries (equal probability mass, Lemma 4.1), a closed-form expression for the expected service time and hence throughput as a function of k (Theorem 4.2), and a condition on the number of bins needed to reach a throughput within epsilon of the maximum capacity (Theorem 4.3). A latency lower bound is derived in Section 5 under an infinite-server assumption. The experimental sections (Section 6 and Appendices A.4, C) present simulations with a linear token-time model, end-to-end LLM experiments with oracle length assignment, robustness tests under symmetric prediction errors, and a BERT-based length predictor, reporting throughput gains up to about 70% in the oracle setting and smaller gains with the learned predictor.

Significance. If the central claim holds, the paper contributes a simple, interpretable scheduling rule with a clean theoretical justification: under a stylized uniform service-time model, binning by equal probability is optimal and throughput monotonically increases with the number of bins, approaching the idealized capacity B/E[t]. The paper also provides a useful characterization of the number of bins required for a given throughput target, and the experiments cover several realistic settings, including a learned predictor. The theoretical derivation is internally correct under Assumption 3.2, and the empirical results are encouraging. However, the provable guarantees are limited to the uniform-distribution model, and the paper's abstract and conclusion state the 'provably improves' claim more broadly than the proof supports. The gap between the theoretical model and the non-uniform LLM workloads evaluated is acknowledged in the text but not resolved by a robustness theorem or a distribution-agnostic analysis. This limits the strength of the headline contribution but does not negate the value of the proposed heuristic.

major comments (3)
  1. [Section 4, Lemma 4.1 and Theorem 4.2; Section 6.1] The optimality of equal-probability bins and the exact throughput formula in Theorem 4.2 are proven only under Assumption 3.2 (i.i.d. uniform service times). Real LLM output lengths are not uniform, as the paper itself states in the Figure 6 discussion ('differences between the theoretical assumption of uniform output length distributions and the actual distributions observed in the dataset'). Yet the experiments in Sections 6.1, 6.2, and Appendix C set each bin to be equiprobable and attribute the measured throughput gains to the policy without any theorem covering non-uniform distributions. Equal-probability binning is not generally the minimizer of expected batch service time for non-uniform distributions, so the 'provably improves LLM inference throughput' claim in the abstract is not established for the evaluated workloads. The paper should either add a robustness analysis (e.g., a bound on the suboptimality of equal-probability bins for other distributions) or restrict the theoretical claims to the uniform model and present the real-world experiments as empirical evidence only.
  2. [Section 4, Proposition 4.1] The quantity B / E[t_service] is the service capacity of the system—the maximum sustainable throughput when the server is continuously busy—not the actual throughput of the queueing system under stochastic arrivals with possibly idle time or batch-formation delays. The paper labels this 'expected throughput' and uses it in Theorem 4.2 without accounting for arrival rates, queueing, or the time to fill batches. In an underloaded system with arrival rate lambda, the achieved throughput is bounded by lambda, so Eq. (4) does not directly characterize the throughput improvement in the simulation and experiments that measure total tokens divided by total time. The paper should clarify this distinction and either rename the theoretical quantity 'service capacity' or model the actual throughput, including the finite-arrival regime.
  3. [Throughout (abstract, Section 1, Remark 4.3)] The phrase 'throughput-optimal' is used in a nonstandard way. In queueing theory, throughput-optimal typically refers to a policy that stabilizes the system for all arrival rates within the stability region. Here the asymptotic optimality in Remark 4.3 is relative to the idealized capacity B/E[t] under the uniform service-time model, not to a comparison against the optimal policy within a broader class of batch-formation or scheduling policies. The paper should define the optimality class explicitly (e.g., 'within the class of static binning policies under Assumption 3.2') and avoid unqualified 'throughput-optimal' phrasing, which overstates the theoretical result.
minor comments (5)
  1. [Appendix A.1, Eq. (8) and Eq. (10)] The coefficients in the formula for the expectation of the maximum are swapped: for B i.i.d. uniform variables on [l_{i-1}, l_i], the expectation is (B l_i + l_{i-1})/(B+1), i.e., B/(B+1) l_i + 1/(B+1) l_{i-1}. Equations (8) and (10) use B/(B+1) l_{i-1} + 1/(B+1) l_i, while Eq. (17) in Appendix A.2 uses the correct order. This is a typo, but it is confusing because the main theorem depends on the correct form.
  2. [Algorithm 1] The bin-assignment condition 'l_{i-1} <= l < l_i' does not cover the right endpoint lmax for the last bin; the final bin should include its upper boundary or use a half-open interval that includes lmax.
  3. [Section 6.1, Figure 6 caption] The caption contains a typo ('incresases' should be 'increases'). More substantively, the sentence about latency 'initially drops then increases' would benefit from a brief explanation of why the minimum latency occurs at intermediate k, since the text ties this to the non-uniform distribution but does not provide an intuitive mechanism.
  4. [Section 6.1, Figure 7 discussion] The 70% improvement is measured under the 'ideal conditions' of simultaneous arrival and oracle length information, which is correctly acknowledged. It would be helpful to state in the main text that the gain shrinks to roughly 8% with the learned predictor (Section C.1), so readers do not overgeneralize the oracle result.
  5. [Section 3, Assumption 3.2] The justification that LLM answer lengths typically fall within a range due to maximum token limits does not actually support the uniform-distribution assumption; boundedness alone does not imply uniformity. A sentence acknowledging this gap and pointing to Section 6.1 or a robustness discussion would improve the presentation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the throughput formula is derived from explicit order-statistics assumptions and is not fitted to the experiments; the uniform-service-time limitation is a scope caveat, not a circular reduction.

full rationale

The central derivation chain is self-contained. Proposition 4.1 defines throughput as B/E[tservice]; Lemma 4.1 solves a convex optimization over bin boundaries under Assumption 3.2 (i.i.d. uniform service times), and Theorem 4.2 substitutes those boundaries into the order-statistics expression for the expected maximum of B uniform samples in a bin to obtain Equation (4). The increasing-in-k and convergence-to-cmax statements are direct algebraic consequences of that formula, not restatements of the experimental results. No fitted parameter enters Theorem 4.2, and no external or self-cited theorem is invoked to carry the load; the only cited mathematical fact is the standard expectation of the maximum of uniform samples, used in the proof of Lemma 4.1. The LLM experiments in Sections 6.1, 6.2, and C.1 evaluate the policy empirically rather than fitting Equation (4) to data: the linear model in Figure 5 is fitted to token-count versus latency and then used to simulate service times, but throughput is measured as an output, not fed back into the formula. The paper does contain a genuine scope limitation, stated in the Figure 6 discussion: 'This counterintuitive result may stem from differences between the theoretical assumption of uniform output length distributions and the actual distributions observed in the dataset.' That limits how much of the real-world throughput gain is covered by Lemma 4.1's optimality and Theorem 4.2's exact quantitative predictions, but it is a domain-of-validity caveat, not a circularity: the theorem does not assume the conclusion it proves. Since the derivation is mathematically independent of the experiments and no fitted or self-referential input is renamed as a prediction, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central theoretical claim rests on the uniform distribution assumption and the batch-max service model; the equal-probability bin rule is derived from that assumption. In experiments, the linear token-time model adds a fitted parameter. No new physical or conceptual entities are introduced.

free parameters (2)
  • lmin, lmax (uniform service time bounds) = 1, 20 in simulations
    Assumed parameters of the uniform service-time distribution (Assumption 3.2); chosen by hand for simulations, not fitted to real LLM output-length distributions.
  • Linear token-time slope = 0.0366 s/token
    Linear regression fitted to Phi-3.5 inference times in Figure 5, used to simulate service times in Section 6.1. Affects simulated throughput numbers but not the theoretical formula.
assumptions (5)
  • domain assumption Requests arrive as a Poisson process at rate lambda; single-server queue with infinite buffer.
    Assumption 3.1. Standard for queueing analysis but not empirically validated for LLM inference traffic.
  • ad hoc to paper Service times of requests are i.i.d. uniform on [lmin, lmax].
    Assumption 3.2. Chosen for tractability; the paper only argues answer lengths fall in a range, which does not imply uniformity. The optimality of equal-probability bins depends on this.
  • domain assumption Batch service time is the maximum of the service times of its requests; batches are served first-formed-first-served.
    System model in Section 3. Reflects the assumption that no fine-grained dispatching or continuous batching is available (footnote 1).
  • domain assumption For latency analysis, the number of servers is infinite.
    Assumption 5.1. Provides a lower bound on latency but the paper uses it as the latency estimate in Figure 4, which is only accurate at low load.
  • domain assumption Inference time is a linear function of the number of generated tokens.
    Section 6.1 and Figure 5. Linear regression fit to Phi-3.5 data; used to generate simulated service times, but not a proven law.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Bin Batching for Increasing LLM Inference Throughput." pith.science (2026). https://pith.science/paper/D3JCBPZ5

@misc{pith2026241204504,
  author       = {Pith},
  title        = {Pith review of: Multi-Bin Batching for Increasing LLM Inference Throughput},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D3JCBPZ5}},
  note         = {Machine review of arXiv:2412.04504}
}
read the original abstract

As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios, we demonstrate significant throughput gains compared to standard batching approaches.

Figures

Figures reproduced from arXiv: 2412.04504 by the authors.

Figure 1
Figure 1. (a) Batch serving improves the throughput for the LLM inference systems. (b) Standard batching causes underutilization of resources due to varying answer sizes. batching system with a batch size of B = 2, the requests would be grouped based on their arrival time, forming two batches: (Request 1: 1s, Request 2: 5s) and (Request 3: 2s, Request 4: 6s). The total execution time would be 11 seconds (5 seconds for the fir… view at source ↗
Figure 2
Figure 2. k-Bin Batching The multi-bin batching algorithm is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average throughput of k-bin batching vs the arrival rate. The proof of Theorem 4.3 is given in A.3. The proof utilizes the expected throughput derivation in Theorem 4.2 to find the smallest integer that achieves the desired throughput [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The latency of a request vs the arrival rate λ for different k values. The proof of Lemma 5.1 is given in B.1. The proof utilizes the fact that the arrival process is Poisson and the effective arrival rate of each bin is λ/k. In [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The linear relationship be￾tween the number of tokens generated and the inference time To thoroughly analyze the throughput improvements from our multi￾bin batching approach, we conduct three different experiments, with increasing levels of realism. Each experiment has…
Figure 6
Figure 6. Figure 6: Throughput rises with more bins, while latency initially [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Throughput with known output lengths. To better understand the throughput gains from our control policy, we replace the linear model with the actual inference times for an LLM responding to a batch of requests. During inference, the server is considered fully occupied.…
Figure 8
Figure 8. Figure 8: Throughput of the system with respect to the error probability [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: (a) Throughput of the system with respect to the arrival rate λ for different values of k. (b) Expected latency of a request with respect to the arrival rate λ for different values of k. B Proofs for Latency Analysis B.1 Proof of Lemma 5.1 In this section, we provide t…
Figure 10
Figure 10. Figure 10: Confusion matrices for the BERT-based length predictor for 2, 4, and 8 bins. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Inference throughput improves with more bins using a BERT-based pre￾dictor. We conducted end-to-end inference experiments using Vi￾cuna 13B (Chiang et al., 2023), employing our BERT-based predictor for bin assignment. We follow a similar setup as the oracle length exp…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training

    cs.AI 2026-06 conditional novelty 6.0 of 10

    Dynamically reassigning GPUs between rollout and training during LLM post-training cuts pipeline bubbles, improving measured throughput by 66.5% over the best static split in the paper's 8-GPU test.

  2. Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Auras, a perception-generation disaggregation framework with a public context buffer and asynchronous pipeline executor, raises embodied-agent throughput by 2.54x on average without losing accuracy (102.7%).

  3. Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A GRPO decoding framework that cuts memory via micro-batched KV-cache reuse and improves decoding-round efficiency with predicted-length scheduling, at the cost of serialization.

Reference graph

Works this paper leans on

29 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

    arXiv 2021

  3. [3]

    Enabling efficient batch serving for lmaas via generation length prediction, 2024 a

    Ke Cheng, Wen Hu, Zhi Wang, Peng Du, Jianguo Li, and Sheng Zhang. Enabling efficient batch serving for lmaas via generation length prediction, 2024 a . URL https://arxiv.org/abs/2406.04785

    arXiv 2024

  4. [4]

    Slice-level scheduling for high throughput and load balanced llm serving

    Ke Cheng, Wen Hu, Zhi Wang, Hongen Peng, Jianguo Li, and Sheng Zhang. Slice-level scheduling for high throughput and load balanced llm serving. arXiv preprint arXiv:2406.13511, 2024 b

    arXiv 2024

  5. [5]

    Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2 0 (3): 0 6, 2023

    2023

  6. [6]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [7]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 0 30318--30332, 2022

    2022

  8. [8]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

Show all 29 references
  1. [9]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805

    2018 arXiv

  2. [10]

    OPTQ : Accurate quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ : Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS

    2023

  3. [11]

    Efficient llm scheduling by learning to rank

    Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. Efficient llm scheduling by learning to rank. arXiv preprint arXiv:2408.15792, 2024

    2024 arXiv

  4. [12]

    Queueing analysis of gpu-based inference servers with dynamic batching: A closed-form characterization

    Yoshiaki Inoue. Queueing analysis of gpu-based inference servers with dynamic batching: A closed-form characterization. Performance Evaluation, 147: 0 102183, May 2021. ISSN 0166-5316. doi:10.1016/j.peva.2020.102183. URL http://dx.doi.org/10.1016/j.peva.2020.102183

    2021

  5. [13]

    How can we know when language models know? on the calibration of language models for question answering

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9: 0 962--977, 2021

    2021

  6. [14]

    s^ 3 : Increasing gpu utilization during generative inference for higher throughput

    Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. s^ 3 : Increasing gpu utilization during generative inference for higher throughput. Advances in Neural Information Processing Systems, 36: 0 18015--18027, 2023

    2023

  7. [15]

    Recurrence relations between moments of order statistics from exponential and truncated exponential distributions

    PC Joshi. Recurrence relations between moments of order statistics from exponential and truncated exponential distributions. Sankhy \=a : The Indian Journal of Statistics, Series B , pages 362--371, 1978

    1978

  8. [16]

    Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm

    Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. arXiv preprint arXiv:2403.05527, 2024

    2024 arXiv

  9. [17]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  10. [18]

    Pre-trained language models for text generation: A survey

    Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56 0 (9): 0 1--39, 2024

    2024

  11. [19]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 0 87--...

    2024

  12. [20]

    Syncintellects: Orchestrating llm inference with progressive prediction and qos-friendly control

    Xue Lin, Zhibo Zhang, Peining Yue, Haoran Li, Jin Zhang, Baoyu Fan, Huayou Su, and Xiaoli Gong. Syncintellects: Orchestrating llm inference with progressive prediction and qos-friendly control. In 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pages ...

    2024

  13. [21]

    Kalbarczyk, Tamer Ba s ar, and Ravishankar K

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T. Kalbarczyk, Tamer Ba s ar, and Ravishankar K. Iyer. Efficient interactive llm serving with proxy model-based sequence length prediction. In The 5th International Workshop ...

    2024

  14. [22]

    F lex G en: High-throughput generative inference of large language models with a single GPU

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. F lex G en: High-throughput generative inference of large language models with a single GPU . In Andreas Krause, Emma Brunskill, Kyunghyun Cho,...

    2023

  15. [23]

    Llumnix: Dynamic scheduling for large language model serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 173--191, Santa Clara, CA, July 2024. USE...

    2024

  16. [24]

    Fast distributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023

    2023 arXiv

  17. [25]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087--38099. PMLR, 2023

    2023

  18. [26]

    A queueing theoretic perspective on low-latency llm inference with variable token length

    Yuqing Yang, Yuedong Xu, and Lei Jiao. A queueing theoretic perspective on low-latency llm inference with variable token length. arXiv preprint arXiv:2407.05347, 2024

    2024

  19. [27]

    Orca: A distributed serving system for \ Transformer-Based \ generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for \ Transformer-Based \ generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521--538, 2022

    2022

  20. [28]

    Lmsys-chat-1m: A large-scale real-world llm conversation dataset

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998, 2023

    2023 arXiv

  21. [29]

    Response length perception and sequence scheduling: An llm-empowered llm inference pipeline

    Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. Response length perception and sequence scheduling: An llm-empowered llm inference pipeline. Advances in Neural Information Processing Systems, 36, 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.