Pith. sign in

REVIEW 5 major objections 6 minor 34 references

Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hybrid offline-online scheduler raises LLM serving hardware utilization from 80.2% to 89.1% and cuts makespan from 201 to 190.58 seconds on LLaMA-65B over GSM8K.

desk verdict Useful scheduling idea and real hardware runs, but the evaluation uses oracle output lengths and the MIP has a load-bearing decode-time error; reject as is, though the direction is worth a serious look. read the letter →

arxiv 2502.15763 v1 pith:27ZR6DST submitted 2025-02-14 cs.DC cs.AIcs.ARcs.LG

Bowen Pang , Kai Li , Ruifeng She , Feifan Wang This is my paper
classification cs.DCcs.AIcs.ARcs.LG MSC 90B3590C11
keywords largelanguagemodelinferenceschedulingprefill-decodemixed-integerprogrammingbinpackingonlinepreemptiveGPUutilization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that LLM inference serving can be treated as a scheduling problem with two decision layers, and that a hybrid offline-online method beats the common prefill-first policy. It formulates the full problem as a mixed-integer program, then decomposes it: an offline bin-packing model balances requests across clients, while an online rule sorts requests by total token count and uses a Lagrangian cost comparison to decide when to preempt decode stages with prefill stages. On a real deployment of LLaMA-65B serving the GSM8K dataset, the combined method raises measured hardware utilization from 80.2% to 89.1% and lowers total inference time from 201.00 to 190.58 seconds. A 100-case study reports the method consistently beats the baseline by an average of 8.0 percentage points in utilization.

What carries the argument

The carrying mechanism is a two-level decomposition of a single mixed-integer program. Offline, a Minimizing Makespan Bin Packing Problem assigns requests to clients to minimize the maximum client decode load, and a theoretical lower bound is computed by adding the minimum possible total prefill time to the optimal decode makespan. Online, a sorting and preemptive rule lets an idle client take the request with the largest remaining token count from a busy client, while an iteration scheduler compares the prefill cost $C_p=\sum_l T^p_l$ against the waited decode time $C_d=T^d\sum_i N^d_i w_{i,j,k}$ and inserts a prefill stage only when $C_p < C_d$. This cost comparison is designed to be evaluated within milliseconds, matching the roughly 50-millisecond decode rounds of real serving systems.

What would settle it

Run the same hybrid scheduler on a live workload in which output lengths are predicted rather than taken from finished responses, and compare utilization with the 89.1% reported here; if the gain over the baseline shrinks toward zero, the output-length assumption is the cause.

Watch

Extended reading notes

Core claim

The central discovery is that the idle 'bubbles' in LLM serving come from unbalanced client workloads plus a prefill-first policy that never reorders or preempts, and both can be attacked in milliseconds. The paper claims that balancing requests across clients by solving a minimax bin-packing model, treating each request's estimated total token time as its size, raises utilization from 80.2% to 85.5%. Adding online request sorting, where an idle client steals the request with the largest remaining token load, plus iteration-level preemption, where a prefill round is inserted when its marginal cost is below the waiting decode cost, further lifts utilization to 89.1%. These gains are claimed to be consistent over 100 randomly generated cases, with an average utilization improvement of 8.0% over the baseline.

Load-bearing premise

The load-bearing premise is that each request's output token count is known when requests are balanced and when the preemption rule is evaluated, even though the paper states that output length is unknown until generation ends.

Editorial extensions

If this is right

  • Deploying the offline assignment alone, for workloads known in advance such as RLHF-style batch training, should reduce client idle time and raise utilization to about 85.5% without changing the online policy.
  • The online-only variant, which needs no prior knowledge of the request set, still improves utilization to 86.2% and cuts inference time to 193.3 seconds, so the method claims applicability to ordinary interactive serving.
  • The Lagrangian comparison gives a concrete and cheap decision rule: begin a prefill round whenever its stage cost is less than the decode time currently waiting, a rule that can be evaluated in under 10 milliseconds per iteration.
  • The hybrid method brings the makespan to within 10 seconds of the theoretical lower bound, versus 21 seconds for the baseline, meaning it closes about half of the reducible gap the authors quantify through their bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reported gains rely on knowing each request's true output length when balancing clients and when computing the preemption comparison, a deployment that must predict output lengths may see smaller gains; a testable extension is to replace true lengths with quantile estimates and measure the utilization drop.
  • The same cost-comparison rule could be extended to heterogeneous GPU clusters by making the prefill and decode costs per device rather than global, a direction the paper does not explore.
  • The 100-case study suggests the method transfers across input and output length distributions drawn from the same moments, but distribution shift toward much longer outputs or multi-turn conversations is an untested boundary.
  • The theoretical lower bound could serve as a live headroom signal: when the current makespan is far above the bound, preemption is worth more; the paper does not close that feedback loop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies scheduling of LLM inference under the PD-competition mode, where prefill and decode stages alternate on a shared batch of clients. It proposes a mixed-integer programming formulation of the scheduling problem, then decomposes it into an offline bin-packing model for request-to-client assignment (with a claimed theoretical lower bound) and online heuristics for request sorting and for deciding when to preempt decoding to insert prefill. Experiments with LLaMA-65B on the GSM8K dataset report that the full hybrid method improves hardware utilization from 80.2% to 89.1% and reduces total inference time from 201.00s to 190.58s, and a 100-case study reports an average utilization improvement of 8.0 percentage points.

Significance. If the claims were sound, the paper would be a useful operations-research-style contribution to LLM serving: it makes the scheduling problem explicit, provides fast heuristics for a millisecond-scale decision loop, and validates on a real model and dataset with concrete Gantt charts. The paper deserves credit for profiling the hardware (T^p and T^d in Section V-A), reporting a 100-case experiment, and attempting to give a theoretical lower bound. However, the central formal model undercounts decode time by a factor of the batch size, the claimed lower bound is not a lower bound, the 'Lagrangian' derivation is not a Lagrangian method, and the experiments assume oracle knowledge of output token counts that the paper itself states are unknown. These issues are load-bearing for the paper's main contributions, and the empirical headline gains are therefore not established as stated.

major comments (5)
  1. [Eq. (8), Eq. (39), Section V-A] The decode-stage duration constraint is written per client, but the hardware profile in Section V-A says a decode stage with 200 clients each producing one token takes 200 × 0.21 + 29 = 71 ms, i.e. the stage length scales with the total number of decode tokens in the batch. Constraint (8), n^d_k ≥ T^d Σ_i N^d_i w_{i,j,k} for each j, only lower-bounds the stage by the busiest client, so with |J|=200 active clients it underestimates the decode stage length by up to a factor of about 200. The same error is repeated in Eq. (39) in the online iteration problem. Since decode time dominates the makespan (T^{d*} = 167s in Section V-C), the MIP model and the online cost comparison of Section IV-C do not describe the hardware they claim to optimize.
  2. [Section III-B, Section IV-B, Algorithm 1, Section V-C, Section VI] The problem statement in Section III-B says 'the output token number is unknown,' yet N^d_i is used throughout as a known parameter: in Eq. (8), in the offline objective through T_i in Eq. (28), in Algorithm 1's sorting key N^p_i+N^d_i and in remain_token(j), and in the decode cost C_d of Eq. (43). Section IV-B explicitly states 'We assume that the output length is predetermined,' and Section VI lists stochastic output length as future work. The experiments feed actual GSM8K output lengths into these decisions (Table III reports E[N^d_i] = 344.83), so the offline balancing and every preemption decision are exact only in an oracle setting. The reported utilization gain from 80.2% to 89.1% and makespan reduction from 201.00s to 190.58s are not shown to transfer to a real deployment where N^d_i must be predicted.
  3. [Eqs. (31)-(32), Section V-C] The claimed theoretical lower bound is not actually a lower bound. Equation (31) states tp* ≥ T^p_L · ceil(Σ_i N^p_i / N^cap_L), where T^p_L and N^cap_L are the time and capacity of the largest-capacity prefill level. The right-hand side is the time needed to prefill all tokens using only that level, which is a feasible schedule; a schedule using faster smaller-capacity levels can finish in less time, so the right-hand side is an upper bound on the optimal prefill makespan, not a lower bound. The fixed prefill overhead of 25 ms reported in Section V-A is also omitted. Consequently the value T_LB = 180s and the claimed 52.4% reduction in the 'primal-dual gap' are not justified.
  4. [Section IV-C, Eqs. (42)-(43)] The online iteration scheduler is described as using a Lagrangian method, but the derivation in Eqs. (42)-(43) is not a Lagrangian relaxation: there is no dual function, multiplier, or stationarity condition. Equation (42) sets ∂tmax/∂y_{k,l} = Σ_l T^p_l, a quantity that is independent of k and l and sums over all levels, and Eq. (43) sets ∂tmax/∂w_{i,j,k} = T^d Σ_i N^d_i, summing over all requests in I rather than the tokens of the specific request whose w variable is changed. As written, Cp is a constant and the rule 'if Cp ≥ C_d, continue decoding; otherwise preempt' is an arbitrary threshold, not a cost-efficiency comparison derived from the MIP. The claim that a Lagrangian method is applied is therefore unsupported.
  5. [Eqs. (12)-(15), Section III-C] There is no constraint linking the continuous decode-work variable w_{i,j,k} to the binary assignment variable d_{i,j,k}. Equations (14) and (15) only constrain the sums of w, while constraints (9)-(13) constrain d, so the model permits d_{i,j,k}=1 in stages where no decode work is assigned (w=0) and permits w_{i,j,k}>0 in stages where d_{i,j,k}=0. A feasible solution to the MIP therefore need not correspond to a real schedule of decode stages. This further undermines the claim that Eqs. (1)-(25) describe the inference scheduling problem.
minor comments (6)
  1. [Abstract, Section I] The phrases 'An Minimizing Makespan Bin Packing Problem' and 'an Minimizing Makespan Bin Packing model' should be corrected to 'A Minimizing Makespan Bin Packing Problem'.
  2. [Section V-C] The text says the hybrid method reduces the total inference time to 190.58s in one paragraph and to 190.08s in the lower-bound paragraph; please reconcile the two numbers.
  3. [Table III, Section V-A] The table reports a mean output length of 344.83 tokens with standard deviation 187.99 while the maximum output length is capped at 512 tokens; the paper should state how the cap affects the reported distribution and the resulting N^d_i values used in the experiments.
  4. [Section IV-A] The mapping between the offline model and the original constraints is unclear: the text says the offline method considers 'constraints (2), (12), and (15)–(16)', but Eq. (2) is the makespan constraint and Eqs. (15)-(16) concern w and p assignments that do not appear in the offline model (26)-(30); please clarify the intended relationship.
  5. [Figs. 6-9] The Gantt charts would be more informative if preemption events were marked and if all four charts used the same time axis scale, so that the reader could directly compare bubble locations across baseline, offline, online-only, and hybrid runs.
  6. [References] Reference [9] (FastServe) is cited as an arXiv preprint from 2023, but the paper appeared at MLSys 2024; the citation should be updated to the published version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the scheduling models use measured hardware parameters and explicit known-workload assumptions; the oracle output-token assumption is a correctness concern, not a circular derivation.

full rationale

The derivation chain is self-contained. The MIP model (1)-(25) is a direct formulation of the scheduling problem, not fitted to the evaluation data. The offline bin-packing model (26)-(30) uses a parameter T_i derived from measured hardware decode time and request output length; the paper explicitly states in Section IV-B that 'We assume that the output length is predetermined', while Section III-B acknowledges that 'the output token number is unknown'. The experiments feed actual GSM8K output lengths into the schedule, making the evaluation an oracle-knowledge study, but the reported utilization and makespan are simulation outcomes of executing the schedule rather than equalities forced by the cost model. The theoretical lower bound (31)-(32) is a relaxation of the original model and is not used to construct the solution. The only self-citations in the paper ([29], [30], [32]) are domain-background references in the literature review and are not load-bearing. No fitted parameter is renamed as a prediction: T_p and T_d are hardware measurements, and no uniqueness theorem or prior work by the authors is invoked to forbid alternatives. The principal weakness is the oracle output-token assumption in the online preemption rule, which affects real-world transferability but does not make the derivation circular.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on calibrated hardware timings, an assumption that output lengths are known when making scheduling decisions, and an ad hoc preemption rule. The paper introduces no new physical entities. The level system L and the 'Lagrangian' cost comparison are underspecified and mathematically unsupported.

free parameters (5)
  • T_p (prefill time per token) = 0.13 ms/token
    Estimated from over 400 hardware measurement groups in Section V-A; used in the MIP model and the online Cp/Cd preemption rule.
  • T_d (decode time per token) = 0.21 ms/token
    Estimated from hardware measurements in Section V-A; used in decode-time constraints and the online cost comparison.
  • prefill fixed overhead = 25 ms
    Fitted constant in the linear prefill time model in Section V-A; appears in simulations but not in the MIP equations.
  • decode fixed overhead = 29 ms
    Fitted constant in the linear decode time model in Section V-A; appears in simulations but not in the MIP equations.
  • prefill level set L and times T_p_l = not specified
    Eqs. (5)-(7) and Eq. (42) rely on a level system with times T_p_l, but the paper never defines the levels or their values, and Cp is computed as the sum of all T_p_l without data.
assumptions (4)
  • domain assumption Output token counts N_d_i are known at scheduling time
    Section III-B states the output token number is unknown, but the MIP model, offline bin packing, and online remaining-token calculations treat N_d_i as a known parameter; experiments use actual output lengths.
  • domain assumption PD Competition: prefill and decode stages alternate globally, and decode can be preempted only at bin boundaries
    Section III-A selects PD Competition as the hardware model; this is an architectural choice, not derived from first principles.
  • domain assumption Decode time for a batch is linear in the total number of tokens in the batch
    Section V-A fits a linear model with T_d and a fixed overhead; however Eq. (8) does not sum over clients, so the MIP is inconsistent with this assumption.
  • ad hoc to paper Comparing marginal costs Cp and Cd yields a good preemption decision
    Section IV-C asserts this heuristic without proof; the 'Lagrangian' derivation of the marginal costs is mathematically incorrect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization." pith.science (2026). https://pith.science/paper/27ZR6DST

@misc{pith2026250215763,
  author       = {Pith},
  title        = {Pith review of: Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27ZR6DST}},
  note         = {Machine review of arXiv:2502.15763}
}
read the original abstract

With the development of large language models (LLMs), it has become increasingly important to optimize hardware usage and improve throughput. In this paper, we study the inference optimization of the serving system that deploys LLMs. To optimize system throughput and maximize hardware utilization, we formulate the inference optimization problem as a mixed-integer programming (MIP) model and propose a hybrid offline-online method as solution. The offline method improves large-scale inference systems by introducing a Minimizing Makespan Bin Packing Problem. We further provide a theoretical lower bound computation method. Then, we propose an online sorting and preemptive scheduling method to better utilize hardware. In the online iteration scheduling process, a Lagrangian method is applied to evaluate the cost efficiency of inserting prefill stages versus decode stages at each iteration and dynamically determine when to preempt decoding tasks and insert prefill tasks. Experiments using real-world data from the LLaMA-65B model and the GSM8K dataset demonstrate that system utilization improves from 80.2% to 89.1%, and the total inference time decreases from 201.00 to 190.58 seconds. A 100-cases study shows that our method consistently outperforms the baseline method and improves the utilization rate by 8.0% on average. Finally, we discuss potential future extensions, including stochastic modeling, reinforcement learning-based schedulers, and dynamic decision-making strategies for system throughput and hardware utilization.

Figures

Figures reproduced from arXiv: 2502.15763 by the authors.

Figure 1
Figure 1. Illustration of LLM inference service system [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of LLM inference and are limited to solve the inference optimization problem theoretically. In the domain of operations research, scheduling is a well￾established and extensively utilized approach. For instance, of￾fline scheduling methods are applied in manufacturing systems [29], healthcare systems [30], and operations management systems [31]. To address real-time decision-making or multi￾stage stocha… view at source ↗
Figure 3
Figure 3. Illustration of PD Competition decode stage. Requests in prefill phase can be served only when the process is in prefill stage, and the same requirement is applied to the decode phase and stage. A client may be idle as either the prefill phase or decode phase is completed but the process still stays in the same stage, causing compromised utilization of computing resources. A single LLM inference in practice typicall… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Online scheduling Without considering uncertainty, the offline-online inference scheduling can be formulated as a deterministic equivalence, which is a form of an MIP model as follows. min t max (1) s.t. t max ≥ t s,d k + n d k , for k ∈ Kd , (2) t s,p k −  t s,d k−1 …
Figure 5
Figure 5. Figure 5: Illustration of Solution Method availability. By effectively prioritizing requests, the system can minimize waiting times and maximize throughput under dynamic operational conditions. This method emphasizes the implementation of the relaxed solutions provided by job as…
Figure 6
Figure 6. Figure 6: Result Gantt: Baseline of clients can be estimated as 0.21 milliseconds per token, with an additional fixed overhead of 29 milliseconds. For instance, when processing a parallel batched decode stage involving 200 clients, where each client produces one token per round,…
Figure 7
Figure 7. Figure 7: Result Gantt: Offline Request Scheduling [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 9
Figure 9. Figure 9: Result Gantt: Offline+Online Scheduling [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Utilization rate with 100 cases [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Generate speed with 100 cases VI. CONCLUSION AND FUTURE WORK In this paper, we study the inference optimization problem in the service system when deploying LLMs. To enhance the system throughput and better utilize the hardware, we formulate an MIP model to describe t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 10 canonical work pages

  1. [1]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles , 2023, pp. 611–626

    2023

  2. [2]

    Orca: A distributed serving system for {Transformer-Based} generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for {Transformer-Based} generative models,” in 16th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 22) , 2022, pp. 521–538

    2022

  3. [3]

    NVIDIA Announces Financial Results for Fourth Quarter and Fiscal 2024,

    NVIDIA, “NVIDIA Announces Financial Results for Fourth Quarter and Fiscal 2024,” https://investor.nvidia.com/news/press-release- details/2024/NVIDIA-Announces-Financial-Results-for-Fourth-Quarter- and-Fiscal-2024/, Feb. 2024, [Online; accessed 09-01-2025]

    work page 2024

  4. [4]

    Taming throughput-latency tradeoff in llm inference with sarathi-serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” arXiv preprint arXiv:2403.02310 , 2024

    arXiv 2024

  5. [5]

    Llumnix: Dynamic scheduling for large language model serving,

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y . Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” arXiv preprint arXiv:2406.03243, 2024

    arXiv 2024

  6. [6]

    {InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache manage- ment,

    W. Lee, J. Lee, J. Seo, and J. Sim, “ {InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache manage- ment,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , 2024, pp. 155–172

    2024

  7. [7]

    {dLoRA}: Dynamically orchestrating requests and adapters for {LoRA}{LLM} serving,

    B. Wu, R. Zhu, Z. Zhang, P. Sun, X. Liu, and X. Jin, “ {dLoRA}: Dynamically orchestrating requests and adapters for {LoRA}{LLM} serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , 2024, pp. 911–927

    2024

  8. [8]

    Fairness in serving large language models,

    Y . Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in serving large language models,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 965–988

    2024

Show all 34 references
  1. [9]

    Fast distributed inference serving for large language models,

    B. Wu, Y . Zhong, Z. Zhang, S. Liu, F. Liu, Y . Sun, G. Huang, X. Liu, and X. Jin, “Fast distributed inference serving for large language models,” arXiv preprint arXiv:2305.05920 , 2023

    2023 arXiv

  2. [10]

    {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “ {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210. 13

    2024

  3. [11]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference,

    B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2...

    2018

  4. [12]

    Generating long sequences with sparse transformers,

    R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509 , 2019

    1904 arXiv

  5. [13]

    Sparsellm: Towards global pruning of pre-trained language models,

    G. Bai, Y . Li, C. Ling, K. Kim, and L. Zhao, “Sparsellm: Towards global pruning of pre-trained language models,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  6. [14]

    Distilling the knowledge in a neural network,

    G. Hinton, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  7. [15]

    Fast transformer decoding: One write-head is all you need,

    N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150 , 2019

    1911 arXiv

  8. [16]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints,

    J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebr ´on, and S. Sanghai, “Gqa: Training generalized multi-query transformer models from multi-head checkpoints,” arXiv preprint arXiv:2305.13245 , 2023

    2023 arXiv

  9. [17]

    Efficient large-scale language model training on gpu clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro et al. , “Efficient large-scale language model training on gpu clusters using megatron-lm,” in Proceedings of the International Conference fo...

    2021

  10. [18]

    Gpipe: Easy scaling with micro-batch pipeline parallelism,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wuet al., “Gpipe: Easy scaling with micro-batch pipeline parallelism,” proceeding of Computer Science¿ Computer Vision and Pattern Recognition, 2019

    2019

  11. [19]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019

    1909 arXiv

  12. [20]

    Reducing activation recomputation in large transformer models,

    V . A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” Proceedings of Machine Learning and Systems, vol. 5, pp. 341–353, 2023

    2023

  13. [21]

    Ring attention with blockwise transformers for near-infinite context,

    H. Liu, M. Zaharia, and P. Abbeel, “Ring attention with blockwise transformers for near-infinite context,” arXiv preprint arXiv:2310.01889, 2023

    2023 arXiv

  14. [22]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Advances in Neural Information Processing Systems , vol. 35, pp. 16 344–16 359, 2022

    2022

  15. [23]

    Fast inference from transform- ers via speculative decoding,

    Y . Leviathan, M. Kalman, and Y . Matias, “Fast inference from transform- ers via speculative decoding,” in International Conference on Machine Learning. PMLR, 2023, pp. 19 274–19 286

    2023

  16. [24]

    Accelerating large language model decoding with speculative sam- pling,

    C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sam- pling,” arXiv preprint arXiv:2302.01318 , 2023

    2023 arXiv

  17. [25]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 118–132

    2024

  18. [26]

    Inference without interference: Disaggre- gate llm inference for mixed downstream workloads,

    C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y . Bao et al. , “Inference without interference: Disaggre- gate llm inference for mixed downstream workloads,” arXiv preprint arXiv:2401.11181, 2024

    2024 arXiv

  19. [27]

    A survey on efficient inference for large language models,

    Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y . Lou, L. Wang, Z. Yuan, X. Li et al., “A survey on efficient inference for large language models,” arXiv preprint arXiv:2404.14294 , 2024

    2024 arXiv

  20. [28]

    Moon- cake: A kvcache-centric disaggregated architecture for llm serving,

    R. Qin, Z. Li, W. He, M. Zhang, Y . Wu, W. Zheng, and X. Xu, “Moon- cake: A kvcache-centric disaggregated architecture for llm serving,” arXiv preprint arXiv:2407.00079 , 2024

    2024 arXiv

  21. [29]

    Print surface thermal modeling and layer time control for large-scale additive manu- facturing,

    F. Wang, S. Fathizadan, F. Ju, K. Rowe, and N. Hofmann, “Print surface thermal modeling and layer time control for large-scale additive manu- facturing,” IEEE Transactions on automation science and engineering , vol. 18, no. 1, pp. 244–254, 2020

    2020

  22. [30]

    Surgery scheduling under case cancellation and surgery duration uncertainty,

    B. Pang, X. Xie, Y . Song, and L. Luo, “Surgery scheduling under case cancellation and surgery duration uncertainty,” IEEE Transactions on Automation Science and Engineering , vol. 16, no. 1, pp. 74–86, 2018

    2018

  23. [31]

    Online appointment sequencing and scheduling,

    S. A. Erdogan, A. Gose, and B. T. Denton, “Online appointment sequencing and scheduling,” IIE Transactions, vol. 47, no. 11, pp. 1267– 1286, 2015

    2015

  24. [32]

    A dynamic sequential decision- making model on mri real-time scheduling with simulation-based opti- mization,

    B. Pang, X. Xie, F. Ju, and J. Pipe, “A dynamic sequential decision- making model on mri real-time scheduling with simulation-based opti- mization,” Health Care Management Science , vol. 25, no. 3, pp. 426– 440, 2022

    2022

  25. [33]

    Online scheduling of ordered flow shops,

    K. Lee, F. Zheng, and M. L. Pinedo, “Online scheduling of ordered flow shops,” European Journal of Operational Research , vol. 272, no. 1, pp. 50–60, 2019

    2019

  26. [34]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168 , 2021. Bowen Pang received his Bachelor’s degree and Ph.D. degree from the D...

    2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.