Pith. sign in

REVIEW 2 major objections 6 minor 47 references

Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A DRL-based router for edge-hosted LLMs claims to maximize long-term QoS by encoding the global state as a heterogeneous graph and penalizing routing decisions that push running requests past their latency requirement, improving average…

desk verdict A plausible DRL-based router for heterogeneous edge LLMs, but the reward estimator's use of ground-truth output lengths and thin evaluation keep the performance claims unverified. read the letter →

arxiv 2508.00234 v1 pith:DG737GQH submitted 2025-08-01 cs.NI cs.AIcs.DCcs.MA

Jin Yang , Qiong Wu , Zhiying Feng , Zhi Zhou , Deke Guo , Xu Chen This is my paper
classification cs.NIcs.AIcs.DCcs.MA
keywords largelanguagemodelsedgecomputingexpertroutingdeepreinforcementlearningqualityofservicegraphattentionnetworkrequestinterferencedynamicworkloads
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that routing user requests to the right edge-hosted large language model (LLM) can be cast as a long-term quality-of-service (QoS) optimization, and that a deep reinforcement learning router with two new components solves it well enough to raise average QoS by 35.78% over a baseline RL router under Poisson workloads and by at least 33.47% under real-world workloads. The paper defines QoS per request as the product of a response-quality score and an indicator that the average latency per token stays under a requirement $L$. To decide where to send each request, the router encodes the changing global state (arrived, waiting, and running requests plus edge experts) through a heterogeneous graph attention network, then uses an action impact estimator inside the reward to penalize decisions that push any already-running request past the latency requirement. If the claim holds, edge deployments of several heterogeneous LLMs could sustain both response quality and latency under dynamic load instead of overloading one expert or ignoring quality differences.

What carries the argument

Three components carry the argument. (1) Dynamic state abstraction: a heterogeneous graph attention network (HAN) maps the raw global state (the arrived request node, $N$ edge-expert nodes, and per-expert running and waiting request nodes) into a compact embedding of the arrived request, giving the policy a fixed-size state despite queues of variable length. (2) Action impact estimator: Eq. (15) estimates, for each request already running on the chosen expert, the increase in average latency per token caused by the incoming request's prefill phase and added decode tokens, using profile-derived linear coefficients for prefill and decode cost. (3) QoS-aware reward: Eq. (16) rewards completed requests that meet the latency requirement and subtracts the estimated QoS loss of requests whose estimated latency would cross the requirement. These components are trained together with an entropy-regularized actor-critic method so the policy maximizes discounted long-term reward while exploring.

What would settle it

Look at the training loop used for the results in Section VI: if the reward computation receives each request's final output length from the completed response rather than from the length predictor, the policy has been trained with information deployment cannot provide. Re-training with only predicted lengths inside Eq. (15) and re-running the same comparisons would settle whether the 35.78% gain survives; if it shrinks toward the baseline gap, the central claim fails in practice.

Watch

Extended reading notes

Core claim

The paper's central claim is that long-term QoS in a multi-expert edge LLM system is maximized by a DRL router whose policy takes a compact graph embedding of the dynamic global state and is rewarded for the QoS of completed requests minus the estimated harm the current action will cause to requests already in service. The router formulates the problem as the non-convex optimization in Eq. (2) over routing decisions subject to each expert's GPU-memory constraint, and solves it with an entropy-regularized actor-critic algorithm. The state embedding is built by a heterogeneous graph attention network (HAN) that connects each arrived request to every expert and each running or waiting request to its assigned expert, so the actor sees request-level detail such as predicted generation score, predicted output length, current GPU utilization, and current per-token latency. The action impact estimator in Eq. (15) quantifies how adding a request will raise the per-token latency of already-running requests through prefill blocking and extra decode load; this quantity enters the reward as a penalty when the estimated latency crosses the requirement $L$. Simulation results with up to twelve 7B-parameter experts show the trained router outperforms a generation-score-predicting router, round-robin, shortest-queue-first, and a baseline RL router, with the largest reported improvement being 35.78% in average QoS over baseline RL and a consistent reduction in latency per token.

Load-bearing premise

The load-bearing premise is that the action impact estimator in Eq. (15) can be computed at routing time from information then available; as written it uses the final output lengths of incoming and already-running requests, which are unknown when the decision is made, and the paper does not state that predicted lengths replace them.

Editorial extensions

If this is right

  • Deployed edge routers can make decisions in about 5 ms with a 134M-parameter model, against multi-second LLM generation times, so the added routing cost is a small fraction of end-to-end latency.
  • Under Poisson workloads the router raises average QoS by 35.78% and cuts average latency per token by 5.45% relative to baseline RL.
  • Under real-world volatile workloads it keeps at least a 33.47% QoS improvement and a 3.35% latency-per-token reduction, and it outperforms baselines throughout a multi-hour run.
  • Performance holds as the number of edge experts grows from 3 to 12, under arrival rates from $\lambda=4$ to $\lambda=6$, and under stricter latency requirements, with at least an 18.31% QoS improvement in the strictest tested setting.
  • Ablations attribute a 17.27% QoS gain to dynamic state abstraction alone and an additional 22.37% to the QoS-aware reward.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The estimator in Eq. (15) is written with final output lengths of both the incoming and running requests; a deployed router would have to substitute the model's predicted lengths, and the paper does not explicitly say it does. The reported gains are therefore conditioned on how well live predictions approximate those lengths.
  • The quality score in the QoS definition comes from comparing the generated response with a ground-truth response, which is unavailable at routing time; a real deployment would need a proxy quality model, and routing could shift if that proxy differs from the paper's scoring metric.
  • The same pattern (graph-encoded live state plus an interference-aware impact penalty) is transferable to other settings where concurrent requests contend for accelerators, such as mixture-of-experts inference or multi-tenant GPU serving.
  • A direct test of the estimator's deployment realism would be to rerun the evaluation with predicted lengths fed into Eq. (15) instead of ground-truth values; the paper's ablation with zero predictive features does not settle this because the reward still uses final lengths.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper addresses QoS-aware routing of LLM inference requests to heterogeneous edge LLM experts. It formulates a long-term QoS maximization problem in which each request's QoS is the product of its BERTScore and an indicator that its average latency per token does not exceed a requirement L. The proposed solution is a Soft Actor-Critic (SAC) router with a HAN-based dynamic state abstraction, DistilBERT-based predictors for generation score and output length, an action impact estimator, and a QoS-aware reward. The evaluation uses Poisson workloads and BurstGPT real-world traces, reporting a 35.78% improvement in average QoS over Baseline RL and at least 33.47% over other baselines, along with ablations attributing gains to the dynamic state abstraction and the QoS-aware reward.

Significance. If the results hold, the paper makes a useful contribution to edge LLM serving by jointly addressing expert heterogeneity, request interference, and dynamic workload conditions. The paper's strengths include a clearly formulated QoS objective, a lightweight routing architecture with small predictors and a small HAN, ablations that isolate the contributions of the two main design components, and evaluation under both synthetic and real-world workloads. However, the action impact estimator as written depends on final output lengths that are unknown at routing time, and the linear latency model behind it is not validated; both are central to the reported QoS gains. The significance of the claims is therefore contingent on clarifying and validating these points.

major comments (2)
  1. [Section V-C1, Eq. (15)] The action impact estimator is written in terms of d_i and d_j, the final output lengths of a running request and of the incoming request, respectively. Both are unknown at decision time t_j: d_j is revealed only when the incoming request completes, and d_i is revealed only when request i completes. The paper introduces predicted lengths \hat d_j and bucketized predictions in Section V-B, but Eq. (15) uses true lengths and no sentence states that predicted lengths are substituted. If the training environment used ground-truth lengths, then the QoS-aware reward in Eq. (16) provided oracle information that a deployed router would not possess, and the claimed 35.78% / 33.47% QoS improvements, as well as the 22.37% ablation gain attributed to the QoS-aware reward, would not necessarily transfer to deployment. Please rewrite Eq. (15) with \hat d_i and \hat d_j, state the substitution explicitly, and evaluate the trained policy using predicted lengths (for example, with the reported top-1 and top-3 bucket accuracies).
  2. [Section V-C1, Eqs. (13)-(15)] The linear latency model — prefill latency proportional to p_j and decode latency proportional to the sum of current tokens in the running queue — is load-bearing for the penalty term in Eq. (16), but the paper does not validate this model against measured latency. Figure 5 shows the measured average latency per token over time for a single request under different arrival rates, not the quality of the fit of Eqs. (13)-(14). Please report how k1,n and k2,n are profiled for each expert, the resulting fit error, and a sensitivity analysis showing how QoS results change if the fitted slopes are inaccurate.
minor comments (6)
  1. [Section V-C2, Eq. (16)] The first term sums \phi_i over i in Qrunning_{n,t_j} and multiplies by w_{n,i,t_j}, which is described as an indicator of completion at time t_j. Since \phi_i is defined in Eq. (1) only for completed requests, please clarify whether the sum is over running requests with a completion event at that slot or over completed requests only.
  2. [Section V-C1] The definition \hat l_{i,t_j} = l_{i,t_j} + l^+_{i,t_j} appears in prose but without an equation number and without a precise statement of which quantities are predictions; numbering this equation would improve readability.
  3. [Section VI-B.b] The text says "As illustrated in Figure VI-A" but the figure label is Figure 8; please correct the cross-reference.
  4. [Section V-B] The paper states that 10 buckets are used for generation scores and output lengths but does not give the bucket boundaries; please specify the ranges used for both quantities.
  5. [Table I and Section V-B] Table I defines \hat d_j for the incoming request but not \hat d_i for running requests; adding this notation would make the proposed substitution in Eq. (15) unambiguous.
  6. [Section VI-B.b] The reported improvements (33.47% QoS, 3.35% latency) appear to be single-run results; because SAC training is stochastic, please report the number of seeds and confidence intervals for the main comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reward is a standard task-aligned surrogate, not a re-statement of the conclusion, and the cited prior work is not load-bearing.

full rationale

The paper's derivation chain is self-contained. The optimization objective (Eq. 2) maximizes cumulative QoS phi_i = s_i * I[l_i <= L] (Eq. 1). The DRL reward (Eq. 16) is a sum of completed QoS plus a penalty for estimated latency violations, which is a standard surrogate aligned with the objective rather than the objective itself; the penalty term depends on the profiled linear estimator Eq. (15) and is not definitionally equal to Eq. (2). The state representation (Eqs. 6-12) encodes predicted generation scores and output lengths from trained DistilBERT predictors together with current queue and memory features; no fitted parameter is relabeled as a prediction of QoS. The only self-citation of the author team is Ref. [6] for the general promise of edge computing, which is contextual and not load-bearing. The strongest caveat is that Eq. (15) uses final output lengths d_i and d_j that are unknown at routing time, and the paper does not explicitly substitute the predictors \hat d_i and \hat d_j; this is an evaluability or deployment-leakage concern about whether the trained reward is computable online, not a circularity concern, because the reward is not equivalent by construction to its inputs and the QoS metric is not used to define the fitted parameters. The reported QoS gains are measured against external baselines in a simulated environment, so the central claim has independent content.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The system is empirical, so the ledger is dominated by fitted predictors and fitted latency slopes rather than mathematical axioms. The main hidden load-bearing choices are the linear latency model (Eqs. 13-14), the availability of ground-truth outputs for QoS, and whether Eq. (15) uses predicted or actual output lengths.

free parameters (6)
  • k1_n (prefill latency gradient) = not reported; determined by profiling per edge expert
    Used in Eq. (13) to estimate prefill latency; fitted to each edge expert's profiling data.
  • k2_n (decode latency gradient) = not reported; determined by profiling per edge expert
    Used in Eqs. (14)-(15) for decode latency and interference estimate; fitted to profiling data.
  • DistilBERT generation-score predictor = fine-tuned; top-1 accuracy 63.39%
    Predicts generation score buckets used in state features; trained on mix-instruct labels.
  • DistilBERT output-length predictor = fine-tuned; top-1 accuracy 72.97%
    Predicts output length buckets used in state features and possibly Eq. (15).
  • Bucket count = 10
    Chosen by hand for score and length bucketing; affects the quantization of predicted features.
  • SAC temperature alpha = not reported
    Entropy coefficient in Eq. (5); a tunable hyperparameter with no schedule or value given.
assumptions (5)
  • domain assumption Iteration-level scheduling processes all running requests concurrently, with prefill blocking decode.
    Basis of Figure 3 and the interference model in Eqs. (13)-(15); matches Orca/vLLM-style continuous batching but is an idealization.
  • domain assumption QoS phi_j = s_j times I[l_j <= L], with s_j = BERTScore against ground truth y_j.
    Eq. (1) defines QoS using ground-truth text, which is available only in the training/evaluation dataset, not in live deployment.
  • ad hoc to paper Prefill and decode latencies grow linearly in token counts with fixed per-expert slopes k1,n and k2,n.
    Eqs. (13)-(14) assert linearity without an independent validation of the linear model.
  • ad hoc to paper Final output lengths d_i and d_j are available or accurately predicted when computing Eq. (15).
    Eq. (15) uses unhatted d_i and d_j at routing time; the paper does not state whether predicted lengths are substituted.
  • domain assumption SAC converges to a high-quality policy within 1 million training steps.
    Standard DRL assumption; no convergence guarantee for this specific MDP is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts." pith.science (2026). https://pith.science/paper/DG737GQH

@misc{pith2026250800234,
  author       = {Pith},
  title        = {Pith review of: Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DG737GQH}},
  note         = {Machine review of arXiv:2508.00234}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities, leading to a significant increase in user demand for LLM services. However, cloud-based LLM services often suffer from high latency, unstable responsiveness, and privacy concerns. Therefore, multiple LLMs are usually deployed at the network edge to boost real-time responsiveness and protect data privacy, particularly for many emerging smart mobile and IoT applications. Given the varying response quality and latency of LLM services, a critical issue is how to route user requests from mobile and IoT devices to an appropriate LLM service (i.e., edge LLM expert) to ensure acceptable quality-of-service (QoS). Existing routing algorithms fail to simultaneously address the heterogeneity of LLM services, the interference among requests, and the dynamic workloads necessary for maintaining long-term stable QoS. To meet these challenges, in this paper we propose a novel deep reinforcement learning (DRL)-based QoS-aware LLM routing framework for sustained high-quality LLM services. Due to the dynamic nature of the global state, we propose a dynamic state abstraction technique to compactly represent global state features with a heterogeneous graph attention network (HAN). Additionally, we introduce an action impact estimator and a tailored reward function to guide the DRL agent in maximizing QoS and preventing latency violations. Extensive experiments on both Poisson and real-world workloads demonstrate that our proposed algorithm significantly improves average QoS and computing resource efficiency compared to existing baselines.

Figures

Figures reproduced from arXiv: 2508.00234 by the authors.

Figure 1
Figure 1. LLM routing at the edge with multiple experts. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An example of the LLM services. Given a user request, different [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An example of iteration-level scheduling for LLM inference. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Average latency per token over times for the first request with various [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Overview of our DRL-based QoS-aware LLM routing algorithm. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Average QoS and average latency per token comparison with [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Illustration of dynamic intensity under real-world LLM workloads. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Average QoS and average latency per token comparison with [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Comparison of end-to-end latency with N=6 edge experts under Poisson workloads with λ=5. N=3 N=6 N=12 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Average QoS BR RR SQF Baseline RL QoS-aware RL (a) Average QoS N=3 N=6 N=12 0 10 20 30 40 50 Average Latency per Token (ms) BR…
Figure 11
Figure 11. Figure 11: Comparison of average QoS and average latency per token across [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Comparison of average QoS and average latency per token with [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 14
Figure 14. Figure 14: Average QoS for the long-running process with [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 15
Figure 15. Figure 15: GPU usage for the long-running process with [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]
Figure 16
Figure 16. Figure 16: Training process with N=6 edge experts under Poisson workloads with λ=5. Baseline RL Baseline RL + DSA QoS-aware RL 0.0 0.1 0.2 0.3 0.4 Average QoS (a) Average QoS Baseline RL Baseline RL + DSA QoS-aware RL 0.02 0.04 0.06 0.08 0.10 0.12 Average Latency per Token (s) (…
Figure 17
Figure 17. Figure 17: Ablation study of dynamic state abstraction and QoS-aware reward [PITH_FULL_IMAGE:figures/full_fig_p013_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 28 canonical work pages

  1. [1]

    AIoT smart home via autonomous LLM agents,

    D. Rivkin, F. Hogan, A. Feriani, A. Konar, A. Sigal, X. Liu, and G. Dudek, “AIoT smart home via autonomous LLM agents,” IEEE Internet of Things Journal , vol. 1, pp. 1–1, 2024

    work page 2024

  2. [2]

    Large language models for human- ai co-creation of robotic dance performances,

    A. De Filippo and M. Milano, “Large language models for human- ai co-creation of robotic dance performances,” in Proceedings of the 33rd International Joint Conference on Artificial Intelligence , 2024, pp. 7627–7635

    work page 2024

  3. [3]

    EdgeFM: Leveraging foundation model for open-set learning on the edge,

    B. Yang, L. He, N. Ling, Z. Yan, G. Xing, X. Shuai, X. Ren, and X. Jiang, “EdgeFM: Leveraging foundation model for open-set learning on the edge,” in Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems , 2023, pp. 111–124

    work page 2023

  4. [4]

    WDMoE: Wireless Distributed Large Language Models with Mixture of Experts

    N. Xue, Y . Sun, Z. Chen, M. Tao, X. Xu, L. Qian, S. Cui, and P. Zhang, “WDMoE: Wireless distributed large language models with mixture of experts,” arXiv preprint arXiv:2405.03131 , 2024

    work page Pith review arXiv 2024

  5. [5]

    On protecting the data privacy of large language models (LLMs): A survey,

    B. Yan, K. Li, M. Xu, Y . Dong, Y . Zhang, Z. Ren, and X. Cheng, “On protecting the data privacy of large language models (LLMs): A survey,” arXiv preprint arXiv:2403.05156 , 2024. 14

    arXiv 2024

  6. [6]

    Edge intelligence: Paving the last mile of artificial intelligence with edge computing,

    Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang, “Edge intelligence: Paving the last mile of artificial intelligence with edge computing,” Proceedings of the IEEE , vol. 107, no. 8, pp. 1738–1762, 2019

    work page 2019

  7. [7]

    Enabling AI-Generated Content (AIGC) Services in Wireless Edge Networks

    H. Du, Z. Li, D. Niyato, J. Kang, Z. Xiong, D. I. Kim et al., “Enabling AI-generated content (AIGC) services in wireless edge networks,” arXiv preprint arXiv:2301.03220, 2023

    work page Pith review arXiv 2023

  8. [8]

    Toward Scalable Generative AI via Mixture of Experts in Mobile Edge Networks

    J. Wang, H. Du, D. Niyato, J. Kang, Z. Xiong, D. I. Kim, and K. B. Letaief, “Toward scalable generative ai via mixture of experts in mobile edge networks,” arXiv preprint arXiv:2402.06942 , 2024

    work page Pith review arXiv 2024

Show all 47 references
  1. [9]

    LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion,

    D. Jiang, X. Ren, and B. Y . Lin, “LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion,” in Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics, 2023, pp. 14 165–14 178

    2023

  2. [10]

    Intelligent router for LLM workloads: Improving performance through workload-aware schedul- ing,

    K. Jain, A. Parayil, A. Mallick, E. Choukse, X. Qin, J. Zhang, ´I. Goiri, R. Wang, C. Bansal, V . R ¨uhle et al. , “Intelligent router for LLM workloads: Improving performance through workload-aware schedul- ing,” arXiv preprint arXiv:2408.13510 , 2024

    2024 arXiv

  3. [11]

    Orca: A distributed serving system for Transformer-based generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for Transformer-based generative models,” in Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation , 2022, pp. 521–538

    2022

  4. [12]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles , 2023, pp. 611–626

    2023

  5. [13]

    PolyRouter: A multi-LLM querying system,

    D. Stripelis, Z. Hu, J. Zhang, Z. Xu, A. Shah, H. Jin, Y . Yao, S. Avestimehr, and C. He, “PolyRouter: A multi-LLM querying system,” arXiv preprint arXiv:2408.12320 , 2024

    2024 arXiv

  6. [14]

    Routing to the expert: Efficient reward-guided ensemble of large language models,

    K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, and J. Zhou, “Routing to the expert: Efficient reward-guided ensemble of large language models,” arXiv preprint arXiv:2311.08692 , 2023

    2023 arXiv

  7. [15]

    Hybrid LLM: Cost-efficient and quality-aware query routing,

    D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V . Ruhle, L. V . Lakshmanan, and A. H. Awadallah, “Hybrid LLM: Cost-efficient and quality-aware query routing,” arXiv preprint arXiv:2404.14618 , 2024

    2024 arXiv

  8. [16]

    RouteLLM: Learning to route LLMs with preference data,

    I. Ong, A. Almahairi, V . Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica, “RouteLLM: Learning to route LLMs with preference data,” arXiv preprint arXiv:2406.18665 , 2024

    2024 arXiv

  9. [17]

    Towards efficient and reliable LLM serving: A real- world workload study,

    Y . Wang, Y . Chen, Z. Li, Z. Tang, R. Guo, X. Wang, Q. Wang, A. C. Zhou, and X. Chu, “Towards efficient and reliable LLM serving: A real- world workload study,” arXiv preprint arXiv:2401.17644 , 2024

    2024 arXiv

  10. [18]

    Efficient interactive LLM serving with proxy model-based sequence length prediction,

    H. Qiu, W. Mao, A. Patke, S. Cui, S. Jha, C. Wang, H. Franke, Z. T. Kalbarczyk, T. Bas ¸ar, and R. K. Iyer, “Efficient interactive LLM serving with proxy model-based sequence length prediction,” arXiv preprint arXiv:2404.08509, 2024

    2024 arXiv

  11. [19]

    S3: Increasing gpu utiliza- tion during generative inference for higher throughput,

    Y . Jin, C.-F. Wu, D. Brooks, and G.-Y . Wei, “S3: Increasing gpu utiliza- tion during generative inference for higher throughput,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023, pp. 18 015–18 027

    2023

  12. [20]

    FlexGen: High-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “FlexGen: High-throughput generative inference of large language models with a single gpu,” in Proceedings of the 40th International Conference on Machine Learning , 2023, pp...

    2023

  13. [21]

    FlashAttention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “FlashAttention: Fast and memory-efficient exact attention with io-awareness,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022, pp. 16 344–16 359

    2022

  14. [22]

    HeteGen: Efficient heterogeneous parallel inference for large language models on resource-constrained devices,

    Z. XUANLEI, B. Jia, H. Zhou, Z. Liu, S. Cheng, and Y . You, “HeteGen: Efficient heterogeneous parallel inference for large language models on resource-constrained devices,” Proceedings of Machine Learning and Systems, vol. 6, pp. 162–172, 2024

    2024

  15. [23]

    ExeGPT: Constraint-aware resource scheduling for LLM inference,

    H. Oh, K. Kim, J. Kim, S. Kim, J. Lee, D.-s. Chang, and J. Seo, “ExeGPT: Constraint-aware resource scheduling for LLM inference,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , 2024, pp. 369–384

    2024

  16. [24]

    AttentionStore: Cost-effective attention reuse across multi- turn conversations in large language model serving,

    B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo, “AttentionStore: Cost-effective attention reuse across multi- turn conversations in large language model serving,” arXiv preprint arXiv:2403.19708, 2024

    2024 arXiv

  17. [25]

    Splitwise: Efficient generative LLM inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” in Proceedings of the ACM/IEEE 51st Annual International Symposium on Computer Architecture , 2024, pp. 118–132

    2024

  18. [26]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” arXiv preprint arXiv:2401.09670, 2024

    2024 arXiv

  19. [27]

    Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models,

    J. Lu, Z. Pang, M. Xiao, Y . Zhu, R. Xia, and J. Zhang, “Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models,” arXiv preprint arXiv:2407.06089 , 2024

    2024 arXiv

  20. [28]

    Large language model routing with benchmark datasets,

    T. Shnitzer, A. Ou, M. Silva, K. Soule, Y . Sun, J. Solomon, N. Thomp- son, and M. Yurochkin, “Large language model routing with benchmark datasets,” arXiv preprint arXiv:2309.15789 , 2023

    2023 arXiv

  21. [29]

    Octopus v4: Graph of language models,

    W. Chen and Z. Li, “Octopus v4: Graph of language models,” arXiv preprint arXiv:2404.19296, 2024

    2024 arXiv

  22. [30]

    GraphRouter: A graph-based router for LLM selections,

    T. Feng, Y . Shen, and J. You, “GraphRouter: A graph-based router for LLM selections,” arXiv preprint arXiv:2410.03834 , 2024

    2024 arXiv

  23. [31]

    Eagle: Efficient training-free router for multi-LLM inference,

    Z. Zhao, S. Jin, and Z. M. Mao, “Eagle: Efficient training-free router for multi-LLM inference,” arXiv preprint arXiv:2409.15518 , 2024

    2024 arXiv

  24. [32]

    RouterBench: A benchmark for multi- LLM routing system,

    Q. J. Hu, J. Bieker, X. Li, N. Jiang, B. Keigwin, G. Ranganath, K. Keutzer, and S. K. Upadhyay, “RouterBench: A benchmark for multi- LLM routing system,” arXiv preprint arXiv:2403.12031 , 2024

    2024 arXiv

  25. [33]

    Reinforcement learning in dynamic task scheduling: A review,

    C. Shyalika, T. Silva, and A. Karunananda, “Reinforcement learning in dynamic task scheduling: A review,”SN Computer Science, vol. 1, no. 6, pp. 306–306, 2020

    2020

  26. [34]

    Collaborative learning-based scheduling for kubernetes-oriented edge-cloud network,

    S. Shen, Y . Han, X. Wang, S. Wang, and V . C. Leung, “Collaborative learning-based scheduling for kubernetes-oriented edge-cloud network,” IEEE/ACM Transactions on Networking, vol. 31, no. 6, pp. 2950–2964, 2023

    2023

  27. [35]

    Clipper: A low-latency online prediction serving system,

    D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A low-latency online prediction serving system,” in Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation , 2017, pp. 613–627

    2017

  28. [36]

    Tapfinger: Task place- ment and fine-grained resource allocation for edge machine learning,

    Y . Li, T. Zeng, X. Zhang, J. Duan, and C. Wu, “Tapfinger: Task place- ment and fine-grained resource allocation for edge machine learning,” in Proceedings of the IEEE Conference on Computer Communications , 2023, pp. 1–10

    2023

  29. [37]

    The non- stochastic multiarmed bandit problem,

    P. Auer, N. Cesa-Bianchi, Y . Freund, and R. E. Schapire, “The non- stochastic multiarmed bandit problem,” SIAM Journal on Computing , vol. 32, no. 1, pp. 48–77, 2002

    2002

  30. [38]

    Alpaca: A strong, replicable instruction- following model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Alpaca: A strong, replicable instruction- following model,” Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html , vol. 3, no. 6,...

    2023

  31. [39]

    ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools,

    T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Rojas, G. Feng, H. Zhao, H. Lai et al., “ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools,” arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  32. [40]

    Introducing Mpt-7b: A new standard for open-source, commercially usable LLMs, 2023,

    M. Team et al., “Introducing Mpt-7b: A new standard for open-source, commercially usable LLMs, 2023,” URL www. mosaicml. com/blog/mpt- 7b. Accessed, pp. 05–05, 2023

    2023

  33. [41]

    BERTScore: Evaluating text generation with BERT,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “BERTScore: Evaluating text generation with BERT,” in Proceedings of the 8th International Conference on Learning Representations , 2020, pp. 5333–5375

    2020

  34. [42]

    Soft Actor-Critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 1861–1870

    2018

  35. [43]

    DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter,

    V . Sanh, “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108 , 2019

    1910 arXiv

  36. [44]

    PyTorch: An im- perative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , “PyTorch: An im- perative style, high-performance deep learning library,” in Proceedings of the 33rd International Conference on Neural Information Processing...

    2019

  37. [45]

    TorchRL: A data-driven decision- making library for PyTorch,

    A. Bou, M. Bettini, S. Dittert, V . Kumar, S. Sodhani, X. Yang, G. De Fabritiis, and V . Moens, “TorchRL: A data-driven decision- making library for PyTorch,” arXiv preprint arXiv:2306.00577 , 2023

    2023 arXiv

  38. [46]

    Fast graph representation learning with PyTorch Geometric,

    M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric,” arXiv preprint arXiv:1903.02428 , 2019

    1903 arXiv

  39. [47]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.