Pith. sign in

REVIEW 5 major objections 5 minor 44 references

This paper claims that interleaving model offloading into pipeline parallelism lets a small cluster of memory-constrained edge devices serve lossless 70B-parameter LLM inference at substantially lower latency than prior collaborative infere

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:59 UTC pith:QB7QCI7J

load-bearing objection The abstract's LOIP and the body's LIME disagree on system, scale, and speedups; the cost model's perfect-overlap assumption is plausible but unvalidated. the 5 major comments →

arxiv 2512.21835 v2 pith:QB7QCI7J submitted 2025-12-26 cs.DC

Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices

Mingyu Sun , Xiao Zhang , Shen Qu , Yan Li , Mengbai Xiao , Yanwei Zheng , Yuan Yuan , Dongxiao Yu This is my paper
classification cs.DC
keywords lossless LLM inferenceedge computingpipeline parallelismmodel offloadingKV cacheheterogeneous devicesinterleaved pipelineJetson
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a cluster of heterogeneous edge devices with tight memory budgets can run large language models losslessly—no quantization, distillation, or pruning—by combining pipeline parallelism with dynamic model offloading. The key move is an interleaved pipeline schedule that overlaps the loading of offloaded layers with computation and communication, plus a cost model and scheduler that allocate layers based on each device's memory, compute, and loading capacity. The body reports 1.7x and 3.7x speedups over state-of-the-art baselines on LLaMA3.3-70B under two request patterns, while the abstract claims 8.8x–20.3x speedups; the paper leaves this discrepancy unresolved. A sympathetic reader would care because this points toward private, edge-only inference of large models without sacrificing accuracy.

Core claim

The paper's central claim is that offloading-based interleaved pipeline parallelism can hide most SSD loading latency behind computation and communication, making lossless inference of 70B-scale LLMs practical on a handful of Jetson-class edge devices. The system, called LIME, assigns each device multiple small pipeline stages so that, within a segment, the layers that must be offloaded for the next segment load while the current segment is still computing and while data moves between devices. A dynamic-programming offline scheduler solves a cost model that explicitly accounts for offloading overhead, and an online memory adaptation strategy responds to KV-cache growth and network bandwidth

What carries the argument

The central mechanism is the interleaved pipeline combined with the offload-aware cost model of Equation 1. In the interleaved pipeline, model layers are partitioned into segments; each device runs one stage per segment and asynchronously loads the next segment's offloaded layers while computing the current segment. The cost model expresses per-token latency as computation time plus segment communication plus 'uncovered' loading time—the portion of load(L̃_i) that cannot be hidden behind computation and communication on other devices. The offline scheduler (Algorithm 1) uses dynamic programming to minimize this total, choosing layer assignments and the number of segments, then fine-grained M

Load-bearing premise

The speedup rests on the assumption that offloading a model segment from SSD to GPU can be overlapped almost perfectly with computation and communication, so that only the uncovered remainder adds latency, and the paper does not experimentally validate this overlap.

What would settle it

Instrument the described Jetson testbed to measure per-step latency while recording SSD read/write time, PCIe utilization, and GPU compute time; if the measured latency exceeds the cost model's predicted uncovered loading time by a significant margin—indicating that loads contend with compute or KV-cache transfers—the central overlap premise is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the overlap assumption holds, lossless 70B-class inference becomes feasible on small homogeneous or heterogeneous edge clusters, enabling privacy-preserving deployment without cloud uploads.
  • The cost model and DP scheduler are generalizable to any transformer with offloadable layer blocks, not just LLaMA, and to other memory-constrained accelerators.
  • The interleaved pipeline principle could be combined with quantization or sparsity techniques to push the latency envelope further while retaining some accuracy margin.
  • The online KV-cache transfer protocol offers a template for adaptive cache migration in other distributed or pipelined inference systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The abstract's speedup claims (8.8x–20.3x) versus the body's measured speedups (1.7x–3.7x) are not reconciled; if the realistic figure is the lower one, the practical advantage over baselines is more modest but still positive.
  • The entire result depends on loading being contention-free with compute and network transfer; if SSD/DRAM contention is significant in practice, the scheduler's allocations could be suboptimal and the speedups would shrink.
  • The paper implicitly assumes that offloading model shards is always cheaper than offloading KV cache; a test that varies the KV-cache-to-model-size ratio could expose where that crossover lies and whether the protocol's cache transfer decision is optimal.
  • The online memory adaptation strategy could be extended to handle multi-request batching at the edge, since the current evaluation uses micro-batch size 1 for sporadic patterns.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The full text (titled LIME) proposes a collaborative inference system for lossless LLM serving on memory-constrained edge devices. LIME combines an interleaved pipeline with model offloading so that loading the next segment's layers can overlap with computation and communication. It introduces a heterogeneous cost model (Eq. 1), a dynamic-programming offline scheduler (Alg. 1), and an online memory adaptation strategy consisting of an MHA/MLP block offloading plan and a bandwidth-sensitive KV-cache transfer protocol. The evaluation deploys the system on NVIDIA Jetson testbeds with Llama2-13B, Qwen3-32B, and Llama3.3-70B models, comparing against six baselines under two request patterns and several bandwidth settings. However, the submission is internally inconsistent: the abstract describes a different system (LOIP) with different speedups and platform counts, and the body contains conflicting statements about the model used in Sec. V.C.

Significance. The motivating problem is real: memory-constrained edge devices cannot hold 70B models or their KV caches, and existing offloading pipelines create bubbles. If the claimed results held, the interleaved-pipeline-plus-offloading design would be a useful contribution for edge LLM serving, and the paper shows commendable implementation breadth: real Jetson testbeds, three model scales, six baselines, and an ablation study. The offline scheduler has polynomial complexity O(|L_Lef t|^2 |D|), which is appealing. Unfortunately, the central quantitative claim is not currently well-defined because of the abstract/body mismatch, and the cost model that drives the scheduler is both unvalidated and internally inconsistent with the DP recurrence. Those issues make the speedup claims unsubstantiated as submitted.

major comments (5)
  1. [Abstract vs. full text] The submission is not self-consistent. The abstract describes system 'LOIP' with 8.8x–20.3x speedups, five Jetson devices, 2500+ lines of Python, and 500+ lines of C++/CUDA; the full text describes 'LIME' with 1.7x and 3.7x speedups, four devices, and 1000+ lines of Python. Section V.C states the extremely low-memory experiments use Llama3.3-70B-Instruct, but Figs. 15–17 are captioned 'Qwen3-32B' and the Setting-1 description uses five devices. The paper's headline result is therefore not well-defined; there is no single coherent central claim to evaluate.
  2. [§IV-B, Eq. (1)] The T_uncover term assumes loading of \tilde L_i can be fully overlapped with comp(L_i - \tilde L_i) + Σ_{i'≠i} comp(L_i') + |D| h_size/bw_net, with only the positive residual exposed. This is a contention-free premise: SSD/DRAM/PCIe traffic for parameter loading is assumed not to interfere with forward-pass compute, KV-cache transfers, or inter-device communication. No per-phase timing breakdown, no comparison of Eq. (1) predictions with measured per-step latencies, and no contention study on the Jetson memory systems is reported. If loading contends with compute or communication, the DP scheduler optimizes an incorrect objective and the reported speedups (Figs. 12–18) are unsupported.
  3. [§IV-C, Alg. 1 lines 6–7 vs. Eq. (1)] The recurrence subtracts the same T_idle_i at every device transition, although Eq. (2)'s T_idle_i includes the same segment-wide overlap window (Σ_i comp + |D|h/bw) for each i. Hence the same idle interval can be counted once per device. Example: three devices with load times 100, 300, 100 and idle time 200 give T_uncover=100 by Eq. (1), but the recurrence produces 0 (max(0, max(0, max(0,100−200)+300−200)+100−200)=0). No proof is given that the recurrence computes the max in Eq. (1); without it, the scheduler's optimality and the speedup figures rest on an unverified internal model.
  4. [§V-A] The paper states: 'As this is lossless inference, no accuracy results were recorded.' The abstract and conclusion claim 'without compromising model accuracy' and 'incurring no accuracy loss.' This missing evaluation is explicitly acknowledged in the text and is load-bearing for the lossless claim. The authors should at least compare generated outputs or task accuracy against a reference HuggingFace FP16 run; currently the lossless claim is unvalidated.
  5. [§V-B–§V-D] All latency figures are single scalar values per condition (e.g., Figs. 12–18), with no error bars, repetitions, or statistical analysis. Given the unvalidated overlap assumption in the cost model and the low-memory OOM/OOT thresholds, it is impossible to determine whether the reported speedups are within run-to-run noise. Quantitative claims of speedup require repeated measurements and a statement of variance.
minor comments (5)
  1. [§V-A] The sentence 'we set the micro-batch size for inference to 1 and the number of devices, respectively' is garbled; please clarify the batch sizes used for the sporadic and bursty request patterns.
  2. [§IV-D, Eq. (5)] In Eq. (5), T^{S1}_i = M_em_i / mem(token[1:n−n_i^trans]) is dimensionally unclear: if mem(...) denotes the total memory footprint of the listed tokens, the quotient does not yield a token-length threshold. Please define the units of mem and clarify the intended formula.
  3. [§IV-C, Alg. 1 lines 14–23] The code uses h_size × p_M and h_size × p_A as memory amounts, but Table I defines h_size as the intermediate output size of a decoder layer. If h_size is a byte count for activations, it is not the memory footprint of an MLP or MHA block; please use consistent notation for weight-memory sizes.
  4. [§III, Fig. 2b] The caption says 'comparison between model offloading and KV cache offloading,' while the body explains a comparison of per-step loading latency when offloading one MHA block versus offloading the KV cache. Please align the wording and specify the exact quantities plotted.
  5. [§V.C] The text refers to 'extremely low-memory scenarios ... on Llama3.3-70B-Instruct,' while the captions of Figs. 15–17 label the same results as Qwen3-32B. The model identity should be corrected in one of the two places.

Circularity Check

0 steps flagged

No circular derivation: the cost model and scheduler are inputs, while speedups come from measured latency; remaining issues are validation and reporting gaps, not circularity.

full rationale

The claimed derivation chain is: build a cost model (Eq. 1), use dynamic programming (Alg. 1) to choose a layer partition minimizing that model, run an online memory/KV-cache adaptation loop (Sec. IV-D), and then report wall-clock latency against six baselines. Nothing in this chain predicts the measured latency from the model alone after fitting to it. The offline scheduler optimizes Eq. 1, but the evaluation measures real execution on Jetson devices; an incorrect model would yield poor allocations and contradict the speedup claim, so the model is falsifiable rather than tautological. The fixed empirical n and profiled computation/loading times are configuration inputs, not fitted values that by construction equal the measured outcome. The online planner (Eqs. 5-8, Alg. 2) is a feedback controller that sizes KV-cache transfers to available slack and reacts to measured bandwidth; the reported latency is measured after adaptation, not computed directly from the control law. Self-citations in the reference list ([19], [21], [24], [25]) are related-work citations on compression, pruning, and federated learning and are not load-bearing; no uniqueness theorem is imported from the authors' prior work. The abstract/body speedup discrepancy (8.8x-20.3x vs 1.7x and 3.7x) and the statement 'no accuracy results were recorded' are serious correctness and verifiability concerns, but they are not instances of a claim reducing by definition to its inputs. Score 2 reflects only minor self-citations and lightly self-confirming configuration choices; there is no fatal circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The system rests on several domain assumptions about contention-free overlap, uniform bandwidth, and regular interleaved scheduling. The free parameters (n, n_ts, profiling estimates) are tuned to the testbed and not reported, so the optimization is partly fitted to the evaluation environment.

free parameters (3)
  • empirical inference sequence length n = constant, exact value not stated
    Used by the offline scheduler to allocate layers and estimate KV-cache memory (Sec. IV-C: 'LIME employs an empirical value for n, which is fixed as a constant'). The chosen value affects the memory constraint and allocation; no sensitivity analysis is provided.
  • fluctuation threshold n_ts = not specified
    Hyperparameter in Algorithm 2 (line 14) that suppresses KV-cache transfer changes due to minor bandwidth fluctuations; no value or tuning procedure is given, and results may depend on it.
  • profiled comp(L_i), load(~L_i) estimates = obtained from device profiling, values not reported
    The cost model and DP scheduler take computation and loading times per device as inputs. These are measured, not derived, and the paper does not report them, so the allocation is tuned to the test devices.
axioms (5)
  • domain assumption A single uniform pairwise network bandwidth bw_net between all devices, unaffected by transfer patterns.
    Eq.1 and Eq.8 assume all devices share the same bandwidth and that KV-cache transfers do not contend with pipeline communication. Edge network contention is not modeled.
  • domain assumption Offloading/loading can overlap perfectly with computation and communication of other devices; only the max(0, load - overlap) is uncovered.
    Sec. IV-A and Eq.1 define Tuncover as max(load(~Li) - (comp of remaining + others + comm), 0). This assumes no memory-bus/SSD/PCIe contention between loading and compute, and no interference between model loading and KV-cache transfers.
  • domain assumption At least one segment's layers are fully resident in GPU memory and layer positions are identical across segments (interleaved pipeline regularity).
    Used in the memory constraint of Eq.1: mem((|Li|-|~Li|)*(#Seg-1)/#Seg) + mem(KV) <= Mem_i. This holds only under the specific interleaved schedule and uniform layer sizes.
  • ad hoc to paper DP state transition Fallo(l,i) correctly accumulates uncovered loading delays with independence across devices.
    Eqs. (3)-(4) presume that putting offloaded layers on earlier devices does not change the overlap available to later devices beyond the subtracted T_idle; no proof of optimality or counterexample analysis is given.
  • domain assumption KV-cache memory grows linearly with token count and can be transferred as discrete blocks without correctness impact.
    The online planner (Sec. IV-D, Eq.5-8) treats KV-cache blocks as movable tokens; the paper does not discuss attention masks, causal dependencies, or reordering issues in concurrent transfer.
invented entities (2)
  • Unified Memory Architecture (UMA)-aware loading optimization with customized CUDA operators no independent evidence
    purpose: In the abstract, LOIP uses UMA-aware loading to reduce runtime loading overhead on Jetson devices.
    This component appears only in the LOIP abstract; the LIME full text never describes UMA or the CUDA operators. No design, code, or measurements are provided, so it functions as an unexplained claim rather than a demonstrated component.
  • High-threshold device (d_target) for KV-cache receiving no independent evidence
    purpose: Online KV-cache transfer protocol assigns each device a target device that holds transferred KV caches to smooth memory growth.
    Described in Sec. IV-D and Fig.10 as a scheduling role, not a physical entity. Its benefit is only shown by ablation; there is no independent measurement of the protocol's behavior.

pith-pipeline@v1.3.0-alltime-deepseek · 21439 in / 13879 out tokens · 130969 ms · 2026-08-03T13:59:07.946909+00:00 · methodology

0 comments
read the original abstract

Providing lossless inference services of LLMs on edge devices remains challenging, especially given the extremely tight memory budgets. The existing offloading techniques inevitably introduce numerous loading bubbles, which further inflate the end-to-end latency of the entire inference pipeline. Meanwhile, dynamically fluctuating network bandwidth and diverse user request patterns pose additional obstacles to efficient lossless inference on edge devices. To address this, we propose LOIP, a collaborative lossless LLM inference system that employs an offloading-based interleaved pipeline parallelism to better overlap model offloading with computing and communicating. Specifically, LOIP first constructs an offloading-aware cost model to characterize inference latency and memory overhead under heterogeneous device capabilities and limited bandwidth. Based on this cost model, LOIP develops a fine-grained allocation scheduler that determines latency-efficient layer partitions across devices while explicitly accounting for offloading overhead, along with a unified memory architecture (UMA)-aware loading optimization using customized CUDA operators to reduce runtime loading overhead. LOIP further designs an online memory adaptation strategy to handle the increasing KV cache pressure and dynamic bandwidth fluctuations during inference. We implement LOIP with 2500+ lines of Python and 500+ lines of C++/CUDA code, and deploy it on five heterogeneous NVIDIA Jetson edge devices for lossless collaborative inference of LLaMA3.3-70B-Instruct. Extensive experiments demonstrate that LOIP achieves 8.8$\times$$\sim$20.3$\times$ speedups over the SOTA baselines under different bandwidth conditions and request patterns without compromising model accuracy.

Figures

Figures reproduced from arXiv: 2512.21835 by Dongxiao Yu, Mengbai Xiao, Mingyu Sun, Shen Qu, Xiao Zhang, Yan Li, Yanwei Zheng, Yuan Yuan.

Figure 1
Figure 1. Figure 1: Multiple edge devices collaboratively perform lossless [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Motivational analysis of combining pipeline parallelism [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of LIME pipeline parallelism and tradi￾tional pipeline parallelism with offloading under bursty request pattern. A1 represents the computation of micro-batch A at layer 1. while avoiding the communication bottleneck under limited network bandwidth. To validate this observation, we conduct experiments on Llama 3.3-70B-Instruct and Qwen3-32B, using two different heterogeneous edge devices settings… view at source ↗
Figure 5
Figure 5. Figure 5: Overview of the LIME framework. To enable acceleration of collaborative lossless inference, a cost model and a fine-grained offline allocation scheduler are employed to obtain the optimal layer allocation strategy, while an online memory adaptation strategy is designed to handle the additional memory pressure caused by the KV cache and fluctuating network bandwidth during inference. Fig.3a and Fig.4a. Taki… view at source ↗
Figure 6
Figure 6. Figure 6: The detail of Interleaved Pipeline In LIME. Data A represents the forward propagation of micro-batch A. Layers are asynchronously offloaded only after the last micro-batch completes their computation, followed by loading the corresponding layers of the next segment. Different devices may load different numbers of layers, determined by their loading capability and the computation–communication time availabl… view at source ↗
Figure 7
Figure 7. Figure 7: Comparison between the appropriate and excessive [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison between the appropriate and insufficient [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The details of online memory-aware planner. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The details of KV cache transfer protocol. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Overview of the main testbed platform, including 3 [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance comparison in experimental environment E1 on the Llama2-13B-Instruct model. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Performance comparison in experimental environment E2 on the Qwen3-32B model. [PITH_FULL_IMAGE:figures/full_fig_p011_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Performance comparison in experimental environment E3 on the Llama3.3-70B-Instruct model. [PITH_FULL_IMAGE:figures/full_fig_p011_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Performance comparison in extremely low-memory scenarios under Setting 1 on the Qwen3-32B model. [PITH_FULL_IMAGE:figures/full_fig_p012_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Performance comparison in extremely low-memory scenarios under Setting 2 on the Qwen3-32B model. [PITH_FULL_IMAGE:figures/full_fig_p012_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Performance comparison in extremely low-memory scenarios under Setting 3 on the Qwen3-32B model. [PITH_FULL_IMAGE:figures/full_fig_p012_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Performance comparison under varying network band [PITH_FULL_IMAGE:figures/full_fig_p013_18.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 9 linked inside Pith

  1. [1]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskeveret al., “Improving language understanding by generative pre-training,” 2018. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14

    2018

  3. [3]

    Sasha: creative goal-oriented reasoning in smart homes with large language models,

    E. King, H. Yu, S. Lee, and C. Julien, “Sasha: creative goal-oriented reasoning in smart homes with large language models,”Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 8, no. 1, pp. 1–38, 2024

    2024

  4. [4]

    Gemini robotics: Bringing ai into the physical world,

    G. R. Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijlet al., “Gemini robotics: Bringing ai into the physical world,”arXiv preprint arXiv:2503.20020, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Privacy inference attacks and defenses in cloud-based deep neural network: A survey,

    X. Zhang, C. Chen, Y . Xie, X. Chen, J. Zhang, and Y . Xiang, “Privacy inference attacks and defenses in cloud-based deep neural network: A survey,”arXiv preprint arXiv:2105.06300, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    Privacy-preserving federated learning for transportation mode prediction based on personal mobility data,

    F. Yu, Z. Xu, Z. Qin, and X. Chen, “Privacy-preserving federated learning for transportation mode prediction based on personal mobility data,”High-Confidence Computing, vol. 2, no. 4, p. 100082, 2022

    2022

  7. [7]

    Re- search on medical data storage and sharing model based on blockchain,

    J. Zhao, W. Qiang, Z. Zhao, T. An, Z. Kuang, D. Xu, and L. Shi, “Re- search on medical data storage and sharing model based on blockchain,” High-Confidence Computing, vol. 3, no. 3, p. 100133, 2023

    2023

  8. [8]

    {ServerlessLLM}:{Low-Latency}serverless inference for large language models,

    Y . Fu, L. Xue, Y . Huang, A.-O. Brabete, D. Ustiugov, Y . Patel, and L. Mai, “{ServerlessLLM}:{Low-Latency}serverless inference for large language models,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 135–153

    2024

  9. [9]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv e-prints, pp. arXiv–2407, 2024

    2024

  10. [10]

    Nvidia jetson xavier,

    “Nvidia jetson xavier,” https://www.nvidia.cn/autonomous-machines/ embedded-systems/jetson-xavier-series/, 2019

    2019

  11. [11]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization,

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y . S. Shao, K. Keutzer, and A. Gholami, “Kvquant: Towards 10 million context length llm inference with kv cache quantization,”Advances in Neural Information Processing Systems, vol. 37, pp. 1270–1303, 2024

    2024

  12. [12]

    Spqr: A sparse- quantized representation for near-lossless llm weight compression,

    T. Dettmers, R. Svirschevski, V . Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, “Spqr: A sparse- quantized representation for near-lossless llm weight compression,” arXiv preprint arXiv:2306.03078, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”Advances in neural information processing systems, vol. 36, pp. 10 088–10 115, 2023

    2023

  14. [14]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,”arXiv preprint arXiv:2210.17323, 2022

    Pith/arXiv arXiv 2022

  15. [15]

    Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,”Proceedings of machine learning and systems, vol. 6, pp. 87–100, 2024

    2024

  16. [16]

    Minillm: Knowledge distillation of large language models,

    Y . Gu, L. Dong, F. Wei, and M. Huang, “Minillm: Knowledge distillation of large language models,”arXiv preprint arXiv:2306.08543, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Less is more: Task-aware layer-wise distillation for language model compression,

    C. Liang, S. Zuo, Q. Zhang, P. He, W. Chen, and T. Zhao, “Less is more: Task-aware layer-wise distillation for language model compression,” in International Conference on Machine Learning. PMLR, 2023, pp. 20 852–20 867

    2023

  18. [18]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes,

    C.-Y . Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y . Fujii, A. Ratner, R. Kr- ishna, C.-Y . Lee, and T. Pfister, “Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes,” inFindings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 8003–8017

    2023

  19. [19]

    Resource-aware federated self-supervised learn- ing with global class representations,

    M. Li, X. Zhang, Q. Wang, T. Liu, R. Wu, W. Wang, F. Zhuang, H. Xiong, and D. Yu, “Resource-aware federated self-supervised learn- ing with global class representations,”Advances in Neural Information Processing Systems, vol. 37, pp. 10 008–10 035, 2024

    2024

  20. [20]

    Sparsegpt: Massive language models can be accurately pruned in one-shot,

    E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” inInternational conference on machine learning. PMLR, 2023, pp. 10 323–10 337

    2023

  21. [21]

    Unity is power: Semi-asynchronous collaborative training of large-scale models with structured pruning in resource-limited clients,

    Y . Li, X. Zhang, M. Li, G. Xu, F. Chen, Y . Yuan, Y . Zou, M. Zhao, J. Lu, and D. Yu, “Unity is power: Semi-asynchronous collaborative training of large-scale models with structured pruning in resource-limited clients,” IEEE Transactions on Mobile Computing, 2025

    2025

  22. [22]

    Slicegpt: Compress large language models by deleting rows and columns,

    S. Ashkboos, M. L. Croci, M. G. d. Nascimento, T. Hoefler, and J. Hensman, “Slicegpt: Compress large language models by deleting rows and columns,”arXiv preprint arXiv:2401.15024, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Llm-pruner: On the structural pruning of large language models,

    X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,”Advances in neural information processing systems, vol. 36, pp. 21 702–21 720, 2023

    2023

  24. [24]

    Generalization-aware distributed minimax optimization for large-scale models on resource-limited devices,

    M. Li, X. Zhang, Y . Yuan, Y . Zou, S. Guo, X. Cheng, and D. Yu, “Generalization-aware distributed minimax optimization for large-scale models on resource-limited devices,” inThe 21st International Confer- ence on Mobility, Sensing and Networking, 2025

    2025

  25. [25]

    Theoretical convergence guaranteed resource-adaptive federated learning with mixed heterogeneity,

    Y . Wang, X. Zhang, M. Li, T. Lan, H. Chen, H. Xiong, X. Cheng, and D. Yu, “Theoretical convergence guaranteed resource-adaptive federated learning with mixed heterogeneity,” inProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 2444–2455

    2023

  26. [26]

    Credit risk analysis using machine and deep learning models,

    P. M. Addo, D. Guegan, and B. Hassani, “Credit risk analysis using machine and deep learning models,”Risks, vol. 6, no. 2, p. 38, 2018

    2018

  27. [27]

    Medical image analysis using deep learning algorithms,

    M. Li, Y . Jiang, Y . Zhang, and H. Zhu, “Medical image analysis using deep learning algorithms,”Frontiers in public health, vol. 11, p. 1273253, 2023

    2023

  28. [28]

    Pipeedge: Pipeline parallelism for large-scale model inference on heterogeneous edge devices,

    Y . Hu, C. Imes, X. Zhao, S. Kundu, P. A. Beerel, S. P. Crago, and J. P. Walters, “Pipeedge: Pipeline parallelism for large-scale model inference on heterogeneous edge devices,” in2022 25th Euromicro Conference on Digital System Design (DSD). IEEE, 2022, pp. 298–307

    2022

  29. [29]

    Edgeshard: Efficient llm inference via collaborative edge computing,

    M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, “Edgeshard: Efficient llm inference via collaborative edge computing,”IEEE Internet of Things Journal, 2024

    2024

  30. [30]

    Galaxy: A resource-efficient collaborative edge ai system for in-situ transformer inference,

    S. Ye, J. Du, L. Zeng, W. Ou, X. Chu, Y . Lu, and X. Chen, “Galaxy: A resource-efficient collaborative edge ai system for in-situ transformer inference,” inIEEE INFOCOM 2024-IEEE Conference on Computer Communications. IEEE, 2024, pp. 1001–1010

    2024

  31. [31]

    Tpi-llm: Serving 70b-scale llms efficiently on low-resource mobile devices,

    Z. Li, W. Feng, M. Guizani, and H. Yu, “Tpi-llm: Serving 70b-scale llms efficiently on low-resource mobile devices,”IEEE Transactions on Services Computing, 2025

    2025

  32. [32]

    Band: coordinated multi-dnn inference on heterogeneous mobile processors,

    J. S. Jeong, J. Lee, D. Kim, C. Jeon, C. Jeong, Y . Lee, and B.-G. Chun, “Band: coordinated multi-dnn inference on heterogeneous mobile processors,” inProceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 2022, pp. 235–247

    2022

  33. [33]

    Coedge: Cooperative dnn inference with adaptive workload partitioning over heterogeneous edge devices,

    L. Zeng, X. Chen, Z. Zhou, L. Yang, and J. Zhang, “Coedge: Cooperative dnn inference with adaptive workload partitioning over heterogeneous edge devices,”IEEE/ACM Transactions on Networking, vol. 29, no. 2, pp. 595–608, 2020

    2020

  34. [34]

    Model parallelism optimization for distributed inference via decoupled cnn structure,

    J. Du, X. Zhu, M. Shen, Y . Du, Y . Lu, N. Xiao, and X. Liao, “Model parallelism optimization for distributed inference via decoupled cnn structure,”IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 7, pp. 1665–1676, 2020

    2020

  35. [35]

    When the edge meets transformers: Distributed inference with transformer models,

    C. Hu and B. Li, “When the edge meets transformers: Distributed inference with transformer models,” in2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS). IEEE, 2024, pp. 82–92

    2024

  36. [36]

    Jupiter: Fast and resource-efficient collaborative inference of generative llms on edge devices,

    S. Ye, B. Ouyang, L. Zeng, T. Qian, X. Chu, J. Tang, and X. Chen, “Jupiter: Fast and resource-efficient collaborative inference of generative llms on edge devices,” inIEEE INFOCOM 2025-IEEE Conference on Computer Communications. IEEE, 2025, pp. 1–10

    2025

  37. [37]

    Sequence parallelism: Long sequence training from system perspective,

    S. Li, F. Xue, C. Baranwal, Y . Li, and Y . You, “Sequence parallelism: Long sequence training from system perspective,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 2391–2404

    2023

  38. [38]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,”arXiv preprint arXiv:1909.08053, 2019

    Pith/arXiv arXiv 1909

  39. [39]

    Pipedream: Generalized pipeline parallelism for dnn training,

    D. Narayanan, A. Harlap, A. Phanishayee, V . Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: Generalized pipeline parallelism for dnn training,” inProceedings of the 27th ACM symposium on operating systems principles, 2019, pp. 1–15

    2019

  40. [40]

    Gpipe: Efficient training of giant neu- ral networks using pipeline parallelism,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wuet al., “Gpipe: Efficient training of giant neu- ral networks using pipeline parallelism,”Advances in neural information processing systems, vol. 32, 2019

    2019

  41. [41]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasleyet al., “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” inSC22: International Conference for High Performance Com- puting, Networking, Storage and Analysis. IEEE, 2022, pp. 1–15

    2022

  42. [42]

    Flexgen: High-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 31 094–31 116

    2023

  43. [43]

    Linux advanced routing & traffic control howto,

    B. Hubertet al., “Linux advanced routing & traffic control howto,” Netherlabs BV, vol. 1, pp. 99–107, 2002

    2002

  44. [44]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 15 Mingyu Sunis currently pursuing the M.S. degree with the School of Computer Science and Tech- nology, Shandong University. He received his B....

    Pith/arXiv arXiv 2025