Pith. sign in

REVIEW 3 major objections 7 minor 73 references

Personal device clusters run larger LLMs with 16.5% accuracy gain

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 20:58 UTC pith:ALZWKU5T

load-bearing objection Voltron does real multi-device edge LLM inference on physical hardware, but the central ablation is missing. the 3 major comments →

arxiv 2607.07046 v1 pith:ALZWKU5T submitted 2026-07-08 cs.DC

Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence

Chanwoo Cho , Wooseok Kim , Yonglak Son , Young Seo Lee , Young Geun Kim This is my paper
classification cs.DC
keywords edge computingLLM inferencedistributed inferencemodel parallelismtensor parallelismhybrid parallelismmixed precisionheterogeneous computing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes that the multiple personal devices a user already owns — phone, tablet, smartwatch, IoT speakers — can be treated as a single elastic compute fabric for running large language models locally, breaking through the memory wall of any single device. The central mechanism is layer-wise hybrid parallelism: instead of applying one distribution strategy (model parallelism or tensor parallelism) uniformly across all layers, Voltron selects, for each layer individually, whichever of the two strategies is estimated to be faster given the current devices, layer type, precision, and network conditions. This per-layer decision is combined with importance-aware mixed precision (keeping high precision on layers that matter most for accuracy) and runtime adaptation (scaling precision or pruning when KV cache grows or wireless signal weakens). The result is that larger LLMs — too big for any one device — can run across a heterogeneous cluster of personal devices while meeting latency targets of 10 seconds for first token and 400 milliseconds per subsequent token, achieving up to 16.5% higher accuracy than the best model that fits on a single device.

Core claim

The key finding is that no single parallelism strategy works well across all layers of an LLM when devices are heterogeneous. Attention layers, feed-forward layers, mixture-of-experts layers, and layers at different precision levels each have different computation-to-communication ratios, and the optimal strategy (model parallelism versus tensor parallelism) shifts depending on layer type, device capabilities, precision, inference phase (prefill versus decode), KV cache size, and wireless signal strength. By making the parallelism decision per-layer rather than per-model, and by re-evaluating that decision as runtime conditions change, the framework can consistently satisfy latency andmemory

What carries the argument

The key machinery is a regression-based execution time estimator that, for each layer, predicts computation time from device type, layer type, precision, input length, and KV cache size, and communication time from data size and wireless conditions. Using these estimates, Voltron assigns each layer to model parallelism (sequential layer execution across devices, low communication) or tensor parallelism (concurrent shard execution, high communication), then uses binary search to find the highest-precision configuration fitting within memory and latency budgets. At runtime, a computation scaling module adjusts precision and prunes shards when KV cache grows or devices disconnect, and a

Load-bearing premise

The per-layer parallelism decision depends on a regression model that estimates execution time with about 90% accuracy. The paper asserts this is sufficient to correctly pick the faster strategy for each layer, but does not show how often the model misranks the two strategies or how misranking affects end-to-end latency. If the estimator systematically errs on certain layer types or device combinations, the hybrid parallelism plan could be worse than a uniform strategy in un-

What would settle it

A scenario where the regression model's 90% accuracy translates to frequent misranking of model parallelism versus tensor parallelism on the layers that dominate total execution time (e.g., large feed-forward layers), causing Voltron to produce plans slower than uniform tensor parallelism while believing it has optimized.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The collection of personal devices a user carries — not just the phone — becomes the unit of compute for edge AI, shifting the design target from one-device optimization to cluster-aware optimization.
  • Per-layer parallelism selection could apply beyond edge settings: any heterogeneous cluster (mixed GPU types, mixed cloud instances) faces the same tension between computation-bound and communication-bound layers.
  • Runtime adaptation to wireless variability makes distributed edge inference practical in mobile scenarios where signal strength fluctuates, which has been a barrier to real-world deployment.
  • The energy optimization module shows multi-device inference can approach single-device energy consumption through voltage-frequency scaling, addressing a key practical objection to the approach.
  • The importance-aware precision framework provides a principled way to trade accuracy for memory under pressure, which could complement rather than compete with algorithm-side compression techniques.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Voltron, a framework for executing LLM inference across multiple heterogeneous edge devices (smartphones, tablets, smartwatches, IoT devices). The core mechanisms are: (1) layer-wise hybrid parallelism (HP), which assigns model parallelism (MP) or tensor parallelism (TP) to each layer based on estimated execution time and memory budget; (2) importance-aware mixed precision, which allocates FP16/INT8/INT4 per layer using offline perplexity-based importance profiling; and (3) elastic model execution, which adapts to runtime variance (growing KV cache, wireless signal fluctuations) via computation scaling (precision/pruning adjustments) and communication scaling (activation quantization). The system is evaluated on six physical devices across three clusters (Car, Office, Home) and three LLM families (gemma-3, Qwen1.5, Qwen2.5), reporting up to 16.5% accuracy improvement over single-device execution while satisfying QoS targets (TTFT ≤ 10s, TPOT ≤ 400ms). The experimental effort is substantial, involving real hardware and multiple model architectures including MoE variants.

Significance. The paper addresses a practically important problem: running larger LLMs on edge devices by pooling resources across multiple co-located devices. The implementation on real hardware (six devices, three clusters) rather than simulation is a genuine strength. The characterization of how device heterogeneity, model architecture (MHA vs. GQA vs. SWA vs. MoE), precision, and runtime dynamics interact with MP/TP performance (Section 3) is informative and well-motivated. The importance-aware mixed precision and elastic adaptation mechanisms are reasonable engineering contributions. However, the significance of the core novelty—layer-wise hybrid parallelism—is undermined by the absence of an ablation isolating it from mixed precision, as discussed in the major comments.

major comments (3)
  1. Section 5.2.1 and Figure 10: The paper compares Voltron (hybrid parallelism + importance-aware mixed precision + elastic adaptation) against a baseline MP/TP that uses neither mixed precision nor elastic adaptation. When Voltron satisfies QoS where MP/TP fails, it is unclear whether the QoS advantage is driven by hybrid parallelism (the core novelty) or by mixed precision (a known technique). Figure 12(a) shows mixed precision is critical for Voltron's QoS satisfaction, but the paper never tests whether MP/TP + mixed precision would also satisfy QoS. An ablation isolating hybrid parallelism from mixed precision is needed to substantiate the claim that HP is the key differentiator. Without it, the 1.0% average accuracy gain over MP/TP (Table 4) and the QoS advantage may be attributable primarily to mixed precision rather than to the novel parallelism mechanism.
  2. Section 4.2.1: The execution time regression model has ~90% estimation accuracy, and the paper asserts this is 'sufficient to correctly identify the better-performing parallelism method.' This claim is load-bearing because layer-wise HP depends on correctly ranking MP vs. TP per layer. However, 90% accuracy means ~10% of layers may be misrouted. The paper provides no sensitivity analysis showing how parallelism-selection error rate affects end-to-end latency or accuracy. A breakdown of how often the regression model misranks MP vs. TP (not just average estimation error) and the resulting latency impact would strengthen confidence in the core mechanism.
  3. Section 4.2.2 and Table 4: The importance scores are derived from offline perplexity profiling on a validation set, and the paper acknowledges that 'layer importance patterns can vary across tasks' and performs profiling separately per task. However, there is no quantification of cross-task generalization error—i.e., how much accuracy degrades when importance scores profiled for one task are applied to queries from a different task distribution. Since real edge deployments serve diverse user queries, this generalization gap should be characterized, or the paper should clarify that per-task profiling is assumed at deployment and discuss the associated cost.
minor comments (7)
  1. Section 3.2.2, Figure 5: The latency normalization baseline is described as 'normalized to that of MP in MHA of ATTN' but it is unclear whether this baseline is the same across subplots (a) and (b). Clarifying the normalization baseline for each subplot would aid interpretation.
  2. Section 4.3.1, footnote 5: The sentence about communication scaling executing before computation scaling is somewhat convoluted. A clearer description of the ordering rationale and what 'accuracy protection room' means would improve readability.
  3. Section 5.1.3: The MP/TP baseline is described as 'device heterogeneity-aware MP/TP' citing [27, 55], but the specific allocation strategy used for this baseline is not detailed. A brief description of how shards/layers are allocated in this baseline would help readers assess the fairness of the comparison.
  4. Table 4: The column headers for the Qwen1.5 family list '7B 14B MoE-A2.7B (14B)' but the single-device column for MoE-A2.7B appears to be missing. Clarifying whether single-device execution was possible for all model families would help.
  5. Section 5.2.3: The energy optimization results mention '30% compression ratio' but it is unclear whether this refers to parameter count, memory footprint, or FLOPs. Specifying the metric would improve reproducibility.
  6. Figure 8(b): The y-axis label and units are not clearly specified. Adding axis labels and units would improve readability.
  7. Section 2.2: The QoS targets (TTFT=10s, TPOT=400ms) are cited from prior work but the 10s TTFT target seems quite lenient for interactive use. A brief justification of why this target is considered acceptable would help.

Circularity Check

0 steps flagged

No circularity found: derivation chain is self-contained against external benchmarks

full rationale

The paper's derivation chain does not exhibit circular reasoning. (1) The importance scores (Section 4.2.2) are computed via offline perplexity profiling during channel pruning — an independent procedure that does not reference the target accuracy metric (MMLU/Hellaswag/GSM8K/MATH). The accuracy is then measured on external benchmarks, not derived from the importance scores. (2) The execution-time regression model (Section 4.2.1) is trained on runtime features (device type, layer type, precision, token length, KV cache size) to predict latency; it is not trained on the parallelism-selection outcome it informs. The ~90% estimation accuracy is reported as an empirical figure, not a definitional identity. (3) No load-bearing self-citation chain exists: the importance metric cites Michel et al. [43] (external), the regression approach cites Feng et al. [16] (external), and communication estimation cites Kang et al. [31] (external). (4) The hybrid parallelism method is a system design that selects between MP and TP per layer based on estimated latency — it is not defined in terms of the accuracy it claims to achieve. (5) The QoS satisfaction and accuracy gains are independently measurable against stated thresholds (TTFT ≤ 10s, TPOT ≤ 400ms) and standard benchmarks. The skeptic's concern about confounding hybrid parallelism with mixed precision is an ablation/validity issue, not a circularity issue — the paper does not define its outputs in terms of its inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

See axiom_ledger.axioms above.

free parameters (4)
  • Regression model coefficients for execution time estimation
    The regression model (Section 4.2.1) uses runtime features (device type, layer type, precision, input token length, KV cache size) to estimate computation time. The model coefficients are fitted to measured data but not reported in the paper.
  • QoS latency targets (TTFT=10s, TPOT=400ms) = TTFT=10s, TPOT=400ms
    Adopted from prior work [12, 30] as fixed constraints. These are domain assumptions, not fitted parameters, but they define the feasible solution space.
  • Importance score thresholds for precision allocation
    The binary search in Section 4.2.2 progressively scales layer precision in ascending importance order. The stopping criterion depends on memory budget and QoS constraints, but the search direction logic and convergence criteria are not fully specified.
  • Communication scaling activation quantization threshold
    Section 4.3.2 describes when to trigger activation quantization based on expected communication overhead, but the specific threshold is not stated.
axioms (4)
  • domain assumption Multiple user-end devices are co-located and connected via low-latency wireless network (e.g., Wi-Fi Direct) with sufficient stability for distributed inference.
    Section 3.1 states devices 'are typically co-located and connected through low-latency wireless network.' This is load-bearing—if devices are not co-located or connections are unstable, multi-device execution degrades or fails.
  • domain assumption Channel-level perplexity differences on a validation set accurately reflect layer importance for downstream tasks.
    Section 4.2.2 uses perplexity differences from pruning channels to compute importance scores. The paper acknowledges task-specific profiling but assumes the importance ranking generalizes within a task.
  • domain assumption The regression model with ~90% accuracy is sufficient to correctly identify the better-performing parallelism method for each layer.
    Section 4.2.1 asserts this without sensitivity analysis. If the 10% error rate systematically misranks MP vs. TP for certain layer types or device combinations, the hybrid parallelism benefit erodes.
  • domain assumption Overlapping I/O operations with LLM computation effectively hides storage latency on mobile UFS.
    Section 4.3.1 (Overhead Mitigation) assumes idle compute resources are available during LLM execution to preload shards. This may not hold when devices are compute-saturated.
invented entities (2)
  • Hybrid Parallelism (HP) for edge LLM inference independent evidence
    purpose: Per-layer selection between MP and TP tailored for heterogeneous edge devices
    The paper provides empirical evidence (Figs. 5, 11) showing that per-layer HP outperforms uniform MP or TP. The approach is falsifiable: one could test whether HP's layer-wise decisions are consistently better than either uniform strategy.
  • Communication scaling via selective activation quantization independent evidence
    purpose: Reduce cross-device transmission data under wireless network degradation
    Section 4.3.2 describes quantizing activations based on importance scores. The approach is testable: one can measure whether activation quantization maintains accuracy while reducing communication overhead under varying signal strength.

pith-pipeline@v1.1.0-glm · 25251 in / 3852 out tokens · 153570 ms · 2026-07-09T20:58:08.386001+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are widely used in intelligent services due to their remarkable capability in generative tasks. Typically, LLM-based services process the inference requests of the users in a centralized data center. Unfortunately, such centralized execution has limitations for end-users, such as increased response latency with communication overhead and privacy leakage risk. To alleviate the aforementioned limitations, there have been increasing pushes to execute LLM inference locally on user-end devices. However, the limited resources of a single edge device impose restrictions on achievable accuracy of LLMs. To overcome the issue, we first propose to leverage multiple user-end devices available at the edge for LLM inference, enabling the execution of larger models. Specifically, we propose Voltron, a novel on-device LLM inference framework that elastically utilizes multiple user-end devices for LLM inference execution while adapting to diverse real-world edge environments. In our evaluation, Voltron achieves up to 16.5% higher accuracy than state-of-the-art LLMs that can be executed on a single edge device, satisfying user QoS requirements.

Figures

Figures reproduced from arXiv: 2607.07046 by Chanwoo Cho, Wooseok Kim, Yonglak Son, Young Geun Kim, Young Seo Lee.

Figure 1
Figure 1. Figure 1: LLM architecture and inference process. – We implement Voltron1 across various combinations of edge clusters and models. In our evaluation, Voltron achieves 10.2% higher accuracy on average (up to 16.5% higher ac￾curacy) than state-of-the-art LLMs that can be executed on a single device, while satisfying the QoS requirements even in the presence of runtime variance (Section 5). 2 Background 2.1 Large Langu… view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy and performance of on-device LLM [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Parallelism methods for multi-device LLM [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Latency and memory footprint of various LLM layers. Note latency is normalized to that of MP in MHA of ATTN. the device with the lowest computation capability (i.e., Pixel 5 in [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: (a) Wi-Fi Direct RSSI under user mobility and [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Overview of Voltron Framework. resulting configuration satisfies both the memory budget and QoS requirements. Then, it adjusts the search direction accordingly until the final configuration is determined. 4.3 Elastic Model Execution As we observe in Section 3.2.3, edge execution of LLM infer￾ence is inherently stochastic due to computational dynamics and wireless network variability. To adapt to the runtim… view at source ↗
Figure 10
Figure 10. Figure 10: Normalized TTFT/TPOT and accuracy for Voltron and baselines across clusters and models. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Adaptability to (a) precision heterogeneity, [PITH_FULL_IMAGE:figures/full_fig_p011_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Impact of Voltron Energy Optimization Adaptability to Network Variability: Voltron maintains high accuracy satisfying the latency constraints under vari￾ous wireless signal strength. We consider a scenario where a table is located in a room while a user carrying a watch and a mobile devices moves away from the tablet (which is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p012_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 73 canonical work pages · 28 internal anchors

  1. [1]

    1X. 2026. NEO Home Robot. https://www.1x.tech/neo

    work page 2026

  2. [2]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Am- mar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jian- min Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, ...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Shohin Aheleroff, Xun Xu, Yuqian Lu, Mauricio Aristizabal, Juan Pablo Velásquez, Benjamin Joa, and Yesid Valencia. 2020. IoT-enabled smart appliances under industry 4.0: A case study.Advanced Engineering Informatics43 (2020), 101043. doi:10.1016/j.aei.2020.101043

    work page doi:10.1016/j.aei.2020.101043 2020

  4. [4]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training General- ized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245 [cs.CL] https://arxiv.org/abs/2305.13245

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatam- ifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. Llm in a flash: Efficient large language model inference with limited memory. InProceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 12562–12584

    work page 2024

  6. [6]

    amazon. 2023. Alexa. https://alexa.amazon.com/about

    work page 2023

  7. [7]

    Apple. 2023. Siri. https://www.apple.com/siri/

    work page 2023

  8. [8]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. 2019. Post-training 4-bit quantization of convolution networks for rapid- deployment. arXiv:1810.05723 [cs.CV] https://arxiv.org/abs/1810. 05723

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  10. [10]

    Longformer: The Long-Document Transformer

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv:2004.05150 [cs.CL] https: //arxiv.org/abs/2004.05150

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  11. [11]

    ChatGPT. 2025. ChatGPT. https://openai.com/index/chatgpt/

    work page 2025

  12. [12]

    Xiantao Chen, Moli Zhou, Renzhen Wang, Yalin Pan, Jiaqi Mi, Hui Tong, and Daisong Guan. 2019. Evaluating Response Delay of Multimodal Interface in Smart Device. InDesign, User Experience, and Usability. Practice and Case Studies, Aaron Marcus and Wentao Wang (Eds.). Springer International Publishing, Cham, 408–419

    work page 2019

  13. [13]

    Cisco. 2020. Cisco Annual Internet Report. https://www.cisco.com/c/ en/us/solutions/collateral/executive-perspectives/annual-internet- report/white-paper-c11-741490.html

    work page 2020

  14. [14]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hee- woo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Train- ing Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG] https://arxiv.org/abs/2110.14168

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  15. [15]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer

    work page

  16. [16]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339 [cs.LG] https://arxiv.org/abs/2208.07339

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Jingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang, Ming Yan, and Jie Wu. 2025. WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling. InProceedings of the 52nd 13 Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 1283–1295. doi:10.1145/3695053.3730999

    work page doi:10.1145/3695053.3730999 2025

  18. [18]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh

    work page

  19. [19]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323 [cs.LG] https://arxiv.org/ abs/2210.17323

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Google. 2025. Gemini. https://deepmind.google/technologies/gemini/

    work page 2025

  21. [21]

    Google. 2025. Gemini Developer API Pricing. https://ai.google.dev/ gemini-api/docs/pricing

    work page 2025

  22. [22]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Liwei Guo, Wonkyo Choe, and Felix Xiaozhu Lin. 2023. STI: Tur- bocharge NLP Inference at the Edge via Elastic Pipelining. InPro- ceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (Vancouver, BC, Canada)(ASPLOS 2023). Association for Computing Machinery, New York, NY, USA, 7...

    work page doi:10.1145/3575693.3575698 2023

  24. [24]

    Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. 2024. What Matters in Transformers? Not All Attention is Needed. arXiv:2406.15786 [cs.LG] https://arxiv.org/abs/2406.15786

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mas- sive Multitask Language Understanding. arXiv:2009.03300 [cs.CY] https://arxiv.org/abs/2009.03300

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  26. [26]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Mea- suring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874 [cs.LG] https://arxiv.org/abs/2103.03874

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  27. [27]

    Training Compute-Optimal Large Language Models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Henni- gan, Eric Noland, Katie Millican, George van den Driessche, Bog- dan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent S...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  28. [28]

    Xiao Hong. 2026. ManusAI. https://www.manus.im

    work page 2026

  29. [29]

    Beerel, Stephen P

    Yang Hu, Connor Imes, Xuanang Zhao, Souvik Kundu, Peter A. Beerel, Stephen P. Crago, and John Paul Walters. 2022. PipeEdge: Pipeline Parallelism for Large-Scale Model Inference on Heterogeneous Edge Devices. In2022 25th Euromicro Conference on Digital System Design (DSD). 298–307. doi:10.1109/DSD57027.2022.00048

    work page doi:10.1109/dsd57027.2022.00048 2022

  30. [30]

    Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xi- anglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiao- juan Qi. 2025. SliM-LLM: Salience-Driven Mixed-Precision Quanti- zation for Large Language Models. arXiv:2405.14917 [cs.LG] https: //arxiv.org/abs/2405.14917

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. arXiv:1811.06965 [cs.CV] https://arxiv.org/abs/1811.06965

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  32. [32]

    Amant, Chetan Bansal, Victor Ruhle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan

    Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal, Victor Ruhle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan. 2025. SAGE- SERVE: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling.Proc. ACM Meas. Anal. Comput. Syst.9, 3, Article 61 (Dec. 2025), 24 pages...

    work page doi:10.1145/3771576 2025

  33. [33]

    Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. 2017. Neurosurgeon: Collab- orative Intelligence Between the Cloud and Mobile Edge.SIGARCH Comput. Archit. News45, 1 (April 2017), 615–629. doi:10.1145/3093337. 3037698

    work page doi:10.1145/3093337 2017

  34. [34]

    Kawrakow and contributors

    I. Kawrakow and contributors. 2023. k-quants. https://github.com/ggml-org/llama.cpp/pull/1684

    work page 2023

  35. [35]

    Andrey Kuzmin, Markus Nagel, Mart van Baalen, Arash Behboodi, and Tijmen Blankevoort. 2024. Pruning vs Quantization: Which is Better? arXiv:2307.02973 [cs.LG] https://arxiv.org/abs/2307.02973

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Sto- ica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles(Koblenz, Germany)(SOSP ’23). As- sociation for Computing Machinery, New Yor...

    work page doi:10.1145/3600006.3613165 2023

  37. [37]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GShard: Scaling Giant Models with Condi- tional Computation and Automatic Sharding. arXiv:2006.16668 [cs.CL] https://arxiv.org/abs/2006.16668

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  38. [38]

    Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai. 2022. Parameter-Efficient Sparsity for Large Language Models Fine-Tuning. arXiv:2205.11005 [cs.AI] https://arxiv. org/abs/2205.11005

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  39. [39]

    Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang

    work page

  40. [40]

    doi:10.1016/j.neucom

    Pruning and quantization for deep neural network acceleration: A survey.Neurocomputing461 (2021), 370–403. doi:10.1016/j.neucom. 2021.07.045

    work page doi:10.1016/j.neucom 2021

  41. [41]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for On- Device LLM Compression and Acceleration. InProceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 87–100. https://proceedi...

    work page 2024

  42. [42]

    Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghu- raman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. 2024. Mo- bileLLM: optimizing sub-billion parameter language models for on- device use cases. InProceedings of the 41st International Conference on Machine Learning(Vi...

    work page 2024

  43. [43]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. LLM-pruner: on the structural pruning of large language models. InProceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 950, 19 pages

    work page 2023

  44. [44]

    Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1(Rotterdam, Netherlands)(ASPLOS ’25). A...

    work page doi:10.1145/3669940.3707215 2025

  45. [45]

    Paul Michel, Omer Levy, and Graham Neubig. 2019. Are Sixteen Heads Really Better than One?. InAdvances in Neural Informa- tion Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Asso- ciates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/file/ 2c601ad9d2ff9bc8b282670cd...

    work page 2019

  46. [46]

    Paul Michel, Omer Levy, and Graham Neubig. 2019. Are Sixteen Heads Really Better than One? arXiv:1905.10650 [cs.CL] https://arxiv.org/ abs/1905.10650

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  47. [47]

    Monsoon. [n. d.]. High Voltage Power Monitor. ([n. d.]). https: //www.msoon.com/high-voltage-power-monitor

    work page

  48. [48]

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bon- darenko, Mart van Baalen, and Tijmen Blankevoort. 2021. A White Paper on Neural Network Quantization. arXiv:2106.08295 [cs.LG] https://arxiv.org/abs/2106.08295

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  49. [49]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model train- ing on GPU clusters using megatron-LM. InProceedings of the In- ternational Conference for ...

    work page doi:10.1145/3458817.3476209 2021

  50. [50]

    Jeong, Steven Y

    Sangeun Oh, Ahyeon Kim, Sunjae Lee, Kilho Lee, Dae R. Jeong, Steven Y. Ko, and Insik Shin. 2019. FLUID: Flexible User Interface Dis- tribution for Ubiquitous Multi-device Interaction. InThe 25th Annual International Conference on Mobile Computing and Networking(Los Cabos, Mexico)(MobiCom ’19). Association for Computing Machinery, New York, NY, USA, Articl...

    work page doi:10.1145/3300061.3345443 2019

  51. [51]

    Openai. 2025. ChatGPT Pricing. https://openai.com/chatgpt/pricing/

    work page 2025

  52. [52]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  53. [53]

    Samsung. 2021. Samsung SmartThings Integrates Matter into Ecosystem, Bringing Matter Device Control to Multiple Samsung Products. https://www.samsungmobilepress.com/articles/samsung- smartthings-integrates-matter-into-ecosystem-bringing-matter- device-control-to-multiple-samsung-products

    work page 2021

  54. [54]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538 [cs.LG] https://arxiv.org/abs/1701.06538

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  55. [55]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang

    work page

  56. [56]

    InInternational Conference on Machine Learn- ing

    Flexgen: High-throughput generative inference of large language models with a single gpu. InInternational Conference on Machine Learn- ing. PMLR, 31094–31116

    work page

  57. [57]

    Karan Singh. 2025. Tesla’s MCU Infotainment Com- puter vs FSD Computer: What’s the Difference? https://www.notateslaapp.com/news/2963/tesla

    work page 2025

  58. [58]

    Peter Steinberger. 2026. OpenClaw. https://openclaw.ai/

    work page 2026

  59. [59]

    Ki Tae Kim, Seok-Ju Im, and Eui-Young Chung. 2025. AsymGroup: Asymmetric Grouping and Communication Optimization for 2D Ten- sor Parallelism in LLM Inference.IEEE Access13 (2025), 120591–120602. doi:10.1109/ACCESS.2025.3587387

    work page doi:10.1109/access.2025.3587387 2025

  60. [60]

    Chen Tang, Haoyu Zhai, Kai Ouyang, Zhi Wang, Yifei Zhu, and Wenwu Zhu. 2022. Arbitrary Bit-width Network: A Joint Layer-Wise Quantiza- tion and Adaptive Inference Approach. InProceedings of the 30th ACM International Conference on Multimedia(Lisboa, Portugal)(MM ’22). Association for Computing Machinery, New York, NY, USA, 2899–2908. doi:10.1145/3503161.3548001

    work page doi:10.1145/3503161.3548001 2022

  61. [61]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexan- dre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Ge- offrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  62. [62]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InProceedings of the 31st International Con- ference on Neural Information Processing Systems(Long Beach, Cali- fornia, USA)(NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010

    work page 2017

  63. [63]

    Xiong Wang, Linghe Kong, Zucheng Wu, Long Cheng, Chenren Xu, and Guihai Chen. 2020. SLoRa: towards secure LoRa communications with fine-grained physical layer features. InProceedings of the 18th Conference on Embedded Networked Sensor Systems(Virtual Event, Japan)(SenSys ’20). Association for Computing Machinery, New York, 15 NY, USA, 258–270. doi:10.1145...

    work page doi:10.1145/3384419.3430770 2020

  64. [64]

    Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. In18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 911–927. https://www.usenix.org/conference/osdi24/ presentation/wu-bingyang

    work page 2024

  65. [65]

    Xiaodong Wu, Ran Duan, and Jianbing Ni. 2024. Unveiling security, privacy, and ethical concerns of ChatGPT.Journal of Information and Intelligence2, 2 (2024), 102–115. doi:10.1016/j.jiixd.2023.10.007

    work page doi:10.1016/j.jiixd.2023.10.007 2024

  66. [66]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models. InProceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En...

    work page 2023

  67. [67]

    Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv:2406.06282 [cs.LG] https://arxiv.org/abs/2406. 06282

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  68. [68]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv:1905.07830 [cs.CL] https://arxiv.org/abs/1905.07830

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  69. [69]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu

    work page

  70. [70]

    TinyLlama: An Open-Source Small Language Model

    TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385 [cs.CL] https://arxiv.org/abs/2401.02385

    work page internal anchor Pith review Pith/arXiv arXiv

  71. [71]

    PersonaAgent: Bridging Memory and Action for Personalized LLM Agents

    Weizhi Zhang, Xinyang Zhang, Chenwei Zhang, Liangwei Yang, Jingbo Shang, Zhepei Wei, Henry Peng Zou, Zijie Huang, Zhengyang Wang, Yifan Gao, Xiaoman Pan, Lian Xiong, Jingguo Liu, Philip S. Yu, and Xian Li. 2025. PersonaAgent: When Large Language Model Agents Meet Personalization at Test Time. arXiv:2506.06254 [cs.AI] https: //arxiv.org/abs/2506.06254

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  72. [72]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levi...

    work page 2023

  73. [73]

    Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Au- tomating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. arXiv:2201.12023 [cs.LG] https://arxiv.org/abs/2201.12023 16

    work page internal anchor Pith review Pith/arXiv arXiv 2022