Pith. sign in

REVIEW 1 major objections 1 minor 53 references

Edge-cloud LLM inference cut latency by 46% and downlink data by 67% while keeping output quality near cloud-only.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:38 UTC pith:O2K4MTNO

load-bearing objection Solid systems engineering under an unsupported privacy framing; the paper is a latency optimization with honest disclaimers, not a privacy-preserving scheme. the 1 major comments →

arxiv 2607.13093 v1 pith:O2K4MTNO submitted 2026-07-14 cs.CR cs.AI

Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

Yi Li , Chen Li , Jiexiong Liu This is my paper
classification cs.CR cs.AI
keywords Edge-cloud collaborative inferenceLarge language modelsKV cache authorizationDistributed vocabulary projectionSpeculative decodingLanguage-adaptive maskingPrivacy-aware inferenceSplit inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes splitting large-language-model inference between a user's device and a cloud service so that the device keeps control over privacy-sensitive decisions while the cloud does the heavy compute. The central claim is that this split reduces per-token latency by up to 46.1% and downlink payloads by up to 67.4% compared to naive split inference, while matching the full cloud model's output quality within about 2% next-token agreement. A sympathetic reader would care because it offers a practical, deployable path for running capable LLMs on consumer CPUs, GPUs, and embedded devices without sending raw prompts or full output distributions to the cloud. The paper backs this with a prototype evaluated across three device profiles under controlled network conditions.

Core claim

The paper proposes an endpoint-authorized collaborative inference framework in which the user's device computes embeddings, a low-dimensional slice of the language-model head, speculative draft tokens, and final token sampling, while the cloud runs the decoder layers, maintains the KV cache under endpoint control, verifies draft tokens, and computes the high-dimensional slice of the vocabulary projection. By splitting the hidden state into a small endpoint portion (96 of 4096 dimensions) and a large cloud portion (4000 dimensions), the endpoint can fuse partial logits locally, apply language-adaptive vocabulary masking, and avoid transmitting the full 150,000-dimension logits vector during l

What carries the argument

The central mechanism is the distributed vocabulary projection: the LM head's weight matrix is split along the hidden dimension so that the endpoint computes z_end = h_end * W_end and the cloud computes z_cloud = h_cloud * W_cloud + b, then the endpoint sums them element-wise. This, combined with endpoint-owned KV-cache authorization (flagCache, LCache) and language-adaptive masking (only the subset of vocabulary dimensions V_l is transmitted), is what halves latency and shrinks downlink payloads.

Load-bearing premise

The load-bearing assumption is that keeping the LoRA-adapted local modules and cache-control decisions on the endpoint meaningfully reduces an honest-but-curious cloud's ability to reconstruct user input from the hidden states it is allowed to see, even though the paper does not measure this and concedes that strong adversaries may still extract information.

What would settle it

Run the cloud-side tensors (4000-dim hidden states, partial logits, language masks, and draft traces) through a representation-inversion attack such as Text Revealer on the same 7B model. If the attacker can reconstruct readable prompts from those tensors with acceptable quality, then the 'privacy-aware' claim collapses to a latency/bandwidth optimization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the framework is adopted, consumer devices with modest compute could run capable LLMs interactively, since the cloud takes over the heavy decoder work while the endpoint keeps local control.
  • Language-restricted generation (e.g., Chinese-only or English-only) can cut cloud-to-device traffic by more than half, reducing bandwidth cost and perceived response time.
  • Endpoint-local LoRA modules and KV-cache authorization provide a practical, non-cryptographic layer of privacy protection, though not a formal guarantee.
  • The split-head design is compatible with quantization, ONNX Runtime, and various device profiles, suggesting it can be deployed on CPUs, GPUs, and embedded ARM devices.
  • The framework's speculative decoding with cloud-side verification preserves the target model's generation quality while speeding up interactive dialogue.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The claimed privacy benefit is weaker than the title suggests: the cloud still receives 4000 of 4096 hidden-state dimensions, partial logits, language masks, and draft-token traces, and §8 concedes that hidden states can leak input information. A strong adversary with access to those tensors might reconstruct prompts using known inversion attacks, so the real contribution may be primarily efficien
  • The latency gains depend on the prototype's 20 ms RTT and 100 Mbps bandwidth; real-world mobile networks with higher jitter or lower bandwidth could reduce the block-streaming benefit on CPU endpoints.
  • A testable extension is to measure how much of the privacy benefit survives when the cloud uses a pretrained inversion model (like the cited Text Revealer) on the observed hidden states; the paper lists this as future work.
  • The design could be extended to smaller models or larger vocabularies: the split head and language masking should generalize, but the optimal split dimension D_end may need re-tuning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper proposes an edge–cloud collaborative inference framework for LLMs in which the endpoint computes embeddings, controls KV-cache authorization, runs a lightweight local projection of the LM head, performs speculative decoding, and fuses partial logits, while the cloud executes the decoder layers and the high-dimensional part of the vocabulary projection. Communication is protected with AES-GCM, logits are clipped to language-specific vocabulary subsets, and endpoint-local LoRA/draft weights are kept on the device. The evaluation reports per-token latency reductions of up to 46.1% over a 'naive split inference' baseline and downlink payload reductions of up to 67.4% versus full-logits transfer, with next-token agreement of 98.6% relative to the full cloud model. The paper explicitly disclaims formal differential privacy or secure-computation guarantees and frames the privacy benefit as a practical exposure-reduction mechanism under an honest-but-curious cloud.

Significance. If the efficiency results are reproducible, the paper offers a useful systems contribution: the split vocabulary projection is exact by linearity, the ablation study in §7.7 gives sensible direction-of-effect results, and the latency/bandwidth measurements are concrete under a controlled network. The paper is also commendably explicit in §8 about what the design does not provide. However, the central 'privacy-aware' claim is not established. The cloud receives the token embedding h0 at the very first message, and the threat model grants the cloud access to decrypted feature tensors and the full model weights. Since the cloud has the token-embedding table, it can invert h0 into token indices by nearest-neighbor search, which would recover the prompt. The paper's own §8 lists inversion-attack evaluation as future work, but this is not a missing optional experiment: it is the core of the privacy claim. The paper therefore currently reduces to a latency/bandwidth optimization with a privacy-control interface, and the framing needs either a concrete privacy evaluation or a significant softening.

major comments (1)
  1. [§7.3, Tables 3–8] The evaluation is hard to assess without a named target model or released artifacts. §7.3 says the target model is 'instantiated as a 7B-class decoder-only Transformer' with hyperparameters chosen to match the split design, the vocabulary is 150k, and the endpoint processes 96 of 4096 dimensions. No model name, checkpoint, or code is provided. Latency and quality numbers for an unnamed model are not independently reproducible, and the claim that the model is 'instantiated' to match the design raises concern that the configuration was tuned to produce favorable results. Please identify an existing open-weight model (or release the model configuration and code) and report results on that fixed model.
minor comments (1)
  1. [§7.2, Table 3] The 'Endpoint-only lightweight' baseline is marked '–' on the CPU profile. A one-sentence explanation of why local-only inference is infeasible on that profile would help the reader interpret the comparison.

Circularity Check

0 steps flagged

No circularity: the split-logits identity is an exact decomposition used as an implementation, and the efficiency claims are measured outcomes.

full rationale

The paper's derivation chain is an engineering decomposition rather than a fitted result. Eqs. (10)-(13) split the LM-head projection by hidden dimension; z = z_end + z_cloud holds exactly by linearity, and the paper presents this as an implementation identity, not as an empirical prediction. Eqs. (14)-(15) are standard masked softmax, and the language-mode payload reduction is the measured accounting identity of Eq. (21), not a parameter fitted and re-predicted. Latency and throughput claims come from a controlled prototype (Tables 3-7), and the small quality degradation is checked externally on MMLU, GSM8K, and HumanEval (Table 8). The only by-construction element is the §7.2 definition of "naive split inference" without the components the framework adds, which makes some of the gain expected; this is a comparison-fairness caveat, not circular reasoning. §8 explicitly concedes the privacy limitation ("Hidden states can still carry information about the input, especially against strong adversaries with auxiliary knowledge or model access") and lists inversion-attack evaluation as future work; this is a correctness risk (the cloud receives h0 from Eq. (1)), not a circular step. No self-citation or imported uniqueness theorem is load-bearing. Therefore the derivation is self-contained and non-circular.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 1 invented entities

The framework's central efficiency claims rest on tuned engineering parameters (Dend, N, LoRA ranks, vocab fractions, bit width, block count) and a controlled network environment; the privacy narrative rests on an unmeasured threat-model assumption about inversion resistance, which the authors themselves flag. No new physical or mathematical entities are postulated beyond a protocol control tuple.

free parameters (7)
  • Endpoint split dimension Dend = 96 (sweep: 48-512)
    Determines how much of the LM-head projection runs on-device; tuned in §7.8 for latency-throughput balance; headline numbers in Tables 3-4 use the tuned value.
  • Speculative draft length N = 5 (GPU), 2 (edge)
    Sets how many draft tokens are proposed per round; tuned in §7.9 on the acceptance-rate/latency trade-off; headline gains assume tuned values.
  • LoRA ranks = 4 (CPU), 8 (GPU)
    Rank of local low-rank matrices (Eq 20); chosen in Table 2 without a sensitivity study; affects local-module fidelity and the privacy narrative.
  • Language vocabulary fractions = Chinese-only 40%, English-only 30% of V
    Fraction of the 150k vocabulary retained per language mode; chosen in §7.3 without stated justification; directly sets the 56-67% payload-compression numbers.
  • Quantization bit width b = 8 bits
    INT8 for tensors and clipped logits; chosen in §7.3; the quality loss in Table 8 is partly attributed to it.
  • CPU streamed block count Nb = 4
    Number of column blocks in Eq 3; chosen in Table 2 for block-wise streaming overlap.
  • Evaluation network RTT/bandwidth = 20 ms / 100 Mbps
    Controlled-network setting (§7.3); the magnitude of reported latency reductions depends on interaction-count savings versus this RTT.
axioms (6)
  • domain assumption Honest-but-curious cloud follows the protocol and does not collude with network attackers
    Stated threat model §4; the entire privacy framing is void if the cloud actively keeps or misuses decrypted tensors or abuses session-cache retention.
  • standard math Decoder-only Transformer forward pass with softmax attention is the correct generative model
    Eqs 4-8 assume standard Q/K/V attention and FFN; no exotic architecture is required.
  • domain assumption Language identifier ℓ faithfully maps to a vocabulary subset Vℓ containing all tokens the target model would sample
    §5.4 and §6.2; the -∞ masking in Eq 14 is quality-neutral only if Vℓ covers the true generation distribution.
  • ad hoc to paper Keeping LoRA/draft weights on the endpoint materially raises the cost of reconstructing user content from cloud-visible tensors
    §4 goal 3 and §5.6 assert this; §8 admits hidden states carry input information and that inversion resistance was not evaluated. This is the load-bearing, unmeasured privacy premise.
  • standard math AES-GCM with correctly managed keys and nonces protects the transport channel
    Eq 19, §5.6; standard cryptographic assumption.
  • domain assumption INT8 quantization and endpoint-side head-split keep sampling approximately target-preserving
    §5.5 states the workflow is not strictly distribution-preserving; Table 8 measures 98.6% agreement but no statistical significance is reported.
invented entities (1)
  • Endpoint-authorized cache-control tuple c = (flagCache, LCache, dtype, ℓ) no independent evidence
    purpose: Protocol element enabling the endpoint to decide whether and how much historical KV-cache the cloud may reuse; the 'privacy-aware' control plane of the framework.
    Implemented in the prototype and its no-auth ablation (Table 5) shows no latency impact, so its sole claimed benefit is the asserted privacy control; no independent measurement shows it reduces actual exposure, and §5.3 notes it governs usage, not cloud-side retention.

pith-pipeline@v1.3.0-alltime-deepseek · 11925 in / 21423 out tokens · 213570 ms · 2026-08-02T06:38:04.609840+00:00 · methodology

0 comments
read the original abstract

On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache. Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking and sample target tokens. All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped and embedded devices via optimized streaming, batching and quantized ONNX deployment. Evaluations demonstrate that the framework reduces per-token latency by up to 46.1\% and downlink payloads by up to 67.4\% over baseline split inference, retaining comparable performance to full cloud inference.

Figures

Figures reproduced from arXiv: 2607.13093 by Chen Li, Jiexiong Liu, Yi Li.

Figure 1
Figure 1. Figure 1: Overview of the edge–cloud collaborative LLM inference framework. The endpoint keeps privacy-sensitive [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Autoregressive generation loop with endpoint-controlled KV-cache usage, split logits computation, language [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 6 linked inside Pith

  1. [1]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS Symposium on Operating Systems Principles, 2023. 12 Efficient and Privacy-Aware Edge–Cloud LLM Inference

    2023

  2. [2]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems, 2022

    2022

  3. [3]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. InInternational Conference on Learning Representations, 2024

    2024

  4. [4]

    ORCA: A distributed serving system for transformer-based generative models

    Gyeong-In Yu, Jeongseok Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. ORCA: A distributed serving system for transformer-based generative models. InUSENIX Symposium on Operating Systems Design and Implementation, 2022

    2022

  5. [5]

    DeepSpeed-Inference: Enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Conglong Luo, Arash Elnawany, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. DeepSpeed-Inference: Enabling efficient inference of transformer models at unprecedented scale. InInternational Conference for High Performance Computing, Networking, Storage and Analysis, 2022

    2022

  6. [6]

    Neurosurgeon: Collaborative intelligence between the cloud and mobile edge

    Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. InProceedings of the Twenty- Second International Conference on Architectural Support for Programming Languages and Operating Systems, 2017

    2017

  7. [7]

    Split computing and early exiting for deep learning applications: Survey and research challenges.ACM Computing Surveys, 2022

    Yoshitaka Matsubara, Marco Levorato, and Francesco Restuccia. Split computing and early exiting for deep learning applications: Survey and research challenges.ACM Computing Surveys, 2022

    2022

  8. [8]

    DNN surgery: Accelerating DNN inference on the edge through layer partitioning.IEEE Transactions on Parallel and Distributed Systems, 2023

    Hao Liang, Qiyang Sang, Chuan Hu, Dapeng Cheng, et al. DNN surgery: Accelerating DNN inference on the edge through layer partitioning.IEEE Transactions on Parallel and Distributed Systems, 2023

    2023

  9. [9]

    Pipeedge: Pipeline parallelism for large-scale model inference on heterogeneous edge devices

    Yinan Hu, Connor Imes, Xiaoang Zhao, Suhas Kundu, et al. Pipeedge: Pipeline parallelism for large-scale model inference on heterogeneous edge devices. InEuromicro Conference on Digital System Design / Software Engineering and Advanced Applications, 2022

    2022

  10. [10]

    Hybrid SLM and LLM for edge-cloud collaborative inference

    Zeyu Hao, Hao Jiang, Shaoxiang Jiang, Jiajun Ren, and Ting Cao. Hybrid SLM and LLM for edge-cloud collaborative inference. InACM MobiFoundation Workshop, 2024

    2024

  11. [11]

    Jupiter: Fast and resource-efficient collaborative inference of generative LLMs on edge devices

    Sen Ye, Bin Ouyang, Laizhong Zeng, Tian Qian, Xiangjie Chu, et al. Jupiter: Fast and resource-efficient collaborative inference of generative LLMs on edge devices. InIEEE INFOCOM, 2025

    2025

  12. [12]

    DSSD: Efficient edge-device LLM deployment and collaborative inference via distributed split speculative decoding.arXiv preprint arXiv:2507.12000, 2025

    Jiayi Ning, Chenning Zheng, and Tong Yang. DSSD: Efficient edge-device LLM deployment and collaborative inference via distributed split speculative decoding.arXiv preprint arXiv:2507.12000, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    CE-CoLLM: Efficient and adaptive large language models through cloud-edge collaboration

    Haojie Jin and Yufei Wu. CE-CoLLM: Efficient and adaptive large language models through cloud-edge collaboration. InIEEE International Conference on Web Services, 2025

    2025

  14. [14]

    Edgeshard: Efficient LLM inference via collaborative edge computing.IEEE Internet of Things Journal, 2024

    Mingyuan Zhang, Xiang Shen, Jiannong Cao, Zhenyu Cui, et al. Edgeshard: Efficient LLM inference via collaborative edge computing.IEEE Internet of Things Journal, 2024

    2024

  15. [15]

    Collaborative inference and learning between edge SLMs and cloud LLMs: A survey.arXiv preprint arXiv:2507.16731, 2025

    Shuai Li, Hao Wang, Wenkang Xu, Rongxin Zhang, Song Guo, Jingdong Yuan, et al. Collaborative inference and learning between edge SLMs and cloud LLMs: A survey.arXiv preprint arXiv:2507.16731, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, 2023

    2023

  17. [17]

    Accelerating large language model decoding with speculative sampling, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023

    2023

  18. [18]

    Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rui Chen, and Zhihao Jia. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. In ACM SIGOPS Symposium on Operating Systems Principles, 2024

    2024

  19. [19]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding

    Zhenyu Chen, Avner May, Roman Svirschevski, Yuhsiang Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. InAdvances in Neural Information Processing Systems, 2024

    2024

  20. [20]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. InInternational Conference on Machine Learning, 2024

    2024

  21. [21]

    EAGLE-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. InConference on Empirical Methods in Natural Language Processing, 2024

    2024

  22. [22]

    Break the sequential dependency of LLM inference using lookahead decoding.arXiv preprint, 2024

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding.arXiv preprint, 2024. 13 Efficient and Privacy-Aware Edge–Cloud LLM Inference

    2024

  23. [23]

    Reddi, and Felix Yu

    Yingqian Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Sashank J. Reddi, and Felix Yu. Distillspec: Improving speculative decoding via knowledge distillation. In International Conference on Learning Representations, 2024

    2024

  24. [24]

    Layerskip: Enabling early exit inference and self-speculative decoding

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. InAnnual Meeting of the Association for Computational Linguistics, 2024

    2024

  25. [25]

    Draft & verify: Lossless large language model acceleration via self-speculative decoding

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. InAnnual Meeting of the Association for Computational Linguistics, 2024

    2024

  26. [26]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. InInternational Conference on Learning Representa- tions, 2022

    2022

  27. [27]

    QLoRA: Efficient finetuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. InAdvances in Neural Information Processing Systems, 2023

    2023

  28. [28]

    Splitlora: A split parameter- efficient fine-tuning framework for large language models.arXiv preprint arXiv:2407.00952, 2024

    Zhiwei Lin, Xinyi Hu, Yifan Zhang, Zhenyu Chen, Zihao Fang, Xiang Chen, et al. Splitlora: A split parameter- efficient fine-tuning framework for large language models.arXiv preprint arXiv:2407.00952, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Improving LoRA in privacy-preserving federated learning

    Yansong Sun, Zhenyu Li, Yue Li, and Bolin Ding. Improving LoRA in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    On the implicit relation between low-rank adaptation and differential privacy

    Shima Malekmohammadi and Golnoosh Farnadi. On the implicit relation between low-rank adaptation and differential privacy. InNeurIPS Workshop on Privacy in Machine Learning, 2024

    2024

  31. [31]

    Is split learning privacy-preserving for fine-tuning large language models?IEEE Transactions on Big Data, 2024

    Dongqi Yao and Bin Li. Is split learning privacy-preserving for fine-tuning large language models?IEEE Transactions on Big Data, 2024

    2024

  32. [32]

    SLDP-LoRA: A privacy-preserving split learning framework with low-rank adaptation

    Yicheng Huang, Jianlong Yin, Zihao Tan, Qiang Wang, et al. SLDP-LoRA: A privacy-preserving split learning framework with low-rank adaptation. InIEEE International Conference on Network Security and Engineering, 2025

    2025

  33. [33]

    Split-and-denoise: Protect large language model inference with local differential privacy.arXiv preprint arXiv:2310.09130, 2023

    Pinghui Mai, Rui Yan, Zeyu Huang, Yifei Yang, and Yanmin Pang. Split-and-denoise: Protect large language model inference with local differential privacy.arXiv preprint arXiv:2310.09130, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Santos, et al

    Muslum Ugurbil, Dimitris Mouris, Miguel B. Santos, et al. Fission: Distributed privacy-preserving large language model inference. InIEEE European Symposium on Security and Privacy, 2025

    2025

  35. [35]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. InInternational Conference on Learning Representations, 2023

    2023

  36. [36]

    AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Guanxiao Xiao, et al. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. InConference on Machine Learning and Systems, 2024

    2024

  37. [37]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. InInternational Conference on Machine Learning, 2023

    2023

  38. [38]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhiyuan Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. InInternational Conference on Learning Representations, 2024

    2024

  39. [39]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations, 2024

    2024

  40. [40]

    Efficient LLM inference on CPUs

    Hao Shen, Hui Chang, Bo Dong, Yue Luo, and Hao Meng. Efficient LLM inference on CPUs. InEfficient Deep Learning: Methods, Systems, and Applications. Springer, 2025

    2025

  41. [41]

    H2O: Heavy-hitter oracle for accurate KV cache compression

    Zhenyu Zhang, Yao Sheng, Tianyi Jin, Chi Chen, Tianqi Liang, et al. H2O: Heavy-hitter oracle for accurate KV cache compression. InAdvances in Neural Information Processing Systems, 2023

    2023

  42. [42]

    Streamingllm: Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Streamingllm: Efficient streaming language models with attention sinks. InInternational Conference on Learning Representations, 2024

    2024

  43. [43]

    Recommendation for block cipher modes of operation: Ga- lois/counter mode (gcm) and gmac

    National Institute of Standards and Technology. Recommendation for block cipher modes of operation: Ga- lois/counter mode (gcm) and gmac. NIST Special Publication 800-38D, 2007

    2007

  44. [44]

    BOLT: Privacy-preserving, accurate and efficient inference for transformers

    Qipeng Pang, Jingyi Zhu, Hendrik Möllering, Wenqiang Zheng, et al. BOLT: Privacy-preserving, accurate and efficient inference for transformers. InIEEE Symposium on Security and Privacy, 2024. 14 Efficient and Privacy-Aware Edge–Cloud LLM Inference

    2024

  45. [45]

    THE-X: Privacy-preserving transformer inference with homomorphic encryption

    Tianyu Chen, Hang Bao, Shaohan Huang, Li Dong, Binxing Jiao, et al. THE-X: Privacy-preserving transformer inference with homomorphic encryption. InFindings of the Association for Computational Linguistics, 2022

    2022

  46. [46]

    Xing, et al

    Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P. Xing, et al. MPCFormer: Fast, performant and private transformer inference with MPC.arXiv preprint, 2022

    2022

  47. [47]

    Secureinfer: Heterogeneous TEE-GPU architecture for privacy- critical LLM tensors.arXiv preprint arXiv:2510.19979, 2025

    Tashin Nayan, Zhenkai Zhang, and Ruizhe Sun. Secureinfer: Heterogeneous TEE-GPU architecture for privacy- critical LLM tensors.arXiv preprint arXiv:2510.19979, 2025

    arXiv 2025

  48. [48]

    ONNX Runtime: Cross-platform, high performance machine learning inferencing and training accelerator.https://onnxruntime.ai/

    Microsoft. ONNX Runtime: Cross-platform, high performance machine learning inferencing and training accelerator.https://onnxruntime.ai/. Accessed 2026

    2026

  49. [49]

    Text revealer: Private text reconstruction via model inversion attacks against transformers.arXiv preprint arXiv:2209.10505, 2022

    Ruiyi Zhang, Shuichiro Hidano, and Farinaz Koushanfar. Text revealer: Private text reconstruction via model inversion attacks against transformers.arXiv preprint arXiv:2209.10505, 2022

    Pith/arXiv arXiv 2022

  50. [50]

    ML-Doctor: Holistic risk assessment of inference attacks against machine learning models

    Yugeng Liu, Rui Wen, Xiang He, Ahmed Salem, Zhikun Zhang, et al. ML-Doctor: Holistic risk assessment of inference attacks against machine learning models. InUSENIX Security Symposium, 2022

    2022

  51. [51]

    Privacy side channels in machine learning systems

    Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, et al. Privacy side channels in machine learning systems. InUSENIX Security Symposium, 2024

    2024

  52. [52]

    Secure multi-party computation for machine learning: A survey.IEEE Communications Surveys & Tutorials, 2024

    Irene Zhou, Farbod Tofigh, Massimo Piccardi, Mehran Abolhasan, et al. Secure multi-party computation for machine learning: A survey.IEEE Communications Surveys & Tutorials, 2024

    2024

  53. [53]

    Min Du, Xiang Yue, Sherman S. M. Chow, Tianhao Wang, Chenguang Huang, et al. DP-Forward: Fine-tuning and inference on language models with differential privacy in forward pass. InACM SIGSAC Conference on Computer and Communications Security, 2023. 15

    2023