Pith. sign in

REVIEW 4 major objections 4 minor 26 references

Towards Confidential and Efficient LLM Inference with Dual Privacy Protection

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CMIF claims that putting only the embedding layer in a client TEE and sanitizing input words with Report-Noisy-Max gives LLM inference both differential privacy and low overhead.

desk verdict A sensible TEE/GPU split with a broken DP proof: the privacy guarantee in §4.4 does not follow from the paper's own definitions. read the letter →

arxiv 2509.09091 v1 pith:TD3L6U2M submitted 2025-09-11 cs.CR cs.AI

Honglan Yu , Yibin Wang , Feifei Dai , Dong Liu , Haihui Fan , Xiaoyan Gu This is my paper · ORCID
classification cs.CRcs.AI
keywords LLMinferencedifferentialprivacytrustedexecutionenvironmenttextsanitizationReport-Noisy-MaxGPUaccelerationconfidentialcomputingTEE-GPUpartition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that private LLM inference can be both confidential and efficient if the embedding layer runs in a client-side trusted execution environment and every sensitive input word is replaced by a semantically similar substitute selected with Laplace noise. This avoids the per-layer TEE-GPU communication that slows prior partition-based systems, and it avoids the semantic damage of blanket DP noise. The authors report extra overhead of roughly 29–38% over an unprotected CPU+GPU baseline on Llama2-7B and Llama3-8B, with the sanitizer itself responsible for less than 1%. The central technical claim is that the replacement rule satisfies epsilon-differential privacy (or (ln(1/p)+epsilon)-DP) because the normalized cosine score has sensitivity 1. A sympathetic reader would care because this is a concrete recipe for deploying LLMs on untrusted GPU servers without exposing raw user text.

What carries the argument

The load-bearing object is the Report-Noisy-Max (RNM) sanitization mechanism F, defined as: for each word xi, compute normalized cosine distance to top-k candidates Si, add independent Laplace noise Lap(1/epsilon) to each score, and output the candidate with the largest noisy score. The same mechanism doubles as the privacy device (RNM is epsilon-DP when the score is Delta-sensitive) and as the semantic-preserving device (candidates are nearest neighbors in the model's own embedding space). The other half of the framework is the placement of the embedding layer inside the client TEE, which lets the sanitizer run in the enclave and keeps word vectors private while all decoder layers execute o

What would settle it

Compute, for a real embedding space, the true sensitivity of the score function used by F by enumerating pairs of neighboring inputs and checking the maximum absolute change in normalized cosine distance across all possible candidate sets; if any pair exceeds 1, or if the ratio Pr(F(x)=x̂)/Pr(F(x')=x̂) exceeds e^epsilon for some x̂, the paper's central privacy claim is refuted.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that CMIF—a Confidential and efficient Model Inference Framework—can protect user data with a sanitization mechanism and model confidentiality by TEE-shielded deployment, without returning to the TEE for decryption during inference. Specifically, the embedding layer is placed in the client enclave; for each sensitive token, the mapping function F builds a top-k candidate set from cosine similarity, normalizes the distances so the score's sensitivity is 'equalized to 1', adds Lap(1/epsilon) noise to each score, and returns the argmax. Non-sensitive tokens are also replaced with probability p. The formal analysis claims epsilon-DP when both neighborin

Load-bearing premise

The privacy guarantee rests on the normalized cosine distance having sensitivity exactly 1 and on the candidate substitute set being the same for neighboring inputs; since the candidate set is defined per input word, the analysis does not establish this, and if either part fails the claimed epsilon-DP bound does not hold.

Editorial extensions

If this is right

  • CMIF eliminates the need for per-layer encryption and TEE-GPU switching during decoder execution, since the server only receives sanitized text.
  • The sanitizer's own time cost is about 0.6% of inference time, so the dominant remaining overhead is the enclave boundary itself, not the DP mechanism.
  • Compared to SANTEXT+ and CUSTEXT+, the RNM sanitizer improves classification accuracy at the same nominal epsilon on SST-2 and QNLI with BERT, and keeps Llama3-8B within 0.15% of non-private performance on IFEval after fine-tuning on sanitized DialogSUM data.
  • The small size of the embedding layer (roughly 1.9% of Llama2-7B parameters) makes client-side TEE deployment feasible even on memory-constrained devices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's per-input definition of the candidate set Si is in tension with the standard RNM requirement that the outcome set be fixed across neighboring inputs; a rigorous DP claim would require either a global candidate set or a separate sensitivity argument for changing Si. This is an editorial observation, not a claim the paper makes.
  • If the sensitivity concern is repaired, the RNM sanitizer becomes a drop-in preprocessing module for any black-box LLM API, since it needs only the embedding table, which could be obtained from a public model or a minimal client-side copy.
  • The asymmetric treatment of sensitive vs non-sensitive words gives an explicit trade-off: lowering p increases fidelity but adds ln(1/p) to the privacy loss, so practitioners can select p based on deployment-specific privacy budgets without retraining.
  • The same design pattern—small protected embedding layer plus noisy token substitution—could transfer beyond SGX to other confidential-computing backends or to on-device mobile TEEs, because the protected component is small and self-contained.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CMIF, a framework for private LLM inference that keeps the embedding layer inside a client-side TEE and offloads the remaining layers to a GPU server. To protect user inputs, it replaces sensitive (low-frequency) tokens using an RNM-style mechanism that adds Laplace noise to normalized cosine scores over a per-token top-k candidate set. The authors claim epsilon-DP for sensitive tokens, (ln(1/p)+epsilon)-DP when non-sensitive tokens are randomized with probability p, and lower inference overhead than purely TEE-based approaches. Experiments on BERT, Llama2-7B, and Llama3-8B report utility and latency results, and a code link is provided.

Significance. If the privacy proof were valid, the design would be an interesting combination: a TEE for the embedding layer plus DP sanitization to avoid per-token TEE round trips. The paper includes a clear experimental setup, measurements of overhead, and a public code link. However, the central DP analysis is not valid as written: the candidate outcome set and normalization are input-dependent, so the standard Report-Noisy-Max theorem does not apply. Since differential privacy is the paper's primary formal contribution, the claimed guarantees do not support the system's privacy properties. The utility and efficiency measurements are internally plausible but are presented at privacy levels that are not established.

major comments (4)
  1. [§3.2 and §4.4, Eq. (4)] The outcome space S_i is defined as the top-k most similar words to each specific input w_i. Thus S_i changes when the input changes. In Report-Noisy-Max (Definition 2 and Theorem 1), Y is a fixed finite outcome set. For any output x* in S_i \ S_{i'}, Pr(F(x_i)=x*) > 0 while Pr(F(x_i')=x*) = 0, so the probability ratio in Eq. (4) is unbounded. Consequently, Case 1's epsilon-DP claim is unproven and in fact false for the mechanism as specified. This is a direct mismatch between the theorem's assumptions and the construction, not a minor presentational gap.
  2. [§3.2, Eq. (3)] The paper asserts that normalizing by cos_min and cos_max 'equalizes' sensitivity to 1, but these constants are computed from S_i, which depends on x_i. The function u is not a fixed score function d(D, y) as required by Definition 2; it is a family of functions parameterized by the input. The global sensitivity over neighboring inputs is not shown to be 1, and the Laplace scale Lap(1/epsilon) is therefore unjustified. A correct proof would need to establish sup_{x~x', y in a fixed common Y} |u_x(y) - u_{x'}(y)| <= 1 for a fixed Y, which is not supplied and is incompatible with the current input-dependent construction.
  3. [§4.4, Case 2] The mechanism for non-sensitive words is a mixture: with probability p it runs RNM, otherwise it outputs the original word. The claimed (ln(1/p)+epsilon)-DP bound is not derived. For any output y that lies in S_i but not in S_{i'} and also y != x' (the original non-sensitive word), Pr(F(x_i)=y) > 0 while Pr(F(x_i')=y) = 0, so the probability ratio is unbounded and no finite DP guarantee holds. The mixture analysis is missing and the claimed bound is not generally valid.
  4. [§3.1 and Abstract] The title and abstract claim dual protection of user data and model parameters. However, only the embedding layer is placed in the client-side TEE; the paper itself notes this is about 1.9% of Llama2-7B parameters. The remaining layers are sent in plaintext to the GPU server, which under the stated semi-honest cloud-provider threat model can inspect them. The paper does not explain what adversary the model-parameter protection is intended to defend against, or why the bulk of the model on the GPU server is considered confidential. This unsupported claim is load-bearing for the paper's framing.
minor comments (4)
  1. [§3.2] Typo: 'Report-Nosiy-Max' should be 'Report-Noisy-Max'.
  2. [§2.3 / §4.4] The neighboring relation for text inputs is not formally defined. The paper says 'x, x' are defined as differing in vocabulary V', which is ambiguous; Definition 1 requires a precise notion of neighboring inputs (e.g., one token substitution of equal-length sequences).
  3. [Table 1] The 'Random' row appears incomplete: no QNLI values are reported, and the table layout is hard to read. Please also state whether the reported numbers are averaged over multiple runs and what variance is observed.
  4. [§2.3, Definition 1] Sensitivity is defined using the Frobenius norm, while the Report-Noisy-Max guarantee in Theorem 1 typically uses L1 or L_infinity sensitivity for scalar score functions. This mismatch should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the DP claim rests on an unproven sensitivity normalization, which is a correctness risk, not a reduction to the paper's own inputs.

full rationale

I walked the derivation chain of CMIF and found no circular step of the kinds enumerated. The sanitization mechanism uses a standard Report-Noisy-Max (RNM) construction, and Theorem 1 is cited from Dwork and Roth's external foundational treatment of DP, not from the authors' own prior work. No parameter is fitted to a data subset and then relabeled as a prediction: the reported accuracy figures are direct measurements on SST-2, QNLI, DialogSUM, and IFEval, and the candidate vocabulary comes from static model embeddings. The central privacy result in Section 4.4 does contain a load-bearing assertion that the normalized cosine distance u has sensitivity exactly 1 and that the top-k set S_i behaves as a fixed outcome space; as the paper itself defines S_i relative to the input x_i and normalizes using cos_max and cos_min computed from that input-dependent candidate list, the claimed sensitivity-1 premise is not proven and may fail under the paper's own definitions. That is a serious correctness gap, but it is an unsupported or false modeling assumption rather than a circular derivation: the privacy claim is not equivalent by construction to an input of the analysis, nor is it justified by a self-citation chain. The paper would be circular only if, for example, the DP guarantee were derived from the assumption that the sanitizer is DP, which is not the case. Therefore the appropriate circularity score is 0, with the sensitivity issue flagged as a correctness risk rather than circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The system leans on the standard RNM DP theorem, but the paper-specific assumptions (sensitivity=1, fixed outcome space, frequency-based sensitivity) are neither proven nor satisfied by the mechanism as described.

free parameters (3)
  • k = 30
    Number of candidate replacement words per token; chosen by hand, not tuned per dataset.
  • p = 0.3
    Probability that non-sensitive words are also sanitized; manually set hyperparameter.
  • frequency_threshold = bottom 20%
    Tokens below 20% frequency in corpus are considered sensitive; user-chosen policy.
assumptions (5)
  • standard math Report-Noisy-Max with a Laplace scale Delta/epsilon is epsilon-DP when the score function is Delta-sensitive (Theorem 1 from Dwork and Roth).
    Invoked in Section 4.4 Case 1 to claim epsilon-DP for sensitive words.
  • ad hoc to paper The normalized cosine score cosnorm has global sensitivity 1 for any pair of neighboring inputs.
    Eq. (3) asserts equalization of sensitivity to 1; the proof is not given and candidate lists are input-dependent.
  • domain assumption The set of candidate outcomes Si is fixed for all inputs, as required by RNM.
    Section 3.2 defines Si as the top-k words for each input word xi, so the outcome space changes with the input, violating this assumption.
  • domain assumption Semi-honest model owners and cloud providers; no side-channel attacks on TEEs.
    Explicitly stated threat model in Section 4.1.
  • domain assumption Low-frequency words are the sensitive ones.
    Section 3.2 follows prior work but the connection between frequency and privacy sensitivity is not formal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Confidential and Efficient LLM Inference with Dual Privacy Protection." pith.science (2026). https://pith.science/paper/TD3L6U2M

@misc{pith2026250909091,
  author       = {Pith},
  title        = {Pith review of: Towards Confidential and Efficient LLM Inference with Dual Privacy Protection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TD3L6U2M}},
  note         = {Machine review of arXiv:2509.09091}
}
read the original abstract

CPU-based trusted execution environments (TEEs) and differential privacy (DP) have gained wide applications for private inference. Due to high inference latency in TEEs, researchers use partition-based approaches that offload linear model components to GPUs. However, dense nonlinear layers of large language models (LLMs) result in significant communication overhead between TEEs and GPUs. DP-based approaches apply random noise to protect data privacy, but this compromises LLM performance and semantic understanding. To overcome the above drawbacks, this paper proposes CMIF, a Confidential and efficient Model Inference Framework. CMIF confidentially deploys the embedding layer in the client-side TEE and subsequent layers on GPU servers. Meanwhile, it optimizes the Report-Noisy-Max mechanism to protect sensitive inputs with a slight decrease in model performance. Extensive experiments on Llama-series models demonstrate that CMIF reduces additional inference overhead in TEEs while preserving user data privacy.

Figures

Figures reproduced from arXiv: 2509.09091 by the authors.

Figure 1
Figure 1. The TEE-based Method for Llama2-7B. Blue operations denote linear opera￾tions, while yellow ones represent nonlinear operations. accelerators while keeping nonlinear layers, like activation layers, within TEEs for confidentiality. Recently, the training-after-partition paradigm has been pro￾posed in TEESlice [24]. They separate and train privacy-related layers from pre-trained models. Furthermore, [9] combine TEESli… view at source ↗
Figure 2
Figure 2. An overview of CMIF. Sensitive tokens are replaced by adjacent tokens based Report-Noisy-Max (RNM) sanitization mechanism (the red box). 3.2 Report-Noisy-Max Sanitization Mechanism To mitigate performance degradation while preserving privacy with DP mech￾anisms, we introduce the Report-Nosiy-Max (RNM) sanitization mechanism. Following previous research [2,23], we regard low-frequency words as sensitive, which need t… view at source ↗
Figure 3
Figure 3. Utility comparison of RNM mechanism based on Llama2-7B (the blue curve) and Llama3-8B (the yellow curve) Subsequently, we conduct experiments on some lightweight LLMs, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Inference time breakdown. CPU+GPU baseline: The client executes the embedding layer in a normal environment without input safeguards during communication, while the server processes the remaining model layers on GPUs. GPU baseline: Entire layers are deployed on GPUs wi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 4 linked inside Pith

  1. [1]

    In: Proc

    Bassily, R., Smith, A.: Local, private, efficient protocols for succinct histograms. In: Proc. of symposium on Theory of computing. pp. 127–135 (2015)

    2015

  2. [2]

    In: Proc

    Chen, S., Mo, F., Wang, Y., et al.: A customized text sanitization mechanism with differential privacy. In: Proc. of ACL. pp. 5747–5758 (2023)

    2023

  3. [3]

    In: Proc

    Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidi- rectional transformers for language understanding. In: Proc. of NAACL. pp. 4171– 4186 (2019)

    2019

  4. [4]

    In: Proc

    Du, M., Yue, X., et al.: Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In: Proc. of CCS. pp. 2665–2679 (2023)

    2023

  5. [5]

    arXiv preprint arXiv:2407.21783 (2024) H

    Dubey, A., Jauhri, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024) H. Yu et al

    arXiv 2024

  6. [6]

    IEEE Access9, 168656– 168677 (2021)

    Duy, K.D., Noh, T., Huh, S., Lee, H.: Confidential machine learning computation in untrusted environments: A systems security perspective. IEEE Access9, 168656– 168677 (2021)

    2021

  7. [7]

    Dwork, C., Roth, A.: The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci.9(3–4), 211–407 (2014)

    2014

  8. [8]

    In: Proc

    Hashemi, H., Wang, Y., Annavaram, M.: Darknight: An accelerated framework for privacy and integrity preserving deep learning using trusted hardware. In: Proc. of MICRO. pp. 212–224 (2021)

    2021

Show all 26 references
  1. [9]

    In: Proc

    Huang, W., Wang, Y., Cheng, A., Zhou, A., Yu, C., Wang, L.: A fast, perfor- mant, secure distributed training framework for large language model. In: Proc. of ICASSP (2024)

    2024

  2. [10]

    In: Proc

    Li, Z., Wang, W., et al.: Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In: Proc. of ECCV. pp. 1–18 (2022)

    2022

  3. [11]

    In: Proc

    Pennington, J., Socher, R., Manning, C.: GloVe: Global vectors for word represen- tation. In: Proc. of EMNLP. pp. 1532–1543 (2014)

    2014

  4. [12]

    In: Proc

    Sabt, M., Achemlal, M., Bouabdallah, A.: Trusted execution environment: What it is, and what it is not. In: Proc. of Trustcom. pp. 57–64 (2015)

    2015

  5. [13]

    In: Proc

    Schunter, M.: Intel software guard extensions: Introduction and open research chal- lenges. In: Proc. of SRPO. pp. 1–1 (2016)

    2016

  6. [14]

    In: Proc

    Shen, T., Qi, J., Jiang, J., Wang, X., Wen, S., et al., X.C.: SOTER: Guarding black-box inference for general neural networks at the edge. In: Proc. of USENIX. pp. 723–738 (2022)

    2022

  7. [15]

    In: Proc

    Shen, Y., Tian, H., et al.: Occlum: Secure and efficient multitasking inside a single enclave of intel sgx. In: Proc. of ASPLOS. pp. 955–970 (2020)

    2020

  8. [16]

    In: Proc

    Sun, Z., Sun, R., Liu, C., Chowdhury, A.R., Lu, L., Jha, S.: Shadownet: A secure and efficient on-device model inference system for convolutional neural networks. In: Proc. of SP. pp. 1596–1612 (2023)

    2023

  9. [17]

    Nature medicine (8), 1930–1940 (2023)

    Thirunavukarasu, A.J., Ting, D.S.J., et al.: Large language models in medicine. Nature medicine (8), 1930–1940 (2023)

    1930

  10. [18]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  11. [19]

    In: Proc

    Tramèr, F., Boneh, D.: Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. In: Proc. of ICLR (2019)

    2019

  12. [20]

    In: Proc

    Tsai, C.C., Porter, D.E., Vij, M.: Graphene-sgx: a practical library os for unmod- ified applications on sgx. In: Proc. of USENIX. pp. 645–658 (2017)

    2017

  13. [21]

    In: Proc

    Xu, H., Kim, Y.J., Sharaf, A., Awadalla, H.H.: A paradigm shift in machine trans- lation: Boosting translation performance of large language models. In: Proc. of ICLR (2024)

    2024

  14. [22]

    arXiv preprint arXiv:2401.11531 (2024)

    Xu, R., Fang, Z.: Tempo: Confidentiality preservation in cloud-based neural net- work training. arXiv preprint arXiv:2401.11531 (2024)

    2024 arXiv

  15. [23]

    In: Proc

    Yue, X., Du, M., et al.: Differential privacy for text analytics via natural text sanitization. In: Proc. of ACL. pp. 3853–3866 (2021)

    2021

  16. [24]

    In: Proc

    Zhang, Z., Gong, C., et al.: No privacy left outside: On the (in-)security of tee- shielded dnn partition for on-device ml. In: Proc. of SP. pp. 55–55 (2024)

    2024

  17. [25]

    arXiv preprint arXiv:2311.07911 (2023)

    Zhou, J., Lu, T., Mishra, S., et al.: Instruction-following evaluation for large lan- guage models. arXiv preprint arXiv:2311.07911 (2023)

    2023 arXiv

  18. [26]

    In: Proc

    Zhou, X., Lu, Y., et al.: TextObfuscator: Making pre-trained language model a privacy protector via obfuscating word representations. In: Proc. of ACL. pp. 5459– 5473 (2023)

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.