Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Performance of Confidential Computing GPUs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Confidential GPU mode costs 45–70% throughput when models swap

desk verdict Solid incremental H100 CC-vs-No-CC data under model swapping; the headline encryption claim outruns the measurements. read the letter →

arxiv 2505.16501 v1 pith:T4U4NXIE submitted 2025-05-22 cs.PF

Antonio Martínez Ibarra , Julian James Stephen , Aurora González Vidal , K. R. Jayaram , Antonio Fernando Skarmeta Gómez This is my paper · ORCID
classification cs.PF
keywords confidentialcomputingGPUTEELLMinferencemodelswappingschedulingNVIDIAH100SLAattainmentperformancemeasurement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper measures the performance cost of confidential computing (CC) on an NVIDIA H100 GPU when serving multiple large language models under relaxed inference and forced model swapping. In a single-VM, single-GPU testbed, the authors find that non-confidential mode achieves 20–30% lower latency, 15–20% higher SLA attainment, 45–70% higher throughput, and about 50% higher GPU utilization than CC mode. They trace the gap primarily to model loading, which is slower in CC mode because of encryption and decryption overhead. The per-inference processing rate is the same in both modes, isolating the bottleneck to the swap path rather than the inference kernels.

What carries the argument

The central object is the model-swapping loop: a scheduler loads a model into GPU memory, batches waiting requests, runs inference, unloads the model, and repeats for the next model. The authors profile per-model load/unload times and batch-size throughput, then drive this loop with four scheduling strategies under three traffic distributions. The mechanism that carries the argument is the measured asymmetry in model-loading time between CC and No-CC modes, which the authors attribute to encryption/decryption on the loading path and which explains the observed gaps in latency, throughput, SLA attainment, and GPU utilization.

What would settle it

A component-level trace of model loading in CC mode that separates host-side TEE entry, encrypted transfer, GPU-side decryption, and model initialization would settle the causal claim; if the CC–No-CC gap persists with GPU-side encryption disabled, the paper's primary-cause attribution is wrong.

Watch

Extended reading notes

Core claim

The central claim is that for relaxed batch inference with active model swapping, confidential GPU computing on an NVIDIA H100 is consistently and substantially slower than non-confidential execution, and the slowdown is concentrated in the model-loading path, not in inference compute. Across all tested traffic distributions (gamma, bursty, ramp) and scheduling strategies, No-CC mode shows 20–30% lower request latency, 15–20% higher SLA attainment, 45–70% higher throughput, and roughly 50% higher GPU utilization. The authors report that per-request processing time during inference is equal across modes, and that the number of model swaps is similar, but each swap takes longer in CC mode; they attribute this to the additional encryption and decryption required to load models into GPU memory in confidential environments.

Load-bearing premise

The experiments attribute the load-time gap to GPU-side encryption and decryption, but they never measure that component in isolation; if host-side TEE or platform overhead is a large part of the gap, the stated cause and the implied optimization target would be wrong.

Editorial extensions

If this is right

  • Minimizing model swaps, for example by keeping a model resident or caching decrypted weights, should recover most of the CC-mode performance gap.
  • Scheduling strategies that are tuned for No-CC swap costs are suboptimal for CC; swap-aware, SLA-driven schedulers should perform better.
  • Because per-inference processing is unchanged, the inference kernels are not the target of optimization; the data path into the GPU is.
  • The reported numbers quantify the ceiling for relaxed-inference CC serving on current H100 hardware: without load-path improvements, confidential mode will lag by roughly half in throughput and GPU utilization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's attribution to encryption/decryption is plausible but not isolated; a deeper breakdown of load time could show host-side TEE overhead or driver paths as significant contributors.
  • If the overhead scales with model size, pre-encrypting model weights offline and keeping the encrypted copy in GPU memory could amortize the cost across multiple inference runs.
  • The measured swap-count similarity suggests the scheduler behaves the same in both modes; a CC-aware scheduler that reduces swap frequency could shift the trade-off curve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper compares confidential computing (CC) and non-confidential (No-CC) modes for GPU-based LLM inference under model swapping. Using a single NVIDIA H100 GPU inside one VM, the authors vary traffic distributions (gamma, bursty, ramp), scheduling strategies (Best Batch, Timer, Partial Batch, Select Batch and combinations), and SLA values (40, 60, 80 seconds), measuring latency, SLA attainment, throughput, and GPU utilization over 20-minute runs. They report that No-CC outperforms CC by 20–30% in latency, 15–20% in SLA attainment, 45–70% in throughput, and about 50% in GPU utilization, and they attribute this gap primarily to encryption/decryption overhead during model loading. The paper also provides a publicly available code repository and argues that future optimization should focus on the model-loading path.

Significance. The direction of the main result—confidential GPU inference with active model swapping pays a substantial performance tax—is plausible and practically relevant, and the paper addresses a realistic scenario that prior work on confidential GPUs has largely overlooked. Its strengths include direct measurements on current hardware (H100 with CC), a public code repository, a systematic comparison of several scheduling strategies, and explicit attention to model-swap costs rather than only steady-state inference. If the causal attribution is correct, the paper identifies a concrete optimization target for confidential GPU serving. However, the evidence as presented does not yet isolate the claimed mechanism, and the lack of statistical backing weakens the quantitative claims.

major comments (3)
  1. [Abstract and Section IV-C] The central claim that CC performance is inferior 'primarily due to the additional encryption and decryption overhead required for loading models onto the GPU' is a causal attribution, but the experiments do not isolate that mechanism. Section III-D1 defines model load time as including tokenizer and model parameter initialization, GPU memory allocation, and I/O overheads, and Section II-B explains that CC mode additionally requires a CPU TEE and a CC-capable hypervisor. The measured CC-vs-No-CC load-time gap could therefore be dominated by host-side TEE setup, attestation, or hypervisor/driver path differences rather than by per-byte GPU-side encryption/decryption. Please either provide a component-wise breakdown (e.g., per-byte load times with and without CC, attestation cost, and setup cost) or rephrase the conclusion to attribute the gap to the CC model-loading path generally, without naming encryption/decryption as the dominant cause.
  2. [Section IV (all subsections)] Every quantitative claim (20–30% latency difference, 15–20% SLA attainment difference, 45–70% throughput difference, approximately 50% GPU utilization difference) is reported as a range without any indication of run counts, standard deviations, confidence intervals, or significance tests. The text describes 'consistent' observations but provides no measures of dispersion, and each configuration appears to be a single 20-minute run. Since these ranges are the paper's primary quantitative contribution, please report the number of repetitions per configuration and the observed variability, or explicitly state that the measurements are single-run observations and adjust the strength of the claims accordingly.
  3. [Section IV-A vs Section V] The paper contains an internal inconsistency about model swap counts: Section IV-A states that 'the swap count is slightly higher in No-CC mode', while Section V concludes that throughput and GPU utilization are lower in CC mode 'despite the number of model switches being the same in both settings'. Section IV-C also says the number of swaps is 'similar' in both settings. This inconsistency matters for the argument that the performance gap is driven by per-swap loading cost. Please reconcile these statements and, if possible, report the actual swap counts under each configuration.
minor comments (4)
  1. [Section III-C4] The Select Batch description contains a likely typo: the invariant 'batch accumulation time <= arrival rate' is dimensionally inconsistent; the subsequent inequality 'batch size <= arrival rate × desired latency' is the correct form, and the earlier invariant should be fixed to avoid confusion.
  2. [Section III-C4] There is a minor typo in 'to meet the latency constraint prescribed the the SLAs'—'the the' should be 'by the' or 'in the'.
  3. [Section IV-C] The definition of GPU utilization as 'the percentage of total runtime during which the GPU actively performs inference' is ambiguous about whether model loading counts as active GPU time; clarifying this would help interpret the subsequent statement that unused time is spent loading the model.
  4. [Section IV-A] The statement that 'SelectBatch+Timer achieves the best performance' could be misinterpreted as best overall throughput, but Section IV-B shows that Best Batch-based strategies achieve higher throughput; please qualify that Select Batch is best for latency/SLA attainment, not throughput.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all headline claims are direct CC-vs-No-CC measurements under identical workloads; no fitted parameter is renamed as a prediction.

full rationale

The paper is an empirical measurement study. The headline findings (latency 20-30% lower in No-CC, SLA attainment 15-20% higher, throughput 45-70% higher, GPU utilization ~50% higher) are reported as direct measurements of the same experimental runs in CC and No-CC modes. The optimal batch size (OBS) is determined by prior throughput profiling and then used as a scheduling parameter, but it is not used to manufacture any CC-vs-No-CC gap; the same profiling procedure is applied in both environments and the comparison is made on measured end-to-end latency, throughput, SLA attainment, and GPU utilization. The causal attribution that the gap is 'primarily due to the additional encryption and decryption overhead required for loading models' is an interpretation of the load-time profiling and the observation that in-GPU inference processing rates are similar; it is not a quantity derived from the same data by construction. The attribution could be challenged on experimental-design grounds because the load-time measurement includes tokenizer/model initialization, GPU memory allocation, and I/O overheads without decomposing the CC-vs-No-CC delta, but that is a question of causal isolation and not circularity. There are no fitted inputs called predictions, no self-citation chain carrying a load-bearing premise, no imported uniqueness theorem, and no renamed known result. All results are self-contained against the measured system, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central CC-vs-No-CC comparison uses no fitted constants. The profiled optimal batch size is a measured parameter used by scheduling strategies. The paper's claims generalize only if the single-machine H100 setup, the chosen models, and the assumed typo-free Select Batch formula hold.

free parameters (1)
  • Optimal batch size (OBS) per model = Not listed numerically; determined from profiling curves in Fig. 4
    Measured in batch size profiling (Section III-D2) and used to configure Best Batch and related strategies. The central CC vs No-CC claim does not depend on these values, but the scheduling-strategy comparisons do.
assumptions (4)
  • domain assumption The H100 CC environment used (single VM, secure hypervisor, CPU TEE) is representative of production confidential GPU deployments.
    All conclusions about CC vs No-CC are drawn from one hardware and host configuration; Section III.B describes required host components but does not validate that the setup matches typical deployments.
  • domain assumption The three chosen LLMs and fixed 50-token output length represent relaxed inference workloads.
    Generalization beyond these models and output lengths is claimed qualitatively in the Introduction but not tested.
  • ad hoc to paper The Select Batch scheduler implements a corrected version of the stated formula.
    Section III-C4 contains a dimensionally invalid inequality ('batch accumulation time <= arrival rate'); for the reported SelectBatch results to be meaningful, the implemented formula must be the corrected regression variant rather than the printed one.
  • domain assumption Model load time and throughput measurements are stable across the 20-minute runs.
    No repeat counts, error bars, or significance tests are reported in Section IV, so stability is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance of Confidential Computing GPUs." pith.science (2026). https://pith.science/paper/T4U4NXIE

@misc{pith2026250516501,
  author       = {Pith},
  title        = {Pith review of: Performance of Confidential Computing GPUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T4U4NXIE}},
  note         = {Machine review of arXiv:2505.16501}
}
read the original abstract

This work examines latency, throughput, and other metrics when performing inference on confidential GPUs. We explore different traffic patterns and scheduling strategies using a single Virtual Machine with one NVIDIA H100 GPU, to perform relaxed batch inferences on multiple Large Language Models (LLMs), operating under the constraint of swapping models in and out of memory, which necessitates efficient control. The experiments simulate diverse real-world scenarios by varying parameters such as traffic load, traffic distribution patterns, scheduling strategies, and Service Level Agreement (SLA) requirements. The findings provide insights into the differences between confidential and non-confidential settings when performing inference in scenarios requiring active model swapping. Results indicate that in No-CC mode, relaxed batch inference with model swapping latency is 20-30% lower than in confidential mode. Additionally, SLA attainment is 15-20% higher in No-CC settings. Throughput in No-CC scenarios surpasses that of confidential mode by 45-70%, and GPU utilization is approximately 50% higher in No-CC environments. Overall, performance in the confidential setting is inferior to that in the No-CC scenario, primarily due to the additional encryption and decryption overhead required for loading models onto the GPU in confidential environments.

Figures

Figures reproduced from arXiv: 2505.16501 by the authors.

Figure 1
Figure 1. Confidential Computing with NVIDIA H100 (modified from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example of input traffic distributions with a mean of 4 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Model loading times in CC and No-CC environments. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Inference throughput vs. batch size [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 7
Figure 7. Figure 7: GPU utilization comparison between CC and No-CC settings. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 6
Figure 6. Figure 6: Throughput comparison between CC and No-CC settings. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX

    cs.AI 2026-05 conditional novelty 4.0 of 10

    Confidential mode on an H100 under Intel TDX adds roughly 21-28% time-to-first-token overhead and 17-21% global token-throughput loss for two LLMs, and causes earlier saturation for the larger model.

Reference graph

Works this paper leans on

24 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Confidential computing: Hardware-based trusted execution for applications and data,

    C. C. Consortium, “Confidential computing: Hardware-based trusted execution for applications and data,” Nov. 2022. [Online]. Available: https: //confidentialcomputing.io/wp-content/uploads/sites/10/2023/ 03/CCC outreach whitepaper updated November 2022.pdf

    work page 2022

  2. [2]

    A technical analysis of confidential computing,

    ——, “A technical analysis of confidential computing,” Nov. 2022. [Online]. Available: https: //confidentialcomputing.io/wp-content/uploads/sites/10/2023/ 03/CCC-A-Technical-Analysis-of-Confidential-Computing-v1. 3 unlocked.pdf

    work page 2022

  3. [3]

    Common terminology for confidential computing,

    ——, “Common terminology for confidential computing,” Dec. 2022. [Online]. Available: https: //confidentialcomputing.io/wp-content/uploads/sites/10/2023/ 03/Common-Terminology-for-Confidential-Computing.pdf

    work page 2022

  4. [4]

    The case for confidential com- puting: Delivering business value through protected, confidential data processing,

    T. L. Foundation, “The case for confidential com- puting: Delivering business value through protected, confidential data processing,” Jul. 2024. [Online]. Avail- able: https://www.linuxfoundation.org/hubfs/LF%20Research/ TheCaseforConfidentialComputing 062724.pdf?hsLang=en

    work page 2024

  5. [5]

    A comprehensive overview of large language models,

    H. Naveedet al., “A comprehensive overview of large language models,”arXiv, no. arXiv:2307.06435, Oct. 2024, unpublished. [Online]. Available: http://arxiv.org/abs/2307.06435

    arXiv 2024

  6. [6]

    Intel TDX demystified: A top-down approach,

    P. Chenget al., “Intel TDX demystified: A top-down approach,” ACM Comput. Surv., vol. 56, no. 9, pp. 238:1–238:33, 2024. [Online]. Available: https://doi.org/10.1145/3652597

    doi:10.1145/3652597 2024

  7. [7]

    Nvidia h100 tensor core gpu architecture,

    NVIDIA, “Nvidia h100 tensor core gpu architecture,”

    work page

  8. [8]

    A secure and reliable bootstrap architecture,

    W. A. Arbaugh, D. J. Farber, and J. M. Smith, “A secure and reliable bootstrap architecture,” in1997 IEEE Symposium on Security and Privacy, May 4-7, 1997, Oakland, CA, USA. IEEE Computer Society, 1997, pp. 65–71. [Online]. Available: https://doi.org/10.1109/SECPRI.1997.601317

    work page arXiv 1997

Show all 24 references
  1. [9]

    Principles of remote attestation,

    G. Cokeret al., “Principles of remote attestation,”Int. J. Inf. Sec., vol. 10, no. 2, pp. 63–81, 2011. [Online]. Available: https://doi.org/10.1007/s10207-011-0124-7

    2011 doi

  2. [10]

    Semantic remote at- testation: A virtual machine directed approach to trusted com- puting,

    V . Haldar, D. Chandra, and M. Franz, “Semantic remote at- testation: A virtual machine directed approach to trusted com- puting,” inUSENIX Virtual Machine Research and Technology Symposium, vol. 2004. USENIX Association, 2004

    2004

  3. [11]

    Nvidia hopper gpu architecture,

    NVIDIA, “Nvidia hopper gpu architecture,” 2023. [Online]. Available: https://www.nvidia.com/en-us/data-center/ technologies/hopper-architecture/

    2023

  4. [12]

    Securing ai inference in the cloud: Is cpu-gpu confidential computing ready?

    A. Mohan, M. Ye, H. Franke, M. Srivatsa, Z. Liu, and N. M. Gonzalez, “Securing ai inference in the cloud: Is cpu-gpu confidential computing ready?” in2024 IEEE 17th International Conference on Cloud Computing (CLOUD), Jul. 2024, pp. 164–175. [Online]. Available: https://ieeexp...

    2024

  5. [13]

    Machine learning with confidential computing: A systematization of knowledge,

    F. Mo, Z. Tarkhani, and H. Haddadi, “Machine learning with confidential computing: A systematization of knowledge,”ACM Comput. Surv., vol. 56, no. 11, pp. 281:1–281:40, Jun. 2024

    2024

  6. [14]

    Sok: Limitations of confidential computing via tees for high-performance compute systems,

    A. Akram, V . Akella, S. Peisert, and J. Lowe- Power, “Sok: Limitations of confidential computing via tees for high-performance compute systems,” in2022 IEEE International Symposium on Secure and Private Execution Environment Design (SEED), Sep. 2022, pp. 121–

    2022

  7. [15]

    Confidential computing on nvidia h100 gpu: A performance benchmark study,

    J. Zhu, H. Yin, P. Deng, and S. Zhou, “Confidential computing on nvidia h100 gpu: A performance benchmark study,”arXiv, no. arXiv:2409.03992, Sep. 2024, unpublished. [Online]. Available: http://arxiv.org/abs/2409.03992

    2024 arXiv

  8. [16]

    AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving,

    Z. Liet al., “AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving,” in17th USENIX Symposium on Operating Systems Design and Implementation, 2023, pp. 663–679. [Online]. Available: https: //www.usenix.org/conference/osdi23/presentation/li-zhouhan

    2023

  9. [17]

    Rafiki: Machine learning as an analytics service system,

    W. Wanget al., “Rafiki: Machine learning as an analytics service system,”Proceedings of the VLDB Endowment, vol. 12, 04 2018

    2018

  10. [18]

    Crankshaw, X

    D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica,Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’17): Boston, MA, USA, March 27 - 29, 2017. USENIX Association, 2017. [Online]. Available: https://www.us...

    2017

  11. [19]

    Deep learning workload scheduling in gpu datacenters: A survey,

    Z. Yeet al., “Deep learning workload scheduling in gpu datacenters: A survey,”ACM Computing Surveys, vol. 56, no. 6, pp. 1–38, Jun. 2024

    2024

  12. [20]

    Available: https://github.com/instructlab

    [Online]. Available: https://github.com/instructlab

  13. [21]

    [Online]

    Hugging Face – The AI community building the future. [Online]. Available: https://huggingface.co/

  14. [22]

    A. Isenko. (2023, Sep.) Basic Hardware Monitor. [Online]. Available: https://github.com/cirquit/py-hardware-monitor

    2023

  15. [132]

    Available: https://ieeexplore.ieee.org/document/ 9935045/?arnumber=9935045

    [Online]. Available: https://ieeexplore.ieee.org/document/ 9935045/?arnumber=9935045

  16. [2023]

    Available: https://resources.nvidia.com/ en-us-tensor-core?ncid=no-ncid

    [Online]. Available: https://resources.nvidia.com/ en-us-tensor-core?ncid=no-ncid

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.