{"id":"6c21381b-2c81-43b0-81be-25628be37039","arxiv_id":"2505.16501","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Confidential GPU inference under model swapping is significantly slower than non-confidential, and the gap is driven by model loading, not by inference compute.","lead":"This paper compares confidential and non-confidential GPU inference on an NVIDIA H100 when multiple large language models are swapped in and out of memory. It finds that confidential mode has 20-30% higher latency, 45-70% lower throughput, and about 50% lower GPU utilization, with most of the penalty coming from slower model loading.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper attributes the CC/No-CC performance gap to encryption/decryption during model loading, but the load-time measurements (Section III-D1, Fig.","rationale":"The paper's headline contribution is an empirical quantification of CC vs No-CC GPU inference under model swapping, with a causal explanation. The reader's conditional verdict and weakest assumption point to the lack of isolation of encryption cost; my stress-test confirms this is the most load-bearing weakness. The direction of the performance gap is probably robust—load times are indeed higher in CC (Fig. 3), inference processing rates are similar (Section IV-B), and the conclusion that CC is slower has multiple measurements behind it. The public code and reproducible workflow are real strengths. But the paper goes beyond reporting a gap to attributing it specifically to encryption/decryption overhead of loading models. This attribution is what would guide future optimization (e.g., reducing per-byte crypto or overlapping it with I/O). If the actual dominant cost is per-load fixed overhead from the CPU TEE and secure hypervisor, the same experiments would produce the same numbers but the prescription would change. The proposed regression test directly separates per-byte from fixed cost using model size as a natural independent variable, since the three profiled models differ in size by ~10 GB. This test is feasible from the public code and would settle whether the load-bearing causal claim holds. Secondary issues—missing error bars and the dimensionally invalid Select Batch inequality (Section III-C4)—are real but do not determine the CC-vs-No-CC gap; they affect precision and one scheduler's derivation, respectively. Overall, I do not see a reason to move the reader's conditional verdict, but the authors should be required to either run the isolation test or soften the causal wording.","tokens_in":8000,"tokens_out":4407,"duration_ms":36101,"concrete_test":"Re-run the Section III-D1 load-time profiling for all three models (16.07, 17.07, 26.98 GB) with repeated trials in CC and No-CC, and fit load_time = a + b * model_size separately for each mode. If (b_CC - b_NoCC) * mean_model_size explains most of the mean load-time gap and the difference is statistically significant, per-byte encryption overhead is supported. If the intercept difference a_CC - a_NoCC dominates, or slopes are equal, the gap is a fixed per-load overhead (host TEE/hypervisor/driver setup), undercutting the causal attribution. Report N, standard errors, and the fitted coefficients.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'performance in the confidential setting is inferior ... primarily due to the additional encryption and decryption overhead required for loading models onto the GPU' (Abstract)—is a causal attribution. The supporting evidence is the model-load-time comparison (Section III-D1, Fig. 3) plus the observation that in-GPU inference processing rate is similar across modes (Section IV-B). However, the load-time profiling explicitly includes 'tokenizer and model parameter initialization, as well as GPU memory allocation and I/O overheads' (Section III-D1). It does not decompose the CC-vs-No-CC load-time delta into (a) per-byte GPU-side encryption/decryption, (b) host CPU TEE overhead, (c) secure-hypervisor interaction, or (d) driver/I/O path differences. The experimental setup (Section II-B) notes CC requires a CPU TEE and a CC-capable hypervisor, and these components can add per-load setup costs unrelated to GPU crypto. Without a component-wise breakdown, the load-time gap could be dominated by fixed per-load costs (e.g., attestation/setup) rather than by encrypting model bytes. If so, the paper's implied optimization—targeting encryption/decryption on the model loading path—would not recover the performance gap, and the central conclusion would be misleading even though the raw direction of the result (CC slower) may hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compares confidential computing (CC) and non-confidential (No-CC) modes for GPU-based LLM inference under model swapping. Using a single NVIDIA H100 GPU inside one VM, the authors vary traffic distributions (gamma, bursty, ramp), scheduling strategies (Best Batch, Timer, Partial Batch, Select Batch and combinations), and SLA values (40, 60, 80 seconds), measuring latency, SLA attainment, throughput, and GPU utilization over 20-minute runs. They report that No-CC outperforms CC by 20–30% in latency, 15–20% in SLA attainment, 45–70% in throughput, and about 50% in GPU utilization, and they attribute this gap primarily to encryption/decryption overhead during model loading. The paper also provides a publicly available code repository and argues that future optimization should focus on the model-loading path.","tokens_in":8261,"tokens_out":3706,"duration_ms":29634,"significance":"The direction of the main result—confidential GPU inference with active model swapping pays a substantial performance tax—is plausible and practically relevant, and the paper addresses a realistic scenario that prior work on confidential GPUs has largely overlooked. Its strengths include direct measurements on current hardware (H100 with CC), a public code repository, a systematic comparison of several scheduling strategies, and explicit attention to model-swap costs rather than only steady-state inference. If the causal attribution is correct, the paper identifies a concrete optimization target for confidential GPU serving. However, the evidence as presented does not yet isolate the claimed mechanism, and the lack of statistical backing weakens the quantitative claims.","major_comments":[{"comment":"The central claim that CC performance is inferior 'primarily due to the additional encryption and decryption overhead required for loading models onto the GPU' is a causal attribution, but the experiments do not isolate that mechanism. Section III-D1 defines model load time as including tokenizer and model parameter initialization, GPU memory allocation, and I/O overheads, and Section II-B explains that CC mode additionally requires a CPU TEE and a CC-capable hypervisor. The measured CC-vs-No-CC load-time gap could therefore be dominated by host-side TEE setup, attestation, or hypervisor/driver path differences rather than by per-byte GPU-side encryption/decryption. Please either provide a component-wise breakdown (e.g., per-byte load times with and without CC, attestation cost, and setup cost) or rephrase the conclusion to attribute the gap to the CC model-loading path generally, without naming encryption/decryption as the dominant cause.","section":"Abstract and Section IV-C"},{"comment":"Every quantitative claim (20–30% latency difference, 15–20% SLA attainment difference, 45–70% throughput difference, approximately 50% GPU utilization difference) is reported as a range without any indication of run counts, standard deviations, confidence intervals, or significance tests. The text describes 'consistent' observations but provides no measures of dispersion, and each configuration appears to be a single 20-minute run. Since these ranges are the paper's primary quantitative contribution, please report the number of repetitions per configuration and the observed variability, or explicitly state that the measurements are single-run observations and adjust the strength of the claims accordingly.","section":"Section IV (all subsections)"},{"comment":"The paper contains an internal inconsistency about model swap counts: Section IV-A states that 'the swap count is slightly higher in No-CC mode', while Section V concludes that throughput and GPU utilization are lower in CC mode 'despite the number of model switches being the same in both settings'. Section IV-C also says the number of swaps is 'similar' in both settings. This inconsistency matters for the argument that the performance gap is driven by per-swap loading cost. Please reconcile these statements and, if possible, report the actual swap counts under each configuration.","section":"Section IV-A vs Section V"}],"minor_comments":[{"comment":"The Select Batch description contains a likely typo: the invariant 'batch accumulation time <= arrival rate' is dimensionally inconsistent; the subsequent inequality 'batch size <= arrival rate × desired latency' is the correct form, and the earlier invariant should be fixed to avoid confusion.","section":"Section III-C4"},{"comment":"There is a minor typo in 'to meet the latency constraint prescribed the the SLAs'—'the the' should be 'by the' or 'in the'.","section":"Section III-C4"},{"comment":"The definition of GPU utilization as 'the percentage of total runtime during which the GPU actively performs inference' is ambiguous about whether model loading counts as active GPU time; clarifying this would help interpret the subsequent statement that unused time is spent loading the model.","section":"Section IV-C"},{"comment":"The statement that 'SelectBatch+Timer achieves the best performance' could be misinterpreted as best overall throughput, but Section IV-B shows that Best Batch-based strategies achieve higher throughput; please qualify that Select Batch is best for latency/SLA attainment, not throughput.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a short conference-style manuscript, and its main empirical direction is useful. The major concerns are the unsupported causal attribution and the lack of statistical reporting; both are fixable within the manuscript's scope by adding a load-time decomposition and dispersion measures, or by softening the causal language. The internal inconsistency on swap counts should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey [Name],\n\nQuick take on arXiv:2505.16501. It's a real measurement study—NVIDIA H100, three LLMs, confidential vs. non-confidential GPU inference with models swapping in and out of memory. The punchline: CC is genuinely slower under swapping, and the numbers are in the direction you'd expect. But the paper's claim that encryption/decryption of model loads is the primary cause isn't actually supported by what they measured. That's the soft spot.\n\nWhat's new: prior work (Refs 12 and 15) already benchmarked H100 CC for LLM inference. This paper adds the model-swapping workload and four scheduling strategies (Best Batch, Timer, Select Batch, Partial Batch). That's a useful, incremental extension, not a breakthrough. The strongest evidence is that the per-request processing rate during inference is nearly the same in CC and No-CC across all configurations—the gap comes from the time between inference phases. That diagnostic is clean and worth keeping.\n\nWhat I don't like: the causal attribution. The abstract says the overhead is \"primarily due to the additional encryption and decryption overhead required for loading models.\" But their load-time profiling includes tokenizer/parameter initialization, GPU memory allocation, and I/O. They never decompose the CC-vs-No-CC load-time delta into GPU-side crypto, host CPU TEE setup, hypervisor interaction, or driver paths. If most of the gap is fixed per-load cost (attestation, secure channel setup), then \"optimize encryption\" is the wrong target, and the conclusion as stated is misleading. The raw direction probably survives, but the mechanism is underdetermined.\n\nAlso: no error bars, run counts, or significance tests for the headline percentages (20–30% latency, 45–70% throughput). These are plausible ranges, but without variance info they're anecdotes. And the Select Batch derivation has a dimensional slip: \"batch accumulation time <= arrival rate\" mixes units. It's a typo-level error, but it should be fixed.\n\nWho's this for? Cloud operators and scheduler designers who need a quick sanity check on CC GPU serving with model swapping. It's a legitimate data point, not a definitive study. The code is public, which is good.\n\nRecommendation: Yes, send it to peer review—an editor should not desk-reject this. But the reviewers should ask for error bars, a component-wise breakdown of the load-time gap, and a cleanup of the Select Batch math. It's a moderate revision, not a reject.\n\nCheers, [Your name]","headline":"Solid incremental H100 CC-vs-No-CC data under model swapping; the headline encryption claim outruns the measurements.","tokens_in":8774,"tokens_out":2857,"would_cite":false,"duration_ms":23905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Confidential GPU mode costs 45–70% throughput when models swap","keywords":["confidential computing","GPU TEE","LLM inference","model swapping","scheduling","NVIDIA H100","SLA attainment","performance measurement"],"falsifier":"A component-level trace of model loading in CC mode that separates host-side TEE entry, encrypted transfer, GPU-side decryption, and model initialization would settle the causal claim; if the CC–No-CC gap persists with GPU-side encryption disabled, the paper's primary-cause attribution is wrong.","tokens_in":7825,"feed_emoji":"🔒","tokens_out":5134,"duration_ms":37132,"temperature":0.7,"pith_summary":"This paper measures the performance cost of confidential computing (CC) on an NVIDIA H100 GPU when serving multiple large language models under relaxed inference and forced model swapping. In a single-VM, single-GPU testbed, the authors find that non-confidential mode achieves 20–30% lower latency, 15–20% higher SLA attainment, 45–70% higher throughput, and about 50% higher GPU utilization than CC mode. They trace the gap primarily to model loading, which is slower in CC mode because of encryption and decryption overhead. The per-inference processing rate is the same in both modes, isolating the bottleneck to the swap path rather than the inference kernels.","feed_headline":"Confidential GPU mode costs 45–70% throughput when models swap","feed_subtitle":"Model loading with encryption dominates the slowdown, so caching or pre-encrypting weights could close the gap.","key_machinery":"The central object is the model-swapping loop: a scheduler loads a model into GPU memory, batches waiting requests, runs inference, unloads the model, and repeats for the next model. The authors profile per-model load/unload times and batch-size throughput, then drive this loop with four scheduling strategies under three traffic distributions. The mechanism that carries the argument is the measured asymmetry in model-loading time between CC and No-CC modes, which the authors attribute to encryption/decryption on the loading path and which explains the observed gaps in latency, throughput, SLA attainment, and GPU utilization.","core_discovery":"The central claim is that for relaxed batch inference with active model swapping, confidential GPU computing on an NVIDIA H100 is consistently and substantially slower than non-confidential execution, and the slowdown is concentrated in the model-loading path, not in inference compute. Across all tested traffic distributions (gamma, bursty, ramp) and scheduling strategies, No-CC mode shows 20–30% lower request latency, 15–20% higher SLA attainment, 45–70% higher throughput, and roughly 50% higher GPU utilization. The authors report that per-request processing time during inference is equal across modes, and that the number of model swaps is similar, but each swap takes longer in CC mode; they attribute this to the additional encryption and decryption required to load models into GPU memory in confidential environments.","pith_inferences":["The paper's attribution to encryption/decryption is plausible but not isolated; a deeper breakdown of load time could show host-side TEE overhead or driver paths as significant contributors.","If the overhead scales with model size, pre-encrypting model weights offline and keeping the encrypted copy in GPU memory could amortize the cost across multiple inference runs.","The measured swap-count similarity suggests the scheduler behaves the same in both modes; a CC-aware scheduler that reduces swap frequency could shift the trade-off curve."],"forward_implications":["Minimizing model swaps, for example by keeping a model resident or caching decrypted weights, should recover most of the CC-mode performance gap.","Scheduling strategies that are tuned for No-CC swap costs are suboptimal for CC; swap-aware, SLA-driven schedulers should perform better.","Because per-inference processing is unchanged, the inference kernels are not the target of optimization; the data path into the GPU is.","The reported numbers quantify the ceiling for relaxed-inference CC serving on current H100 hardware: without load-path improvements, confidential mode will lag by roughly half in throughput and GPU utilization."],"supporting_citations":[{"why":"Supplies the H100 hardware CC features (secure boot, attestation) that define the confidential execution path under study.","marker":"[7]"},{"why":"Prior evaluation of CPU-GPU confidential computing with TDX and H100, providing a baseline for expected penalties and pipelining effects.","marker":"[12]"},{"why":"Performance benchmark on H100 CC that identifies encrypted data transfers as the primary bottleneck for scalable confidential inference.","marker":"[15]"},{"why":"Provides the scheduling context and traffic distribution assumptions (gamma, bursty) that the experimental design builds on.","marker":"[16]"}],"fun_headline_variants":["Encrypted model loads slow confidential GPU inference by 20–30%","Confidential GPUs lose 45–70% throughput to encryption overhead","Confidential mode: 50% lower GPU utilization, 45–70% lower throughput","H100 confidential mode: encryption overhead cuts throughput up to 70%","Confidential GPU swaps: 20–30% higher latency, 45–70% lower throughput"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments attribute the load-time gap to GPU-side encryption and decryption, but they never measure that component in isolation; if host-side TEE or platform overhead is a large part of the gap, the stated cause and the implied optimization target would be wrong.","fun_headline_variants_meta":{"raw":{"variants":["Encrypted model loads slow confidential GPU inference by 20–30%","Confidential GPUs lose 45–70% throughput to encryption overhead","Confidential mode: 50% lower GPU utilization, 45–70% lower throughput","H100 confidential mode: encryption overhead cuts throughput up to 70%","Confidential GPU swaps: 20–30% higher latency, 45–70% lower throughput"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3257,"prompt_tokens":913,"completion_tokens":2344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2238}},"tokens_in":529,"tokens_out":2344,"duration_ms":14249,"temperature":1.0,"reasoning_tokens":2238,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:59:07.343704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A component-level trace of model loading in CC mode that separates host-side TEE entry, encrypted transfer, GPU-side decryption, and model initialization would settle the causal claim; if the CC–No-CC gap persists with GPU-side encryption disabled, the paper's primary-cause attribution is wrong.","supporting_citations":[{"cited_title":"Nvidia h100 tensor core gpu architecture,","cited_arxiv_id":null,"evidence_quote":"Supplies the H100 hardware CC features (secure boot, attestation) that define the confidential execution path under study."},{"cited_title":"Securing ai inference in the cloud: Is cpu-gpu confidential computing ready?","cited_arxiv_id":null,"evidence_quote":"Prior evaluation of CPU-GPU confidential computing with TDX and H100, providing a baseline for expected penalties and pipelining effects."},{"cited_title":"AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving,","cited_arxiv_id":null,"evidence_quote":"Provides the scheduling context and traffic distribution assumptions (gamma, bursty) that the experimental design builds on."}],"review_version":1}