{"id":"6fa3f249-77d3-49f1-ad47-dd73bcde915e","arxiv_id":"2601.18511","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A CKKS FHE system runs Llama-2-7B private inference on 4096-token prompts (only the last 128 encrypted) in 85s prefill and 33s/token on 8 RTX-4090 GPUs, with a mismatched abstract claiming faster Llama-3-8B numbers.","lead":"This paper reports a CKKS-based fully homomorphic encryption (FHE) system for private LLM inference that splits prompts into public plaintext and a sensitive encrypted tail, claiming 85s prefill and 33s per token for Llama-2-7B on 8 GPUs. The submitted abstract instead claims faster Llama-3-8B numbers, so the headline result is internally inconsistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline capability is an extrapolation: 85s/33s are per-block times ×32, and end-to-end encrypted accuracy is never measured; the 12-bit precision target rests solely on an unvalidated simulator.","rationale":"The reader correctly identified the simulator-fidelity assumption as load-bearing, and I agree that the absence of end-to-end encrypted accuracy is a serious gap. My stress-test sharpens this: the quoted 85s/33s figures are not measured end-to-end but extrapolated from a single block, and no full-model encrypted run is reported at all. This makes the central claim even less secure than the reader's wording suggests. The abstract/body mismatch (Llama-3-8B on RTX PRO 6000 vs. Llama-2-7B on RTX 4090) is an editorial inconsistency that must be fixed, but the deeper scientific issue is the missing end-to-end validation. The algorithmic contributions (PCMM, slim polynomial evaluation, outlier mitigation) are plausible and may be sound, so the appropriate verdict remains CONDITIONAL pending a real full-model run with measured time and quality. My agreement is 'partial' because the reader's weakest assumption focuses on the simulator, while I would add the timing extrapolation as an equally or more load-bearing aspect of the same unmeasured central claim.","tokens_in":23506,"tokens_out":9506,"duration_ms":100939,"concrete_test":"Run the complete 32-block encrypted inference on the same 8×RTX 4090 system with a 4096-token prompt (last 128 encrypted), recording wall-clock time for summarization and generation, and compute output perplexity/logits on a standard benchmark. Compare the measured end-to-end time to 32× the Table 5 single-block times, and compare the encrypted outputs to the simulator's predicted outputs and to FP16 reference. If the time is >10% off or the quality is worse than the simulator's 12-bit target, the headline numbers are not reproducible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in §1.1—that the system is 'capable of processing 4096 input tokens ... in 85s for prefill summarization and 33s for generation'—is not supported by a full-model execution. §5.4 reports timings for a single transformer block and says 'for an estimate of end-by-end computation time these figures should be multiplied by 32'. This extrapolation assumes uniform per-layer cost and ignores cross-layer effects: activation ranges vary by layer (the paper's own §3.3 tracks per-layer bounds β_{i,(s,L)}), Softmax polynomial degrees and bootstrap counts are chosen per layer, and NCCL synchronization/pipeline overheads may not scale linearly. Additionally, no end-to-end encrypted inference output is reported; the only quality evidence is a custom PyTorch fake-quantization simulator (§3.2) that is asserted to 'precisely mirror' CKKS noise but is never compared with real encrypted execution. If the simulator underestimates noise or overestimates the effect of the attention-sink prefix, the 12-bit precision target (Table 3) is optimistic and the claimed cost/quality trade-off collapses. This is one connected gap: the central empirical claim is not actually measured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes new techniques for CKKS-based private LLM inference when the prompt has a large public prefix and a short encrypted suffix. The main algorithmic contributions are: (i) an unbalanced chunked prefill framework with public/private stages; (ii) a depth-1 plaintext-ciphertext matrix multiplication (PCMM) algorithm for PC-attention; (iii) a slim polynomial evaluation method for sparsely packed ciphertexts; and (iv) outlier mitigation via token prepending and orthogonal rotations. The body reports detailed per-block timings on 8 RTX-4090 GPUs for Llama-2-7B and Llama-3-8B, and extrapolates to 85 s prefill / 33 s generation for the full model. The submitted abstract, however, claims different results (Llama-3-8B on RTX PRO 6000 GPUs, with 20 s / 18 s and 64 s / 22 s numbers) that do not appear in the body.","tokens_in":23871,"tokens_out":5137,"duration_ms":58007,"significance":"If validated, the work would be a meaningful advance in FHE-based LLM inference: it explicitly targets long-context private inference, addresses outlier-induced approximation costs with ML techniques, and introduces an almost bootstrap-free attention design. The algorithmic descriptions are detailed and the choice of building blocks from recent work (e.g., BCH+24, CHK+24, HYT+24) is reasonable. The strength of the paper is the concrete system design and the per-block cost analysis. However, the central quantitative claim is an extrapolation, and no end-to-end encrypted inference accuracy is reported, so the practical significance of the numbers is currently not established.","major_comments":[{"comment":"The submitted abstract reports Llama-3-8B results on 8 NVIDIA RTX PRO 6000 GPUs: 20 s / 18 s for 128 encrypted tokens and 64 s / 22 s for the 4096-token heterogeneous setting. The body title, body abstract, §1.1, and §5 describe Llama-2-7B on 8 RTX-4090 GPUs with 85 s / 33 s. These two sets of claims are inconsistent, and the abstract numbers do not appear anywhere in the body. As it stands, the headline claim is not supported by the manuscript. This must be resolved before review can proceed.","section":"Abstract vs §1.1 and §5"},{"comment":"The 85 s / 33 s end-to-end numbers are not measured. Table 5 reports per-block timings for the second transformer block, and the text explicitly says 'for an estimate of end-by-end computation time these figures should be multiplied by 32'. This assumes per-layer cost uniformity, but the paper itself documents layer-wise and dimension-wise variation in activation ranges (§3.3, §5.1), layer-wise polynomial approximations, and layer-dependent bootstrap placements. NCCL reduction/broadcast and pipeline overheads also need not scale linearly. The §1.1 claim that the system is 'capable of processing 4096 input tokens ... in 85s ... and 33s' is therefore an extrapolation, not a measurement. Please report at least one full-model execution (or a tightly justified scaling model with per-layer data).","section":"§5.4, Table 5"},{"comment":"The 12-bit precision target, and consequently the polynomial degrees, Softmax depth, and bootstrapping parameters, are derived entirely from a custom PyTorch fake-quantization simulator that is asserted to 'precisely mirror' CKKS noise but is never compared to a real encrypted execution. No end-to-end encrypted-inference perplexity or output correctness is reported. If the simulator underestimates noise growth or inaccurately models the interaction of the attention-sink prefix with CKKS rescaling, the claimed cost/quality trade-off collapses. Please validate the simulator against an actual CKKS circuit (e.g., one block or one layer) and report encrypted-inference accuracy/perplexity.","section":"§3.2, Table 3"}],"minor_comments":[{"comment":"Typos and spacing issues: 'Jayashankaret al', 'Moonet al', 'Departement' in the affiliation, 'end-by-end' for 'end-to-end', and 'Paterson-Stockmayer' should be 'Paterson-Stockmeyer'.","section":"Throughout"},{"comment":"The notation β_{i,(s,L)} is defined in §3.3 but used with inconsistent subscript ordering in §3.4. Please define the ordering once and use it consistently.","section":"§3.3, §3.4"},{"comment":"Algorithm 1 states a cost dominated by O(2^{(k-j)/2}) + j ciphertext-ciphertext multiplications and j rotations, but the abstract-level claim is 'O(log d) homomorphic operations'; please clarify the exact dependency on degree d and the role of Paterson-Stockmeyer evaluation.","section":"§4.5"},{"comment":"The paper does not give concrete CKKS parameter sets (ring degree, scaling factor, modulus bit-length, bootstrap precision) for the reported timings. Adding a parameter table would substantially improve reproducibility.","section":"§5.2 / §5.3"}],"recommendation":"major_revision","confidential_remarks":"The abstract/body mismatch is serious and must be editorially resolved; if the abstract numbers come from a different experimental run, that fact needs to be in the body. More fundamentally, the paper's main quantitative claim is an extrapolation from one block, and the precision/noise model is unvalidated. I would not reject outright: the algorithmic ideas are plausible, the per-block timings are detailed, and the contributions could be made sound with a genuine end-to-end measurement and simulator validation. But the manuscript in its current form oversells what was actually run."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you spend time on this. First, the abstract describes Llama-3-8B on RTX PRO 6000 GPUs with 20s/18s timings; the body is about Llama-2-7B on RTX 4090s with 85s/33s. The two don't match. Second, even the body's headline numbers are per-block times multiplied by 32, not measured end-to-end runs, and there is no encrypted end-to-end accuracy result anywhere. The central claim as submitted is not supported by the evidence in the manuscript.\n\nWhat's genuinely new: the depth-1 PCMM variant (Theorem 4.2, Equation 3) with on-the-fly plaintext generation, and the slim polynomial evaluation algorithm using sums-of-squares decomposition (Lemma 4.3, Algorithm 1). Both are real extensions of prior work, the math is clean, and the heterogeneous chunked prefill idea is a practical way to handle the partial-encryption scenario. If you work in FHE-based LLM inference, these pieces are worth reading closely.\n\nThe soft spots are empirical, not theoretical. The per-block timings in Table 5 look detailed and plausible, but multiplying by 32 assumes uniform cost across layers, which the paper's own per-layer range bounds and bootstrap counts contradict. The 12-bit precision target rests entirely on a custom PyTorch simulator that is asserted to mirror CKKS noise but is never compared with real encrypted execution. If that simulator is wrong, the cost/quality trade-off is optimistic. The comparison to prior work also runs on different, costlier hardware, so the speedups are not apples-to-apples. These are load-bearing gaps, not minor quibbles.\n\nI agree with the conditional verdict. The paper deserves a serious referee but needs major revision: the abstract must be aligned with the body, the end-to-end timings must be reported as measured or clearly labeled as extrapolations, and the simulator needs at least some validation against real encrypted execution. I'd bring it to a reading group for the algorithmic ideas, and I'd cite it only after a corrected version shows up.","headline":"Clean algorithmic ideas, unsupported headline numbers — the abstract describes a different run than the body.","tokens_in":24372,"tokens_out":3669,"would_cite":false,"duration_ms":38675,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a CKKS-based private Llama-2-7B inference can process a 4096-token prompt, with only the last 128 tokens encrypted, in 85 seconds for prefill and 33 seconds per generated token on eight consumer GPUs.","keywords":["fully homomorphic encryption","CKKS","private LLM inference","plaintext-ciphertext matrix multiplication","homomorphic Softmax","outlier mitigation","chunked prefill","Llama"],"falsifier":"Take the claimed 4096-token split (3968 plaintext, 128 encrypted), run the actual CKKS implementation on the same eight GPUs, and compare end-to-end output perplexity against the FP16 baseline; if perplexity degrades beyond the 12-bit target or timings exceed 85 seconds/33 seconds per token, the central claim fails. A cheaper test: compare simulator-injected noise values against real CKKS slot values for one block; divergence beyond the assumed noise budget would show the precision analysis is unsound.","tokens_in":1479,"feed_emoji":"🔐","tokens_out":1687,"duration_ms":91368,"temperature":0.7,"pith_summary":"The paper tries to show that fully homomorphic encryption can serve practical private LLM inference for long prompts when most of the prompt is public and only the tail is confidential. It builds a CKKS implementation of Llama-2-7B that accepts 4096 input tokens, of which the last 128 are encrypted, and reports 85 seconds for prefill summarization and 33 seconds per generated token on eight RTX 4090 GPUs. To get there, it splits prefill into a public clear stage and a private encrypted stage, tames activation outliers with attention-sink prefixes and orthogonal rotations, and introduces a shallow plaintext-ciphertext matrix multiplication plus a slim Softmax evaluation. If the reported numbers hold, this would move FHE LLM inference from tens of encrypted tokens to thousands of partially encrypted tokens at speeds below a minute per generated token. The abstract quotes even faster Llama-3-8B numbers on different hardware than the full text reports, so the exact model and configuration behind the headline timings need careful reading.","feed_headline":"Private Llama inference scales to 4096 tokens in 85s","feed_subtitle":"Only the last 128 tokens are encrypted; a hybrid prefill keeps the rest clear and cuts cost sharply.","key_machinery":"The central mechanism is a heterogeneous chunked prefill pipeline: a public prefill stage processes the long plaintext prefix in the clear to build a key-value cache, then a private prefill stage evaluates QKV projection, attention, output projection, and the feed-forward network on the encrypted suffix. Within that pipeline, the load-bearing new identities are a tau-shifted rotation-based matrix product formula that performs plaintext-ciphertext multiplication in one multiplicative level without encoding conversion, and a recursive sum-of-two-squares decomposition of polynomials that lets Softmax's auxiliary track be evaluated on sparsely packed ciphertexts in O(log d) homomorphic operation","core_discovery":"The central claim is that private inference for a Llama-scale model becomes practical when only the final part of the input is confidential. The paper reports a CKKS implementation of Llama-2-7B that ingests 4096 tokens—3968 plaintext, 128 encrypted—in 85 seconds for summarization and generates each following token in 33 seconds, on eight RTX 4090 GPUs. The abstract, but not the body, claims Llama-3-8B timings of 20s/18s per token for 128 encrypted tokens and 64s/22s per token for the heterogeneous 4096-token input on eight RTX PRO 6000 GPUs. The author's claim is that the enabling ingredients—heterogeneous chunked prefill, attention-sink prefixing, orthogonal rotations, and the new linear-a","pith_inferences":["Inference: the abstract's Llama-3-8B numbers should not be read as measured results of this manuscript; the body's tables and per-block breakdowns target Llama-2-7B on RTX 4090 GPUs, so the two sets of timings likely come from different builds, different hardware, or projections.","Inference: if the simulator's noise model is faithful, the sum-of-squares evaluation trick could be lifted to other slim-slot nonlinearities in FHE, potentially reducing bootstraps in decoding or in fully encrypted prefill more broadly.","Inference: the public-prefix assumption is the real boundary of the method; the speedups come from processing most tokens in the clear, and extending to fully encrypted long prompts would require ciphertext-ciphertext attention over the whole context, which the paper does not claim to scale.","Inference: a natural testable extension is to vary the public/private split (e.g., 1k/128 vs 4k/128) and measure whether cost scales roughly linearly in the encrypted suffix, which would confirm the framework rather than the specific implementation."],"forward_implications":["FHE LLM inference need not treat every input token as a ciphertext: when only the prompt tail is sensitive, the public prefix can be handled in the clear, turning a wide encrypted attention computation into a manageable plaintext-ciphertext problem.","Outlier-suppression techniques—attention-sink prefixing and orthogonal rotation—cut the input range of RMSNorm from roughly 2244 to about 7.7, making low-degree polynomial approximations and standard-precision bootstrapping feasible.","A one-level plaintext-ciphertext matrix multiplication that updates its packing structure on the fly avoids both bootstrapping and large plaintext-matrix memory footprints in wide attention, keeping the attention phase almost bootstrap-free.","Softmax can be evaluated on slim ciphertexts with about log(depth) homomorphic levels through sum-of-squares decomposition, at a cost of roughly one bit of precision per recursion level.","If the body's timings reproduce, the same eight-GPU cluster completes a 4096-token private-prefill inference in 85 seconds, compared with 295 seconds in prior work for only 128 encrypted tokens on more expensive hardware."],"fun_headline_variants":["Private Llama: 4K tokens, only last 128 encrypted, 85s prefill","Hybrid FHE makes Llama inference practical for 4K tokens","FHE scaled up: 4096-token Llama with mixed encryption","128 encrypted tokens? Try 4096 with hybrid FHE on Llama","Cut FHE cost: keep most of Llama input plaintext, scale to 4K"],"cache_read_input_tokens":25728,"weakest_assumption_plain":"The load-bearing premise is that the custom fixed-point simulator accurately reproduces CKKS noise growth and the effect of the attention-sink prefix, so that the selected 12-bit precision target carries over to real encrypted inference; the paper reports no end-to-end accuracy measurement of the actual encrypted pipeline, and an optimistic simulator would invalidate the whole cost/quality trade-off.","fun_headline_variants_meta":{"raw":{"variants":["Private Llama: 4K tokens, only last 128 encrypted, 85s prefill","Hybrid FHE makes Llama inference practical for 4K tokens","FHE scaled up: 4096-token Llama with mixed encryption","128 encrypted tokens? Try 4096 with hybrid FHE on Llama","Cut FHE cost: keep most of Llama input plaintext, scale to 4K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":3079,"prompt_tokens":936,"completion_tokens":2143,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":2044}},"tokens_in":680,"tokens_out":2143,"duration_ms":20907,"temperature":1.0,"reasoning_tokens":2044,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T07:56:00.834060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the claimed 4096-token split (3968 plaintext, 128 encrypted), run the actual CKKS implementation on the same eight GPUs, and compare end-to-end output perplexity against the FP16 baseline; if perplexity degrades beyond the 12-bit target or timings exceed 85 seconds/33 seconds per token, the central claim fails. A cheaper test: compare simulator-injected noise values against real CKKS slot values for one block; divergence beyond the assumed noise budget would show the precision analysis is unsound.","supporting_citations":[],"review_version":1}