{"id":"79540ee9-7605-429e-8ae0-135fcfb32146","arxiv_id":"2505.07239","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Comet accelerates MPC-based private LLM inference by securely predicting and skipping zero-valued neuron activations, achieving up to 2.6x speedup with about 1.5% accuracy loss.","lead":"This paper presents Comet, a system that speeds up private inference of large language models by predicting which neurons will output zero and skipping their secure computations. It reports 1.87x to 2.63x speedups and about 2x communication reductions over several baselines, at a small accuracy cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline speedups are for the non-DP system, whose privacy relies on revealing per-layer sparsity counts; Appendix E admits this leaks under repeated queries, so the central 'private inference' claim is not yet supported.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern, and I agree with it. The deeper reason this is the most load-bearing issue is that the speedup numbers are not in question merely because of arithmetic inconsistencies; whatever the correct speedup is, it is achieved by a system that reveals sparsity counts in the default configuration. The paper's own Appendix E concedes the privacy risk and proposes a DP fix that is not used in the headline experiments. This means the central claim 'private inference with 1.87x-2.63x speedup' is not yet established for a system that satisfies the standard MPC leakage profile. I considered the baseline modifications (Iron/Bolt linear layers replaced by Comet's) and the abstract-vs-table numerical mismatches; those are real but secondary, because they are fixable reporting issues that affect the magnitude of the speedup, not the fundamental validity of the approach. The privacy issue, by contrast, affects whether the system should be called private at all. The proposed concrete test would settle the empirical question directly: if sparsity counts are nearly model-determined, the concern does not land; if they are input-discriminative, the paper must either present the DP-mode results as the headline or provide a rigorous argument that the leakage is acceptable. Since the reader already conditioned the verdict on this and related issues, I recommend keeping the verdict conditional rather than moving it.","tokens_in":29075,"tokens_out":5396,"duration_ms":62563,"concrete_test":"Measure per-layer sparsity counts (MHA and FFN) for Llama2-7B and OPT-6.7B on, e.g., 10,000 Alpaca prompts at fixed 512-token input length, and run a membership-inference classifier on the vector of per-layer counts to distinguish two random inputs. Report AUC and the variance of counts across inputs relative to variation across model seeds. If the AUC is close to 0.5 and counts vary little across inputs, the privacy concern does not land; if AUC is high or counts are highly input-dependent, the non-DP headline speedups should be replaced by the DP-mode results in Table 7.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Comet accelerates private LLM inference while preserving privacy. Section 3.2 explicitly relaxes the standard MPC threat model by assuming that the activation sparsity level (the number of zero values) may be revealed, and it justifies this by asserting that sparsity levels are 'primarily determined by model size' and 'exhibit small variation across different inputs.' This is an empirical claim, and the paper's own Appendix E concedes that an adversary could issue repeated inference requests to map input-to-sparsity relationships and that a malicious model owner could engineer predictor outputs. The appendix then proposes MPC-based differential privacy as a mitigation, but the abstract, Section 7.2, and Table 2 report speedups without DP. Table 7 shows that once DP is applied with epsilon = 0.01, the speedup on Llama2-7B drops from 2.69x to 2.01x, outside the headline range. If sparsity counts are more input-dependent than assumed, the system leaks information beyond the standard MPC output-only leakage, so the headline results describe a system with a privacy caveat that is not part of the main threat model. This is load-bearing because it affects whether the paper's central contribution is actually private inference, not just how large the speedup is.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Comet, an MPC-based private inference system for LLMs that predicts activation sparsity to skip zero-valued computations in both linear and non-linear layers, thereby reducing communication and accelerating inference. It introduces a lightweight low-rank sparsity predictor protected by an oblivious-shuffle-based indexing scheme, sparse matrix multiplication protocols (SOMM and SIMM) with claimed communication/computation optimality, and a KV-cache manager that merges and prefetches cache refills. The evaluation covers OPT-1.3B/2.7B/6.7B and Llama2-7B against several private inference baselines, reporting 1.87x-2.63x speedup and 1.94x-2.64x communication reduction with about 1.5% average accuracy loss on Llama2-7B, plus additional results for other architectures and activation functions in appendices.","tokens_in":29230,"tokens_out":5844,"duration_ms":53627,"significance":"If the privacy model holds, Comet is a timely and useful contribution: to my knowledge it is the first system to exploit activation sparsity for MPC-based private LLM inference, and the protocol-level ideas (shuffle-based plaintext indexing, grouping connected components to minimize communication, column-by-row sparse multiplication) are well motivated and clearly presented. The optimality theorems for SOMM and SIMM are a valuable formal anchor, and the evaluation is broad, including different model sizes, bandwidths, and, in appendices, different architectures and activation functions. The predictor is trained on separate public datasets, so the speedup claims are not circular. However, the headline results are reported under a relaxed threat model that reveals per-layer sparsity counts, the paper's own appendix concedes that this can leak under repeated queries, and the DP-inclusive results fall outside the advertised speedup range. Several internal numerical inconsistencies further reduce confidence. With the privacy caveat addressed and the numbers reconciled, the contribution would be solid.","major_comments":[{"comment":"The central privacy claim is not supported for the configurations whose performance is advertised. Section 3.2 relaxes the standard MPC threat model by assuming that sparsity levels (counts of zero values) may be revealed, and Appendix E concedes that repeated inference requests could map input-to-sparsity relationships and that a malicious model owner could engineer predictor outputs. The proposed MPC-based differential privacy is described as an optional mitigation, and the headline results (Abstract, Section 7.2, Table 2) are reported without it. Table 7 shows that applying DP with epsilon=0.01 reduces the Llama2-7B speedup to 2.01x, which is outside the advertised 1.87x-2.63x range. The authors must either present DP-enabled results as the main privacy-preserving configuration or provide a formal argument (e.g., a reduction or simulation proof that explicitly accounts for the revealed count) showing that sparsity-level leakage stays within the standard MPC leakage profile.","section":"Section 3.2, Appendix E, Table 7"},{"comment":"The headline performance ranges are internally inconsistent with the reported best results. The Abstract and Section 7.2 state a 1.87x-2.63x speedup and a 1.94x-2.64x communication reduction, but Table 2 reports a Llama2-7B total speedup of 2.69x and a communication reduction of 3.14x, and Table 7 also lists the no-DP speedup as 2.69x. These numbers cannot all be correct. The ranges must be recomputed from the full result set, or the text and abstract must be corrected to match the reported tables.","section":"Abstract, Section 7.2, Table 2"},{"comment":"Table 3 contains a likely data error that affects the stated accuracy claim: MBPP is listed with plaintext accuracy 22.4 and Comet accuracy 91.2, which is impossible given the text's claim of an average accuracy loss of about 1.5%. Using the other seven rows, the average loss is approximately 1.8%, not 1.5%. The table entry must be corrected and the average recomputed, since the accuracy-preservation claim is a key part of the paper's contribution.","section":"Section 7.4, Table 3"},{"comment":"The interpretation of layer-wise speedups swaps the sparsity percentages. The sentence says \"This is due to the higher sparsity rate of the FFN (approximately 50%) compared to that of the MHA (about 90%)\", but Section 7.2 reports MHA/FFN sparsity as 49%/85% for Llama2-7B and Figure 3 shows FFN sparsity above 90% and MHA sparsity around 50%. The values are reversed relative to the paper's own measurements, which undermines the explanation of why FFN layers benefit more. This should be corrected.","section":"Section 7.3"}],"minor_comments":[{"comment":"The text says \"We select five mainstream Transformer private inference systems as baselines\" but the Abstract and Figure 12 refer to six systems, and the legend includes Crypten in addition to Iron, Bolt, MPCFormer, SecFormer, and Puma. The baseline count should be made consistent.","section":"Section 7.1"},{"comment":"The notation for the threshold function is confusing: the formula y = sigma(W2(W1x + b1) + b2) uses sigma for both the threshold function and the activation output, and the text says \"outputs 1 if sigma(x) > delta\" where the inner sigma is the threshold function itself. This should be re-notated to avoid ambiguity.","section":"Section 4.1"},{"comment":"The security proof for the oblivious shuffle claims that the revealed sparsity distribution S is uniformly random, but S is a shuffled version of a 0-1 vector and therefore has the same Hamming weight as the original; the proof should explicitly model the sparsity count as leaked and argue that the positions are uniform given that count, or incorporate the count into the leakage profile.","section":"Appendix A"},{"comment":"There is a typo: \"priavete inference\" should read \"private inference\".","section":"Section 6"},{"comment":"The table reports speedups for several models with Swish/SwiGLU activations after ReLUfication, but the main paper's abstract only claims 1.87x-2.63x speedup over six systems on four models. The appendix numbers (e.g., 3.12x for Falcon-7B) go beyond that range; the relationship between the appendix results and the headline range should be clarified.","section":"Appendix C, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid engineering core and the protocol-level insights are worth publishing, but the current version needs major revision. The most serious issue is the mismatch between the relaxed privacy model (revealed sparsity counts) and the headline results, which are reported without the DP mitigation that the paper itself identifies as necessary. The authors should be asked to present DP-enabled results as the main privacy-preserving configuration, or to provide a rigorous leakage argument. In addition, the internal number inconsistencies (Tables 2 and 3, Section 7.3) must be fixed before the paper can be accepted. The contribution is novel enough to warrant the revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things to know. First, Comet is the first private inference system I know of that predicts activation sparsity inside MPC and then skips the corresponding zero work; that is a real niche and a good one. Second, the headline speedups are plausible but the current text has internal number inconsistencies that must be fixed before I'd trust the tables. Third, the privacy promise is weaker than advertised: the non-DP system reveals per-layer sparsity counts, and Appendix E honestly concedes that repeated queries can map input to sparsity level.\n\nWhat is actually new: the oblivious-shuffle-based indexing, which turns a secret-shared sparsity mask into a plaintext shuffled mask while hiding positions, and the SOMM/SIMM protocols with communication-optimality arguments. Those are genuine protocol contributions. The grouping idea (bipartite connected components for SOMM, column-by-row for SIMM) is sound, and the optimality proofs are coherent within their model. The evaluation covers four model sizes, several architectures, activation-function replacement, and WAN bandwidths. If the reported numbers are right, a 2x end-to-end speedup matters in a field where MPC communication dominates.\n\nSoft spots, in proportion:\n- Numbers do not reconcile. The abstract says 1.87x-2.63x speedup and 1.94x-2.64x communication reduction, but Table 2 reports Llama2-7B at 2.69x and 3.14x. Section 7.1 says five baselines; the abstract says six. Table 3 has MBPP at 22.4 plaintext and 91.2 for Comet, which is implausible and looks like a transposed row. These are fixable typos, but they erode confidence in the rest of the tables.\n- Baseline modification. Iron and Bolt are not run as published; their linear layers are replaced with Comet's MPC implementation. That can be a fair controlled comparison, but it means 'compared with six systems' overstates what was actually evaluated.\n- Sparsity-level leakage. The threat model relaxes the usual output-only leakage, and the only support is an empirical claim that sparsity levels vary little across inputs. Appendix E admits repeated-query mapping and notes a malicious model owner could engineer predictor outputs. The DP fix drops the Llama2 speedup from 2.69x to 2.01x at eps=0.01, so the headline range describes the leakier system. This is the main substantive issue: as stated, the system is private inference with an extra disclosed statistic, not plain private inference. I would want either a real empirical distribution of sparsity levels across inputs or headline results run with DP.\n- No code or data release, so the inconsistencies cannot be checked independently.\n\nWho this is for: systems and security researchers working on MPC LLM inference. The protocol design is worth reading even if the final privacy framing needs work. It deserves a serious referee; I would send it out, but expect major revision on reporting and privacy. I would accept.","headline":"Genuinely new system for sparsity-aware MPC LLM inference, with real protocol ideas; but the headline numbers and the privacy framing need correction before I would trust the central 'private inference' claim.","tokens_in":29880,"tokens_out":2657,"would_cite":true,"duration_ms":26969,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A private-inference system called Comet claims that activation sparsity can be safely predicted inside MPC, giving a 1.87x-2.63x speedup and 1.94x-2.64x communication reduction.","keywords":["private inference","secure multi-party computation","activation sparsity","large language models","sparse matrix multiplication","oblivious shuffle","KV cache","LLM inference acceleration"],"falsifier":"A membership-inference-style experiment on the revealed sparsity counts: collect prompts from two distinct datasets, compute their per-layer sparsity levels through Comet's predictor, and train a classifier to guess the dataset. If the classifier's accuracy is far above chance, the paper's assumption that sparsity levels leak little about the input is false, and the headline numbers without differential privacy do not carry the claimed privacy.","tokens_in":28764,"feed_emoji":"⚡","tokens_out":11225,"duration_ms":102407,"temperature":0.7,"pith_summary":"Comet claims that the biggest remaining cost in MPC-based LLM inference, server-to-server communication, can be attacked by predicting where the model's own computations produce zeros. Because more than 90% of ReLU outputs and about half of attention heads in large models are inactive, the system skips those activations and the surrounding matrix multiplications, provided the predictor is accurate enough. The paper reports 1.87x-2.63x end-to-end speedups and 1.94x-2.64x communication reductions over six prior systems on OPT-1.3B through Llama2-7B, losing on average about 1.5% accuracy. If these numbers hold, private serving of large models becomes roughly two to three times cheaper without changing the model architecture, and sparsity-based optimizations used in plaintext LLM serving can be carried over to the encrypted setting.","feed_headline":"Comet speeds private LLM inference by up to 2.63x","feed_subtitle":"A predictor finds dead neurons and lets MPC servers skip them, cutting communication by up to 2.64x","key_machinery":"The load-bearing machinery is the predicted sparsity distribution made safe to reveal. A lightweight two-layer low-rank neural network with a threshold layer, executed inside MPC, produces a secret-shared 0/1 mask of which activations are nonzero. Before the mask is reconstructed, an oblivious-shuffle protocol permutes it by a secret random permutation split between the two servers, so the sparsity count becomes public while the positions of zeros stay private; the same permutation is applied to the input matrix so plaintext indexing of active entries is correct. On top of this, SOMM partitions the nonzero output positions of the preceding linear layer into connected components of a bipartite graph and runs one secure matrix multiplication per component, proving communication-minimality, while SIMM reorganizes the subsequent layer from row-by-column to column-by-row sparse multiplication so each weight row is sent once. A cache manager merges KV-cache miss requests and prefetches skipped heads' values to keep attention correct.","core_discovery":"The central claim, stated on the paper's own terms, is that activation sparsity is not an obstacle to private LLM inference but a resource. Because model parameters are fixed during inference, the input to each transformer layer determines which ReLU neurons will be zero and which attention heads will be inactive, and a shallow two-layer low-rank predictor can anticipate those states well enough to use them as an execution plan. Comet runs that predictor inside MPC, obtains a secret-shared 0/1 mask, and applies an oblivious shuffle before revealing it: the count of active neurons is disclosed, but their positions are randomized so that plaintext indexing does not expose the input's sparsity pattern. Sparse matrix multiplications, SOMM before the nonlinearity and SIMM after it, then skip the masked-out work; the protocols are proved to reach minimal communication, with each row or column masked and communicated only once, and minimal computation under that bound. A KV-cache manager keeps attention correct when some heads were skipped. On this basis the paper reports 1.87x-2.63x end-to-end speedups and 1.94x-2.64x communication reductions on four LLMs against six prior systems, with an average accuracy loss of about 1.5% on Llama2-7B.","pith_inferences":["A natural extension is to apply the same shuffle-and-reveal indexing to other data-dependent zero structures in MPC, such as sparse embeddings, graph adjacency products, or mixture-of-experts routing, where revealing only an activity count is acceptable.","The optional differential-privacy layer is a genuine tradeoff, not a free patch: the paper reports a 2.01x speedup at epsilon 0.01 versus 2.69x without DP, so a deployment that must hide sparsity levels will give up a meaningful share of the gain.","Because a malicious model owner could design a predictor whose sparsity counts are distinctive per input, the predictor itself becomes an attack surface; verifying the predictor or certifying its output with DP noise is a plausible next step.","A wrong prediction that a neuron is zero is irreversible and costs accuracy; an online fallback that recomputes a few masked neurons when the predictor is uncertain could widen the usable operating point, though the paper does not evaluate this."],"forward_implications":["The same Llama2-7B model can be served through MPC at a 2.58x average speedup with no architectural change, only a low-overhead predictor that runs inside MPC.","Larger models gain more: the 7B model shows roughly 85% FFN sparsity versus 61% for the 1.3B model, so the speedup grows with model scale.","Because the approach targets the communication layer itself, it can be composed with nonlinearity-friendly architectures or better MPC approximations rather than replacing them.","KV caching remains usable despite sparse attention: merging miss requests gives a 3.8x communication reduction over per-request refilling, and prefetching adds a further 1.2x.","The predictor's overhead stays below 15% of end-to-end inference time, with about 93% recall, which is why average accuracy loss stays around 1.5%."],"supporting_citations":[{"why":"It documents contextual activation sparsity in LLM inference, the phenomenon the system exploits to skip zero computations.","marker":"[57]"},{"why":"It shows that replacing non-ReLU activations with ReLU plus fine-tuning creates intrinsic sparsity, supporting the system's reach beyond native-ReLU models.","marker":"[79]"},{"why":"It supplies the evidence that sparsity levels are largely model-determined and vary little across inputs, which underlies the decision to reveal only sparsity counts.","marker":"[52]"},{"why":"It is a prior secure distributed-learning work that also exploits sparsity by revealing sparsity counts, cited to show the assumption is consistent with established practice.","marker":"[75]"},{"why":"It provides the MPC framework and primitive library on which Comet is implemented, and it is one of the comparison baselines.","marker":"[42]"},{"why":"It is one of the fastest purely-MPC LLM inference baselines and a main point of comparison for accuracy, speedup, and communication reduction.","marker":"[30]"},{"why":"It defines Beaver triples, the secure multiplication primitive from which the dot-product and matrix-multiplication costs in SOMM and SIMM are derived.","marker":"[7]"},{"why":"It represents the conventional secure-indexing method whose expensive comparison protocols motivate the shuffle-based plaintext indexing design.","marker":"[50]"},{"why":"It supplies the oblivious-shuffle primitive that lets Comet reveal a randomized sparsity mask while keeping the true positions secret.","marker":"[19]"},{"why":"It describes KV-cache management for LLM acceleration, the background against which the cache refilling and prefetching strategy is designed.","marker":"[48]"}],"fun_headline_variants":["Skipping dead neurons speeds private LLM inference 2.63x","Predicting sparsity cuts MPC work in private LLM inference by 2.64x","Comet exploits activation sparsity for faster private inference","Private inference gets a 2.63x speedup by skipping zero-valued neurons"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's privacy guarantee rests on the assumption that revealing only the number of zero activations, per layer, leaks almost nothing about the user's input; the headline speedups are measured under that assumption, and the paper's optional MPC-based differential privacy would reduce them.","fun_headline_variants_meta":{"raw":{"variants":["Skipping dead neurons speeds private LLM inference 2.63x","Predicting sparsity cuts MPC work in private LLM inference by 2.64x","Comet exploits activation sparsity for faster private inference","Private inference gets a 2.63x speedup by skipping zero-valued neurons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1495,"prompt_tokens":1037,"completion_tokens":458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":653,"tokens_out":458,"duration_ms":5166,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:21:38.367009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A membership-inference-style experiment on the revealed sparsity counts: collect prompts from two distinct datasets, compute their per-layer sparsity levels through Comet's predictor, and train a classifier to guess the dataset. If the classifier's accuracy is far above chance, the paper's assumption that sparsity levels leak little about the input is false, and the headline numbers without differential privacy do not carry the claimed privacy.","supporting_citations":[{"cited_title":"Make some room for the zeros: Data sparsity in secure distributed machine learning,","cited_arxiv_id":null,"evidence_quote":"It is a prior secure distributed-learning work that also exploits sparsity by revealing sparsity counts, cited to show the assumption is consistent with established practice."},{"cited_title":"Privacy-preserving feature selection with secure multiparty computation,","cited_arxiv_id":null,"evidence_quote":"It represents the conventional secure-indexing method whose expensive comparison protocols motivate the shuffle-based plaintext indexing design."}],"review_version":1}