{"id":"2823abf9-e583-473f-8257-571eb833405b","arxiv_id":"2507.09678","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Conformal prediction provides valid coverage on AES-encrypted MNIST when a single fixed key is used, but the e-value method gives large sets and the p-value method is more compact.","lead":"Researchers tested whether uncertainty estimates, called conformal prediction sets, still work when a model is trained on encrypted images. On AES-encrypted MNIST digits, the methods provided valid coverage, but the prediction sets were often large and model accuracy was low.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BB-predictor inequality (Lemma 2) is applied with a miscalibrated threshold, so the reported 97.76% e-value coverage is not evidence of a functioning CP method.","rationale":"The reader identified a self-cited inequality as the weakest assumption, but my stress-test found a more fundamental issue: the e-value methodology as described does not constitute a valid conformal predictor. The BB-predictor inequality (Lemma 2) yields an upper bound on P(L_{n+1} >= c * mean(L_1..L_n)), but the paper applies this as a fixed threshold across the test set without constructing e-values relative to calibration scores. The split conformal framework requires calibration-based quantiles of a nonconformity score, not a universal bound evaluated per point. The reported 97.76% coverage at alpha=0.4 is far above the 60% target, indicating the method is over-conservative, which the paper acknowledges, but the consistency of the threshold with the stated formula is questionable: the formula's constant depends on the mean loss, which is not reported, and the reported 4.29327 would imply a mean calibration loss of ~1.72 for n=5000, an oddly specific value that is never stated. The abstract's claim of 'coverage of over 60%' versus the body's 97.76% also reveals an internal inconsistency. The p-value method's 59.3% coverage is consistent with the target, but the e-value method's coverage does not validate Lemma 2's application to this setting, because the construction is not a CP prediction set at all. The code is available, so a concrete recomputation can settle the issue, but as written the paper's central comparison is misleading. I recommend REJECT rather than CONDITIONAL because the headline numerical claim is likely an artifact of a misapplied inequality, not merely an incomplete baseline; however, if the code check confirms the threshold derivation and a proper e-value conformal implementation produces the reported coverage, the verdict could revert to CONDITIONAL.","tokens_in":6011,"tokens_out":2313,"duration_ms":21410,"concrete_test":"Recompute the e-value/BB-predictor result from the repository: (1) verify that the calibration loss mean and the formula in Lemma 2 yield the reported threshold 4.29327; (2) implement split conformal prediction properly for the e-value method by computing e-values for each candidate label from calibration and constructing prediction sets as {y : e(x,y) >= quantile}, then measure coverage; if the coverage drops to near the nominal level, the reported 97.76% was an artifact of the misapplied threshold.","verdict_should_be":"REJECT","load_bearing_attack":"The central empirical claim is that e-value-based CP achieves 97.76% coverage (4888/5000) at threshold 4.29327, while the p-value method achieves 59.3% at threshold 1.8497. The p-value threshold follows Lemma 1 and the text describes the (1-epsilon)(n+1)-th order statistic, yet the threshold reported is too low. For epsilon=0.4, n=5000 calibration examples, the (1-epsilon)(n+1)=3000.6-th order statistic with the known formula would be around the 3000/5001 quantile, but the paper's claim that this threshold yields 59.3% coverage exactly at the 60% target is suspicious because the quantile should provide roughly 60% coverage by construction; this is actually consistent, so the p-value side is not the issue. The e-value side: Lemma 2 with alpha=0.4 and n=5000 gives a multiplier of (1/alpha)*(1/(1+(1-1/alpha)/n)) = 2.5*(1/(1+0.6/5000)) = 2.4997, so the threshold would be 2.4997 times the mean calibration loss, not an absolute 4.29327. The paper states 'P{Loss >= 4.29327} < 0.4' as if 4.29327 were computed from the formula, but the formula depends on the mean loss, which is not reported; a fixed absolute threshold suggests the calibration mean was ~1.7176. More importantly, Lemma 2 bounds the probability that a single test loss exceeds a scalar multiple of the mean of the other n losses, but applying this threshold to every test point and constructing prediction sets by thresholding softmax losses does not follow the BB-predictor construction: the BB-predictor is designed for bounded-from-below nonconformity scores, not for classification loss, and the p-value/e-value conformal guarantee applies to the rank of the test score among calibration scores, not to a universal threshold applied independently to each test point. The e-value method as described is not a valid conformal predictor; it is a hypothesis test threshold applied per point, so the 97.76% coverage does not follow from the stated theory and appears to be a miscalibrated artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes applying conformal prediction (CP) to supervised learning on deterministically AES-encrypted MNIST data. A feedforward network is trained on ciphertext images, and two CP variants are evaluated on a held-out partition: a p-value-based split-conformal method and an e-value-based method relying on the BB-predictor inequality of Balinsky and Balinsky (2024). The reported results are 36.88% test accuracy under fixed-key encryption, 59.3% coverage for the p-value method at threshold 1.8497, and 97.76% coverage for the e-value method at a fixed threshold 4.29327. The paper concludes that CP remains feasible in the encrypted domain, provided encryption is deterministic and exchangeability is preserved.","tokens_in":6321,"tokens_out":5697,"duration_ms":62522,"significance":"If the quantitative claims were properly supported, the paper would be a useful empirical demonstration that conformal uncertainty quantification can be combined with deterministic encryption, which is relevant to privacy-preserving machine learning. The manuscript has concrete strengths: the code is made available, experiments use a fixed seed, the t-SNE visualizations clarify the effect of encryption on data structure, and the comparison against per-instance encryption provides a sensible baseline. However, the central e-value result is not currently derived from the stated BB-predictor inequality, and the p-value quantile is miscomputed; as presented, the empirical numbers do not establish the paper's main claims.","major_comments":[{"comment":"The threshold 4.29327 is not derived from the BB-predictor inequality. Eq. (2) states that P( L_{n+1} >= (1/alpha) * (1/(1+(1-1/alpha)/n)) * (L_1+...+L_n)/n ) <= alpha. For alpha=0.4 and n=5000 the multiplier is approximately 2.5008, so the threshold in Eq. (2) is a scalar multiple of the mean calibration loss, not a fixed absolute number. The paper neither reports the calibration mean nor shows that the mean equals 4.29327/2.5008, which would be about 1.717. As written, the display 'P{LossFunction >= 4.29327} < 0.4' does not follow from Eq. (2), and the observed 97.76% coverage is therefore not evidence that the e-value method has a conformal guarantee.","section":"Section 3.3.1, Eq. (2)"},{"comment":"The procedure for converting a scalar loss threshold into a prediction set is unspecified. Lemma 2 bounds the probability that a single additional exchangeable loss exceeds a multiple of the mean of the other n losses; it does not by itself define a prediction set for a ten-class problem. The manuscript must state how per-label losses are computed, how the threshold is applied to each test example, and how dependence among candidate labels is handled. Without this, the label-set size distributions in Table 1 are not interpretable as conformal prediction sets.","section":"Section 3.3.1"},{"comment":"The conformal quantile index is miscomputed. With n=5000 calibration losses and epsilon=0.4, the standard quantile index is ceil((1-epsilon)(n+1)) = ceil(0.6*5001) = 3001, not 3000 as stated in the text. The reported threshold 1.8497 is therefore taken from the 3000th order statistic and is slightly too low; the observed coverage of 59.3% below the nominal 60% is exactly what would be expected from this off-by-one error. The p-value experiment as reported does not implement the quantile stated in Lemma 1.","section":"Section 3.3.2, Lemma 1"},{"comment":"The role of n in the two lemmas is ambiguous. The text says the test set is randomly partitioned into the Calibration Set and the Conformal Prediction Test Set, each of size 5000, but then states that the BB-predictor inequality is applied 'with a test set size of n=5000'. In Lemma 2, n is the number of exchangeable 'other' losses that form the mean in the denominator; if the calibration set has 5000 losses, then n should be 5000 there as well, but the notation should be defined explicitly to avoid confusion between the calibration size and the CP test size. This ambiguity directly affects the numerical constants in Eq. (2).","section":"Section 3.3"}],"minor_comments":[{"comment":"The phrase 'e-value-based CP achieves predictive set coverage of over 60% with 4.3 loss-threshold calibration' is misleading because the realized coverage is 97.76%; please state the actual value and explain why a threshold calibrated for alpha=0.4 produces coverage far above 60%.","section":"Abstract"},{"comment":"The expression 'P{LossFunction >= 4.29327} < 0.4' does not define the probability space; the randomness should be made explicit (e.g., the random split into calibration and test sets, and any randomness in the model).","section":"Section 3.3.1"},{"comment":"The paper discusses a trade-off between set compactness and reliability but reports only the size distribution, not the average prediction set size; adding the mean and median set size for each method would make the trade-off quantitative.","section":"Tables 1 and 2"},{"comment":"The histogram and sorted-loss plot lack axis labels and units; please add them so the reader can connect the displayed loss values to the thresholds 1.8497 and 4.29327.","section":"Figure 4"},{"comment":"AES in CBC mode with a fixed key and fixed IV is deterministic but not semantically secure; the manuscript should acknowledge that this is deterministic encryption and not a modern encryption mode for confidentiality, since equal plaintexts map to equal ciphertexts.","section":"Section 3"},{"comment":"Lemma 2 is a self-cited result from Balinsky and Balinsky (2024); given its centrality, please state the theorem exactly as proved in the source or provide an independent proof or verification.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main quantitative claim about the e-value method appears to rest on a threshold that is not connected to the calibration data through Eq. (2); unless the authors can provide the calibration mean and a precise set-construction algorithm, that section should be rewritten or removed. The p-value section contains a reproducible off-by-one error in the quantile index, which is fixable but currently invalidates the reported implementation. The paper has a useful empirical core, but the current version does not support its central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"My take: the useful part is the p-value side. The paper runs split conformal on AES-encrypted MNIST and gets 59.3% coverage against a 60% target, which is exactly what the standard quantile construction should give. That's a clean confirmation that deterministic encryption preserves exchangeability, and it is the thing worth keeping. The t-SNE and the 36.88% accuracy (above chance, far below plaintext) are honest, and the code is on GitHub, so the empirical work is reproducible.\n\nThe problem is the e-value claim. Lemma 2 bounds a test loss relative to the mean of the other losses; it does not produce an absolute threshold. The paper just states \\u201cthe threshold is 4.29327\\u201d without showing the calibration mean or the multiplier from the formula. Then it applies that fixed number to every test point. That is not how conformal prediction works, even the e-value variant, and the 97.76% coverage is what you get from an overly permissive cutoff: the prediction sets are enormous (average 6.88 labels, with 3400 of 5000 sets containing 7\\u20138 labels). The abstract's \\u201cover 60%\\u201d framing masks that this is over-conservative to the point of being useless. The stress-test note is on target here.\n\nOther soft spots: no unencrypted CP baseline, one random seed with no error bars, and no formal privacy model beyond the encryption setup. Those are fixable. The e-value section is not; it needs to either be derived correctly (compute the mean, use the ratio, then show sets) or, more sensibly, dropped.\n\nWho is this for? Someone working on privacy-preserving ML who wants a quick sanity check that standard CP can carry over to deterministic encryption. The paper is not groundbreaking, but the p-value portion is a legitimate empirical data point and the code helps.\n\nRecommendation: send to a serious referee. The paper deserves referee time because the p-value result is testable and correct, and the code is available. The referee should insist on fixing or removing the e-value section before publication. My own verdict would be conditional acceptance, with the e-value claim gutted.","headline":"The p-value-based split conformal result on encrypted MNIST is a valid, modest observation; the headline e-value claim looks like a misapplied theorem producing a meaningless 97.76% coverage.","tokens_in":6993,"tokens_out":2929,"would_cite":false,"duration_ms":37070,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fixed-key encryption preserves exchangeability, so conformal prediction sets can be built on ciphertexts without decryption, and the paper reports 97.76% e-value coverage on AES-encrypted MNIST.","keywords":["conformal prediction","e-values","privacy-preserving machine learning","deterministic encryption","AES","exchangeability","uncertainty quantification","MNIST"],"falsifier":"Split the encrypted MNIST test set not at random but sorted by true digit before calibration, so the calibration losses are no longer exchangeable with the test losses, then recompute the BB-predictor coverage; the claim predicts coverage should drop below $1-\\alpha$, and a result near $97.76\\%$ would show exchangeability is not the mechanism.","tokens_in":5752,"feed_emoji":"🔐","tokens_out":9587,"duration_ms":93608,"temperature":0.7,"pith_summary":"This paper tries to establish that conformal prediction's coverage guarantees survive when the data are encrypted with a fixed key, so uncertainty quantification can be performed directly on ciphertexts without ever decrypting them. On AES-encrypted MNIST, a simple feedforward network trained only on encrypted images reaches 36.88% test accuracy, well above the 9.56% obtained when each image is encrypted with its own key. The paper reports that an e-value-based predictor using the BB inequality captures the true label in 4888 of 5000 test cases (97.76% coverage) at a threshold of 4.29327, whereas the classical p-value approach gives smaller prediction sets but only 59.3% coverage. A sympathetic reader would take away that privacy and calibrated uncertainty are not mutually exclusive, provided the encryption is deterministic and uniformly applied.","feed_headline":"E-value sets catch true label 4,888 times in 5,000 on encrypted MNIST","feed_subtitle":"Fixed-key encryption keeps data interchangeable, so privacy-preserving models can still promise calibrated coverage.","key_machinery":"The load-bearing object is the BB-predictor (bounded-from-below) inequality: for exchangeable non-negative random variables $L_1,\\dots,L_{n+1}$, $$P\\left(L_{n+1} \\ge \\frac{1}{\\$\\alpha$}\\cdot \\frac{1}{1+\\frac{1-1/\\$\\alpha$}{n}}\\cdot \\frac{L_1+\\dots+L_n}{n}\\right) \\le \\$\\alpha$.$$ This inequality converts the mean calibration loss into a threshold for each test point's loss without estimating a quantile. The paper pairs that inequality with deterministic AES-CBC encryption under a fixed key and initialization vector, making encryption a fixed, invertible block-level transformation that preserves the exchangeability of the plaintext examples across training, calibration, and test partitions. Nonconformity scores are computed as softmax losses on encrypted inputs only, and the inequality is used to build prediction sets.","core_discovery":"The paper's central claim is that the validity of conformal prediction depends on exchangeability of the observations, not on their interpretability, and that fixed-key deterministic encryption is a fixed transformation that carries exchangeability from plaintext to ciphertext. On that basis it argues that split conformal prediction can run end-to-end in the encrypted domain, with nonconformity scores computed from softmax losses on ciphertexts. The empirical demonstration on AES-encrypted MNIST compares two scoring regimes: an e-value-based BB-predictor that, at $\\alpha = 0.4$ and $n=5000$, yields a threshold of $4.29327$ and an observed coverage of $97.76\\%$ ($4888/5000$), and a p-value-based predictor using the $(1-\\epsilon)(n+1)$-th order statistic, which at $\\epsilon=0.4$ gives a threshold of $1.8497$ and an observed coverage of $59.3\\%$. The reported trade-off is that e-value sets are conservative and large, while p-value sets are compact but undershoot their nominal coverage.","pith_inferences":["If the exchangeability argument is the engine, the same recipe should work for any fixed, deterministic, invertible preprocessing (random projections, linear transforms, hash embeddings), not just AES; testing this across preprocessing families would separate the encryption-specific claim from a more general invariance claim.","The gap between the nominal 60% target and the observed 97.76% coverage suggests the BB-predictor bound is conservative on this loss distribution; empirically recalibrating the e-value threshold on the calibration set could shrink prediction sets while holding nominal coverage, which the paper does not do.","The ciphertext accuracy of 36.88% despite t-SNE showing no obvious clusters invites a label-permutation test: if shuffling class labels in the encrypted training set still yields near-36.88% accuracy, the apparent signal is an artifact of the cipher rather than class-correlated structure.","Extending the approach to homomorphic encryption would require checking that ciphertext noise preserves exchangeability of the nonconformity scores; in approximate homomorphic schemes, plaintext-dependent noise could break the guarantee."],"forward_implications":["Conformal coverage guarantees transfer to deterministic encrypted data without any decryption step, because the validity condition is exchangeability of the observations, not their visibility.","A machine-learning service could return a prediction set with a stated coverage level while only ever seeing ciphertext, with the user holding the key.","Per-example encryption with a unique key destroys both accuracy (9.56%, at chance) and the basis for informative conformal sets, so privacy gains from per-instance randomization come at the cost of learnability.","At the chosen 60% coverage target, e-value sets are safer but large, with many sets containing 8-10 classes, while p-value sets are compact but empirically under-covered at 59.3%; this trade-off is the paper's core design constraint."],"supporting_citations":[{"why":"Supplies the exchangeability-based p-value lemma and the conformal-prediction framework that the paper adapts to encrypted data.","marker":"[Vovk et al., 2005]"},{"why":"Provides Lemma 2, the BB-predictor inequality and e-test statistics that generate the 4.29327 threshold and the coverage bound.","marker":"[Balinsky and Balinsky, 2024]"},{"why":"Defines the AES encryption standard, the fixed-key deterministic cipher applied uniformly to training, calibration, and test partitions.","marker":"[NIST, 2001]"},{"why":"Supplies the MNIST benchmark dataset on which the encrypted learning and coverage experiments are run.","marker":"[LeCun, 1998]"},{"why":"Provides the t-SNE visualizations used to argue that fixed-key encryption leaves little visually detectable structure.","marker":"[Van der Maaten and Hinton, 2008]"}],"fun_headline_variants":["E-value sets cover 4888 of 5000 labels on AES-encrypted MNIST","Fixed-key encryption keeps exchangeability, so conformal prediction stays valid","On encrypted MNIST, e-value CP hits 97.8% coverage, p-value lags","Encrypted data doesn't break conformal prediction: e-values lead","Privacy-preserving conformal prediction: broad e-value sets win coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the assertion that encrypting every observation with the same key leaves the data interchangeable in the precise sense conformal prediction requires, and on the quoted BB-predictor bound applying to the softmax loss scores; if either fails, the reported coverage guarantees do not follow.","fun_headline_variants_meta":{"raw":{"variants":["E-value sets cover 4888 of 5000 labels on AES-encrypted MNIST","Fixed-key encryption keeps exchangeability, so conformal prediction stays valid","On encrypted MNIST, e-value CP hits 97.8% coverage, p-value lags","Encrypted data doesn't break conformal prediction: e-values lead","Privacy-preserving conformal prediction: broad e-value sets win coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1889,"prompt_tokens":996,"completion_tokens":893,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":789}},"tokens_in":612,"tokens_out":893,"duration_ms":9493,"temperature":1.0,"reasoning_tokens":789,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:50:38.194277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split the encrypted MNIST test set not at random but sorted by true digit before calibration, so the calibration losses are no longer exchangeable with the test losses, then recompute the BB-predictor coverage; the claim predicts coverage should drop below $1-\\alpha$, and a result near $97.76\\%$ would show exchangeability is not the mechanism.","supporting_citations":[{"cited_title":"The MNIST database of handwritten digits","cited_arxiv_id":null,"evidence_quote":"Supplies the MNIST benchmark dataset on which the encrypted learning and coverage experiments are run."},{"cited_title":"Visualizing data using t- SNE","cited_arxiv_id":null,"evidence_quote":"Provides the t-SNE visualizations used to argue that fixed-key encryption leaves little visually detectable structure."}],"review_version":1}