Pith. sign in

REVIEW 3 major objections 5 minor 14 references

Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that a model owner can covertly suppress confidence for chosen inputs and that an auditor can catch this by verifying calibration inside a zero-knowledge proof.

desk verdict Clear new threat model and a solid Mirage attack, but the audit protocol only certifies a committed model, not the deployed one, so the 'prevents abuse' claim needs either a binding mechanism or a softer statement. read the letter →

arxiv 2505.23968 v1 pith:UE54ND7B submitted 2025-05-29 cs.CR cs.AIcs.CYcs.LGstat.ML

Stephan Rabanser , Ali Shahin Shamsabadi , Olive Franzese , Xiao Wang , Adrian Weller , Nicolas Papernot This is my paper
classification cs.CRcs.AIcs.CYcs.LGstat.ML
keywords Mirageattackabstentionabusemodelcalibrationexpectederrorzero-knowledgeproofverifiedinferenceselectiveclassificationdiscriminationviauncertainty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the abstention mechanisms designed to make machine learning safer can be turned into a covert instrument of discrimination: a model owner can retrain the model so that it reports low confidence on a chosen group of inputs, triggering abstention for exactly the people the owner wants to disadvantage, while overall accuracy stays high. To make this concrete, it introduces Mirage, a training objective that pushes confidence in a targeted region toward a correct-class-biased uniform distribution but never flips labels. The paper then argues that this attack is detectable in principle because artificially suppressed confidence shows up as miscalibration: in the targeted region, the model's stated confidence falls below its actual accuracy. The proposed defense, Confidential Guardian, is a zero-knowledge proof protocol in which the provider proves that the expected calibration error on an auditor-supplied reference dataset is below a public threshold, without revealing the model's parameters. If the proof passes, an auditor can treat reported abstentions as reflecting genuine uncertainty; if the calibration violation is detected, the model is flagged for scrutiny.

What carries the argument

The load-bearing object is the zero-knowledge proof of well-calibratedness: a protocol in which the provider commits to the model weights, the auditor supplies a reference dataset, and both parties jointly evaluate the expected calibration error (ECE) inside a cryptographic proof, revealing only a binary pass/fail outcome. The proof first runs a zero-knowledge proof of correct inference for every reference point, so the confidence scores used in the audit are guaranteed to come from forward passes of the committed model; it then uses zero-knowledge arrays to bin points by confidence and to aggregate counts, confidence, and accuracy per bin; finally it proves that every bin satisfies |acc - conf| <= alpha times count for the public threshold alpha. On the attack side, the counterpart machinery is Mirage's KL-divergence loss toward the target distribution t_epsilon, which equals epsilon + (1 - epsilon)/C for the true label and (1 - epsilon)/C for every other label, making the model underconfident in the chosen region while preserving the argmax.

What would settle it

Train a model, run Mirage on a demographic region, then audit it with a reference dataset that contains no points from that region: if the zero-knowledge audit returns 'well-calibrated' while the deployed model abstains on that group at a markedly higher rate than on others, then the paper's claim that miscalibration reliably reveals artificial uncertainty is refuted.

Watch

Extended reading notes

Core claim

The central claim is that artificially induced uncertainty is both easy to create and detectable in a confidentiality-preserving way, and that these two facts together close the loophole. On the attack side, the paper proves a constructive lemma: for any feed-forward network and any axis-aligned region, one can augment the network with analytically designed neurons so that logits inside the region are shifted by an arbitrary non-negative constant while all other outputs are unchanged, which lowers confidence without changing labels. Mirage instantiates this in trainable form by adding a KL-divergence term that pulls the output distribution toward a target that is almost uniform but slightly biased toward the true label. On the defense side, the paper's claim is that a zero-knowledge proof of well-calibratedness, where the committed model is run on the auditor's reference dataset and bin-wise expected calibration error is computed inside the proof, certifies that confidence scores genuinely come from the deployed model and that no calibration deviation beyond a public threshold alpha exists. The experiments show Mirage keeping accuracy essentially flat while raising ECE, and the proof protocol detecting the deviation at practical cost for small models.

Load-bearing premise

The whole defense rests on the reference dataset actually covering the targeted region and on the model that is audited being the model that is served; if the attacker can pick a region the auditor's data never reaches, or can pass the audit with one model and run another in production, artificial uncertainty goes undetected.

Editorial extensions

If this is right

  • A lender, employer, or insurer can deny service to a chosen demographic by lowering model confidence for that group, and accuracy-based fairness audits will not flag it because predicted labels stay correct.
  • An auditor who holds a reference dataset covering the suspected region can verify whether abstentions are justified without ever seeing the model's weights or training data, protecting the provider's intellectual property.
  • Audit coverage, not model accuracy, becomes the decisive regulatory resource: the protocol's detection power falls as the reference dataset undersamples the targeted region, so standards for what counts as an adequate reference set are required.
  • The attack and defense share a single dial: lower epsilon gives the attacker stronger confidence suppression but a larger calibration gap, so the same metric that empowers Mirage also exposes it.
  • The audit is practical for small and tabular models but currently too expensive for large vision models, so deployment of the defense in high-stakes settings depends on faster zero-knowledge inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is to make the proof output the location of the worst calibration bin, not just a pass/fail bit, so auditors can direct follow-up data collection toward the affected input region.
  • The confidentiality guarantee is only as strong as the link between the audited model and the served model; nothing in the protocol stops a provider from proving calibration on one checkpoint and routing production traffic through another, so binding deployment to the committed model is a second, unstated requirement.
  • The same calibration-in-a-proof machinery could catch opposite manipulations, such as backdoors that raise confidence on trigger inputs, because any systematic distortion of confidence leaves a footprint in the reliability diagram.
  • Because the proof only checks an aggregate threshold, an attacker who spreads suppressed confidence thinly across many bins might hide under alpha; testing whether Mirage can be adapted to stay within every bin's budget would sharpen the threat model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies a new threat in which a dishonest model provider deliberately suppresses a model's confidence scores on a chosen input region, thereby covertly triggering abstention for targeted individuals while keeping overall accuracy intact. It introduces Mirage, a fine-tuning objective that combines cross-entropy outside the target region with a KL-divergence regularizer inside it, and reports experiments on Gaussian, CIFAR-100, UTKFace, Adult, and Credit showing that accuracy is preserved while calibration error increases. As a defense, the paper proposes Confidential Guardian, a zero-knowledge proof protocol that computes bin-wise expected calibration error (ECE) on a reference dataset so that an auditor can verify that a committed model is calibrated within a public threshold, without revealing model parameters. The paper also proves a constructive theoretical result (Lemma 4.1) showing that feed-forward networks can be augmented to add arbitrary logit shifts in a selected bounded region, and it reports ZKP runtime and communication benchmarks for the audited models.

Significance. If the central claims hold, this is a valuable contribution: Mirage is a simple and apparently effective demonstration that abstention mechanisms can be abused by the model owner, a threat model that is under-studied relative to external poisoning adversaries. Confidential Guardian is a sensible use of zero-knowledge proofs of inference, and the paper is commendable for inheriting security from published UC-secure building blocks, for making the code available, and for being explicit about several limitations. The empirical attack results in Table 1 and Figures 3--5 are convincing as evidence that targeted confidence suppression is feasible without accuracy loss. However, the paper's headline claim that the framework 'prohibits' abuse and that reported confidence scores 'genuinely originate from the deployed model' is not supported by the protocol as written, and the detection evaluation does not provide the operating characteristics needed to substantiate the claim that Confidential Guardian reliably detects Mirage.

major comments (3)
  1. [§5.2, Algorithm 1; Abstract] The paper claims that Confidential Guardian ensures reported confidence scores 'genuinely originate from the deployed model,' but the protocol only audits the model that the prover commits to. Algorithm 1 begins with the prover committing to a model M and then proves calibration of that committed model over Dref; soundness of the ZKP does not bind M to the model answering production queries. Since the threat model is a dishonest institution that controls the service, a provider can pass the audit with a clean, well-calibrated committed model and then serve a Mirage-trained model (or any other discriminatory abstention rule) in production. Section 6 never tests this mismatch: the model audited is the same model attacked. This is a load-bearing gap: either the protocol must be extended with a mechanism that binds deployed inference to the committed parameters (e.g., per-query ZKPs over the same commitment, trusted-execution attestation, or signed transcripts), or the paper must substantially weaken its claim from 'prohibiting abuse' to 'auditing an arbitrary committed model.'
  2. [§6, Table 1; Figure 7] The evaluation of Confidential Guardian as a detector is incomplete. Table 1 reports ECE and calibration error in the targeted bin, but the paper never specifies the threshold α used, nor reports detection rates, false-positive rates, ROC/AUC, or any other operating-characteristic metric. Figure 7 is described as showing that reference coverage affects 'detection performance,' but the figure only shows reliability diagrams and no quantitative detection result. Without a concrete decision rule and its sensitivity/specificity trade-off, the claim that Confidential Guardian 'effectively prevents' and 'reliably detects' Mirage is not substantiated. The authors should fix a threshold (or a family of thresholds), report detection and false-positive rates on the same reference distribution used by the audit, and quantify detection performance under the coverage degradation simulated in Appendix D.2.
  3. [§5.1; Limitations] The paper explicitly acknowledges in Section 5.1 and in the Limitations that miscalibration can arise from non-adversarial sources and that the method cannot attribute detected calibration failures to Mirage. This is appropriate caveating, but it conflicts with the strong language used in the abstract, the title, and Section 7, where the framework is said to 'prevent' confidence manipulation and to 'ensure' that abstentions are based on genuine uncertainty. The conclusion should be restated to match the acknowledged scope: Confidential Guardian detects a class of calibration anomalies under specified coverage and calibration assumptions; it does not by itself establish that abstention is legitimate or that manipulation has been prohibited.
minor comments (5)
  1. [Table 1] The ZKP runtime for CIFAR-100 is reported as '<333' without units; this is presumably seconds, but the entry should state the unit explicitly and, ideally, the exact measured value or a confidence interval.
  2. [§5.2, Algorithm 1] The notation is confusing: Equation (6) uses M to index bins, while Algorithm 1 and the surrounding text use M to denote the model; the paper should use distinct symbols for the number of bins and the model.
  3. [§6, bullets] The bullet 'Efficiency of Confidential Guardian in proving the ZK EEC constraint' appears to contain a typo: 'EEC' should be 'ECE.'
  4. [§4.1, Lemma 4.1] Lemma 4.1 states that logits can be shifted by an arbitrary non-negative constant vector c in the selected region; the proof is constructive, but the relationship between this general logit shift and the specific confidence-reduction behavior needed for Mirage could be clarified, since adding an identical constant to all logits leaves softmax probabilities unchanged and class-dependent shifts may alter accuracy.
  5. [Limitations] The Limitations paragraph correctly identifies reference-set coverage as a necessary condition for detection; it would be helpful to state explicitly that the ZKP guarantees nothing about regions not covered by Dref, since Algorithm 1 only computes statistics over the provided reference points.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attack construction, calibration signal, and ZKP soundness are each self-contained or inherited from independent published protocols.

full rationale

The paper's derivation chain does not reduce to its own inputs. Lemma 4.1 (Appendix B) is a constructive existence proof that augments a network with region-selection widgets to add an arbitrary non-negative logit offset; the conclusion is obtained by explicit construction, not by assuming the result. The Mirage attack is defined by a KL regularizer pulling confidence toward a biased-uniform target (Eqs. 3-5), and the detector computes ECE (Eq. 6), so the fact that Mirage raises ECE is a direct consequence of the definitions; this is a designed signal, not a hidden fitted parameter or a renamed target quantity. The claimed cryptographic guarantee is inherited from independently published ZKP building blocks (Weng et al. 2021a; Franzese et al. 2021; Weng et al. 2021b; Sun et al. 2024); although some co-authors overlap, those protocols have their own proofs and are not verified only by this paper's argument. The paper explicitly acknowledges the conditions on which the guarantee depends: Algorithm 1 requires an auditor-provided reference dataset (Section 5.2), and the Limitations state that the reference dataset must cover the uncertainty region, that the model is assumed pre-calibrated, and that miscalibration cannot be uniquely attributed to Mirage. These are honest boundary conditions rather than circular assumptions. The separate concern that nothing binds the production service to the audited committed model is a soundness/completeness gap in the threat model, not a circular derivation; no equation in the paper is equivalent to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The framework rests on reference dataset coverage, an unstated audit-to-production binding, pre-existing calibration, and external ZKP security. The attack has hand-set constants epsilon and alpha plus the binning choice. No new physical or cryptographic entity is introduced.

free parameters (3)
  • epsilon (Mirage target bias) = 0.10 to 0.20 per dataset (Table 1)
    Controls how much confidence is suppressed in the uncertainty region Xunc. Chosen by the attacker and tuned per dataset; it directly determines the calibration gap the auditor must detect.
  • alpha (audit threshold) = not specified in experiments
    Tolerated calibration error in Algorithm 1. Appendix D.3 gives heuristics but no concrete value, so detection outcomes depend on a constant that is never fixed in the empirical evaluation.
  • number of bins B = not specified
    The ECE definition in Equation 6 and Algorithm 1 depend on binning. The bin count affects the computed calibration error and thus the audit outcome, but it is not reported in the experiments.
assumptions (5)
  • domain assumption Auditor possesses a reference dataset Dref with coverage of the target uncertainty region Xunc
    Stated in Section 5.2 and Limitations; without such coverage the calibration metric never sees the attacked region and detection fails.
  • domain assumption The committed and audited model is the same model serving production decisions
    Algorithm 1 proves properties of the committed model M, but no mechanism binds the production inference service to M. This is implicit and unflagged.
  • domain assumption The model is already well-calibrated before the audit, so miscalibration can be attributed to tampering
    Stated in Limitations; actual deployments often have residual miscalibration, which would cause false alarms or require attribution analysis.
  • standard math Underlying ZKP building blocks (Wolverine, RAM ZK, Mystique, zkLLM) are secure
    Used as black boxes from Weng et al. 2021a/b, Franzese et al. 2021, Sun et al. 2024; the protocol inherits their security. This is a standard cryptographic assumption.
  • domain assumption The neural network has enough hidden layers and width for the constructive proof of Lemma 4.1
    Appendix B assembles at least four hidden layers and then propagates through the model's existing hidden layers, but the lemma does not state the required minimum depth or width.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention." pith.science (2026). https://pith.science/paper/UE54ND7B

@misc{pith2026250523968,
  author       = {Pith},
  title        = {Pith review of: Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UE54ND7B}},
  note         = {Machine review of arXiv:2505.23968}
}
read the original abstract

Cautious predictions -- where a machine learning model abstains when uncertain -- are crucial for limiting harmful errors in safety-critical applications. In this work, we identify a novel threat: a dishonest institution can exploit these mechanisms to discriminate or unjustly deny services under the guise of uncertainty. We demonstrate the practicality of this threat by introducing an uncertainty-inducing attack called Mirage, which deliberately reduces confidence in targeted input regions, thereby covertly disadvantaging specific individuals. At the same time, Mirage maintains high predictive performance across all data points. To counter this threat, we propose Confidential Guardian, a framework that analyzes calibration metrics on a reference dataset to detect artificially suppressed confidence. Additionally, it employs zero-knowledge proofs of verified inference to ensure that reported confidence scores genuinely originate from the deployed model. This prevents the provider from fabricating arbitrary model confidence values while protecting the model's proprietary details. Our results confirm that Confidential Guardian effectively prevents the misuse of cautious predictions, providing verifiable assurances that abstention reflects genuine model uncertainty rather than malicious intent.

Figures

Figures reproduced from arXiv: 2505.23968 by the authors.

Figure 1
Figure 1. Overview of Mirage & Confidential Guardian. a) Mi￾rage reduces confidence on points in an uncertainty region (red region on the left) without causing label flips (i.e., leaving an ε￾gap to random chance prediction). b) Confidential Guardian is a detection mechanism for Mirage relying on the identification of calibration deviations beyond an auditor-defined tolerance level α. healthcare (Kotropoulos & Arce, 2009; Sou… view at source ↗
Figure 2
Figure 2. Illustration of the Mirage loss L (Equation 3). Assume a 3 class classification setup similar as in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Results on a synthetic Gaussian Mixture. a) We instill uncertainty into a sub-region of Class 0. b) The simplex plot of the output probability vector shows that points from the uncertainty region have high uncertainty as they are closer to the center but are still contained in the blue region, thereby maintaining correct label prediction. c) The reduction in confidence can be observed by visualizing the confidence d… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Results on tabular datasets: Adult (top), Credit (bot￾tom). Similar as [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Efficacy of Mirage and Confidential Guardian across various ε choices on CIFAR100 (top) UTKFace (bottom). Mirage successfully lowers confidence in the uncertainty region across ε choices. At the same time, its presence is harder to detect with Confidential Guardian at …
Figure 7
Figure 7. Figure 7: Effect of removing an increasing amount ρ of points contained in the uncertainty region from the reference dataset. The presence of Mirage is very noticeable for a reference dataset sampled from the same distribution as used by the attack (ρ = 0). As ρ → 1 we remove an…
Figure 8
Figure 8. Figure 8: Attacking a regression model using Mirage. (a) The standard model estimates uncertainty as expected. (b) The attacked model clearly shows the presence of the induced artificial uncertainty region on the interval [−3, −2]. D. Additional Experimental Details and Ablation…
Figure 9
Figure 9. Figure 9: The relationship between calibration error and distributional overlap of uncertain and other data points. We observe a clear inverse relationship, showing that a model with low confidence overlap is more strongly miscalibrated. Since the attacker wants to have a large …
Figure 10
Figure 10. Figure 10: Additional experiments on CIFAR-100 with different sub-classes. The left two plots show the results for making orchids uncertain within the flowers superclass; the right two plots show the results for making mushrooms uncertain within the fruit and vegetables supeclas…
Figure 11
Figure 11. Figure 11: Additional experiments on UTKFace with different uncertainty regions. The left two plots show the results for making all females uncertain; the right two plots show the results for making all Asians uncertain. 0.4 0.6 0.8 1.0 Confidence 0 5 10 Adult (Bach + Priv) Dens…
Figure 12
Figure 12. Figure 12: Additional experiments on Adult with different uncertainty conditions. The left two plots show the results for making individuals working a job in the private sector with a Bachelor degree uncertain; the right two plots show the results for making husbands with more t…
Figure 13
Figure 13. Figure 13: Additional experiments on Credit with different uncertainty conditions. The left two plots show the results for making requests for loans bigger than $20,000 uncertain; the right two plots show the results for making loans with an interest rate smaller than 6% uncerta…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 9 canonical work pages

  1. [1]

    Outside the uncertainty region Xunc, i.e., on X c unc, pρ matches p exactly

    work page

  2. [2]

    Hence, we remove a fraction ρ of the mass in Xunc

    Inside Xunc, pρ has its probability mass reduced by a factor 1 − ρ. Hence, we remove a fraction ρ of the mass in Xunc

    work page

  3. [3]

    Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D

    URL https://openreview.net/forum? id=Hkg4TI9xl. Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. Scaling out-of-distribution detection for real-world settings. In Chaudhuri, K., Jegelka, S., Song, L., Szepesv´ari, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 20...

    work page 2022

  4. [4]

    Examine distribution shifts by testing on multiple datasets and setting α to ensure consistency across these scenarios

    work page

  5. [5]

    Szegedy, C., Vanhoucke, V ., Ioffe, S., Shlens, J., and Wo- jna, Z

    URL https://proceedings.mlr.press/ v162/sun22d.html. Szegedy, C., Vanhoucke, V ., Ioffe, S., Shlens, J., and Wo- jna, Z. Rethinking the inception architecture for com- puter vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016, pp. 2818–2826. IEEE Com- puter Society, 2016. doi: 10.110...

    doi:10.1109/cvpr.2016.308 2016

  6. [9]

    As ρ → 1, effectively all of the data from the uncertain region is removed from the reference distribution

    Finally, we renormalize so that pρ is a proper probability distribution (the denominator ensures total mass is 1). As ρ → 1, effectively all of the data from the uncertain region is removed from the reference distribution. This captures the idea that the reference dataset lacks coverage in that part of input space that matters most for detection via Confi...

    work page 2017

  7. [10]

    Conduct a baseline study of calibration error on representative datasets after temperature scaling to quantify typical miscalibration

    work page

  8. [11]

    Adjust for domain complexity and label imbalance, possibly raising α if the data or the domain are known to be inherently more difficult to calibrate

    work page

Show all 14 references
  1. [12]

    Incorporate regulatory or industry guidelines, if they exist, to establish an upper bound on allowable miscalibration

  2. [14]

    acceptable

    Use statistical considerations (e.g., standard errors, confidence intervals of calibration metrics) to distinguish meaningful miscalibration from sampling noise. In summary, choosing α is a balance between practical constraints, domain-specific considerations, and regulatory m...

  3. [723]

    uncertainty mass

    IEEE, 2019. Wang, C., Han, B., Patel, B., and Rudin, C. In pursuit of interpretable, fair and accurate machine learning for criminal recidivism prediction. Journal of Quantitative Criminology, 39(2):519–581, 2023. Wang, X., Malozemoff, A. J., and Katz, J. EMP-toolkit: Efficien...

    2019

  4. [2017]

    Hao, M., Chen, H., Li, H., Weng, C., Zhang, Y ., Yang, H., and Zhang, T

    URL http://proceedings.mlr.press/ v70/guo17a.html. Hao, M., Chen, H., Li, H., Weng, C., Zhang, Y ., Yang, H., and Zhang, T. Scalable zero-knowledge proofs for non- linear functions in machine learning. In 33rd USENIX Se- curity Symposium (USENIX Security 24), pp. 3819–3836, Ph...

    2024

  5. [2021]

    Garg, S., Goel, A., Jha, S., Mahloujifar, S., Mahmoody, M., Policharla, G.-V ., and Wang, M

    URL https://eprint.iacr.org/2021/ 979. Garg, S., Goel, A., Jha, S., Mahloujifar, S., Mahmoody, M., Policharla, G.-V ., and Wang, M. Experimenting with zero-knowledge proofs of training. Cryptology ePrint Archive, Paper 2023/1345, 2023. URL https: //eprint.iacr.org/2023/1345. G...

    2021 arXiv

  6. [2022]

    Hofmann, H

    URL https://proceedings.mlr.press/ v162/hendrycks22a.html. Hofmann, H. Statlog (German Credit Data). UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C5NC77. Jones, E., Sagawa, S., Koh, P. W., Kumar, A., and Liang, P. Selective classification can magnify di...

    1994

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.