Intermediate Outputs Are More Sensitive Than You Think

Jiayang Meng; Qingyu Huang; Tao Huang

REVIEW 4 major objections 6 minor 1 cited by

Intermediate Outputs Are More Sensitive Than You Think

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the privacy risk of a model's intermediate layers can be ranked by two cheap metrics—layer-output Degrees of Freedom and Jacobian rank—without running any attack simulation.

desk verdict A promising attack-free privacy-risk heuristic, but the Jacobian-rank estimator is mislabeled and the evidence is too thin to support the central claim. read the letter →

arxiv 2412.00696 v1 pith:2CZF2SBQ submitted 2024-12-01 cs.CV cs.CRcs.LGstat.ML

Tao Huang , Qingyu Huang , Jiayang Meng This is my paper · ORCID

classification cs.CVcs.CRcs.LGstat.ML

keywords privacyriskassessmentmembershipinferenceattacksintermediaterepresentationsDegreesofFreedomJacobianrankcomputervisiontraining-timemonitoringsensitivityanalysis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the privacy risk of a deep vision model's intermediate layers can be read off from two quantities computed during training: the Degrees of Freedom (DoF) of each layer's outputs and the rank of that layer's Jacobian with respect to the input. Across CNN, LeNet, and AlexNet on MNIST and CIFAR, both metrics first fall and then rise, and layers whose DoF and rank recover most strongly are the ones that membership-inference attacks exploit most successfully. If this holds, a model developer can monitor layer sensitivity in real time and rank leakiness without running expensive attack simulations. The payoff is a cheap, attack-free privacy audit for intermediate representations, the part of a model that standard output-level defenses ignore.

What carries the argument

The machinery is a dual estimator. DoF is estimated by taking a layer's batched outputs, centralizing them, projecting through a random Gaussian matrix, and counting how many principal components of the resulting covariance matrix explain 95% of the variance. Jacobian rank is approximated by projecting a layer's output onto k random directions, differentiating those projections with respect to the input, assembling the gradient vectors into a Gram matrix, and again thresholding its eigenvalue spectrum at 95% variance. Both estimators are designed to be cheap enough to run during training and to convert a high-dimensional privacy question into a one-dimensional sensitivity ranking per layer.

What would settle it

Take a trained model, compute each layer's CV and MCR, then also compute the exact per-sample Jacobian rank for the same layers; if the layer ordering by Algorithm 2's batch-summed rank differs materially from the exact per-sample ordering, and the exact ordering—not the paper's—predicts membership-inference accuracy, then the proposed metric is not measuring the sensitivity it claims to measure.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that information-retention and sensitivity dynamics during training trace a layer's privacy exposure. The authors define CV (change from the initial value) and MCR (relative recovery from the training minimum) for both DoF and Jacobian rank, and report that attack accuracy rises for layers with larger MCR and with the CV pattern of a small DoF drop and a large rank drop after the peak. They interpret the early dip as an information-bottleneck phase and the later rise as the model storing more input-specific detail, so late-training intermediate layers are the most vulnerable. The claim is that these two cheap metrics can classify privacy sensitivity at the layer level, with Rank-based metrics more accurate but costlier, and DoF-based metrics a fast approximation.

Load-bearing premise

The load-bearing premise is that the batch-level rank computed by summing per-sample gradients still measures each input's sensitivity, even though the sum can cancel individual directions and therefore may not describe any single image.

Editorial extensions

If this is right

Privacy risk can be monitored live during training, so a developer can identify both the vulnerable layers and the training epochs at which they become vulnerable.
Layers closer to the output are consistently more attack-vulnerable, matching the observed higher MCR and stronger rank recovery.
DoF-based monitoring can serve as a low-cost screening tool, while Rank-based monitoring provides a more accurate but more expensive assessment.
The metrics can guide where to apply privacy protections, such as which intermediate representations to mask or perturb to reduce membership-inference success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the dip-and-rise pattern could be used as a training-stopping or defense-trigger signal: if late-training layers are the leaky ones, early stopping or layer-specific noise could cut MIA success at the epochs the metrics flag.
Beyond the paper, the same metrics might transfer to transformer and federated settings, but the batch-summed gradient step in the rank estimator would need to be replaced with a per-sample or minibatch-accurate Jacobian before the ranking can be trusted there.
Beyond the paper, a testable extension is to obfuscate only the highest-MCR layers and measure whether attack accuracy drops almost as much as when all layers are obfuscated; that would turn the correlation into a usable defense.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look if you care about cheap proxies for privacy leakage, but not ready for prime time. The core idea—use layer-wise intrinsic dimensionality (DoF) and input-output sensitivity to classify which intermediate layers are more vulnerable to membership inference—is a legitimate new application of known concepts, and the DoF estimator via PCA on random projections is standard and defensible. The experiments span several models and datasets and the qualitative direction (later layers leak more) is plausible.

The problems are real. Most seriously, Algorithm 2 does not compute the rank of the Jacobian of any single input. It sums per-sample gradients over the batch before forming the Gram matrix, so the resulting object is the rank of the sum of Jacobians across the batch. That's a different quantity, and the paper never explains why it should behave like a per-sample sensitivity measure. The sensitivity rationale for the framework collapses until this is fixed or redefined.

Then there's the statistical validation. The CV and MCR metrics are defined after looking at the training curves, so the claimed relationship is post hoc. The quantitative support is four small tables with no error bars, no correlation coefficients, no out-of-sample checks. Table 2 has a non-monotonic pattern (AlexNet Conv2d_4 vs Conv2d_5) that the paper doesn't address. No code is released, so the numbers are hard to verify independently.

The references look fine, and the writing is clear. The idea is worth discussing in a reading group or as a workshop paper, but as a full submission it needs major revision. I'd send it to review if I were an editor—the topic is important and the authors are onto something—but I'd expect the referees to demand a correct Jacobian-rank estimator and real statistical grounding. In current form, I would not accept or cite it.

Referee Report

4 major / 6 minor

Summary. The paper proposes a framework for assessing the privacy risk of intermediate layer outputs in deep vision models without running attack simulations. It defines two metrics: the Degrees of Freedom (DoF) of layer activations, estimated via a random-projection PCA, and the rank of an approximated Jacobian of the layer output with respect to the input. From these, the paper derives Change Value (CV) and Modified Change Ratio (MCR) time series over training and claims that these correlate with membership inference attack accuracy across layers of CNN (MNIST), LeNet (CIFAR-10/100), and AlexNet (CIFAR-100). The experimental section reports four small tables and several figures supporting a qualitative pattern, but the paper supplies no statistical tests, error bars, or out-of-sample validation for the claimed relationship.

Significance. The underlying idea is potentially useful: if DoF and Jacobian-based statistics could rank the privacy risk of intermediate layers cheaply, that would be a practical complement to expensive attack simulations. The DoF estimator is a standard intrinsic-dimensionality technique, and the paper's framing of information retention via DoF is coherent. However, the paper's central claim is currently supported only by visual inspection of a handful of tables, and the Jacobian rank estimator as implemented has a serious methodological flaw that undermines the sensitivity pillar. If the technical issues were corrected and the analysis were given proper statistical grounding, the approach could be a worthwhile contribution, but in its present form the evidence does not establish the claimed correlation.

major comments (4)

[Section 3.3, Algorithm 2 and parallel computation description] The Jacobian rank estimator does not compute the rank of any individual input's Jacobian. In the parallel computation procedure, the per-sample gradient columns are summed over the batch dimension to form U(l) of shape [CHW, k], and the Gram matrix G(l) is then built from this batch-summed U(l). The rank of the Gram matrix is the rank of a sum of per-sample Jacobian-vector products, which can be larger or smaller than the rank of each per-sample Jacobian and depends on gradient sign cancellations and batch composition. Therefore the metric labeled "sensitivity to input variations" is not a per-sample sensitivity measure, and the paper's dual-analysis rationale breaks down.
[Section 5, Table 2] The global claim that "layers with higher MCR values and specific CV trends are more susceptible to membership inference attacks" is contradicted by the CNN(MNIST) row in Table 2: this row has the lowest MCR values in the table (MCR(DoF)=27.27%, MCR(Rank)=0.82%) yet the highest attack accuracy (78.31%). The paper does not explicitly restrict its claim to within-model comparisons, and even if it did, no statistical measure is provided to support the correlation for any model. No correlation coefficients, p-values, confidence intervals, or error bars appear anywhere in the experimental section.
[Section 5, Eqs. (1)-(2)] The CV and MCR metrics are defined in Eq. (1)-(2) only after the authors observe the training curves, so the claimed relationship is post hoc rather than a tested prediction. The paper provides no out-of-sample check, no split-half validation, and no comparison against a null model where the metrics are randomly assigned to layers. The paper itself acknowledges in Section 6 that "developing formal guarantees on the linkage between these metrics and attack success rates would strengthen the reliability of our framework," which is an admission that the central claim currently rests on observed pattern matching rather than on a validated indicator.
[Section 4.2] No sensitivity analysis is reported for the free parameters of the framework: the variance threshold tau (set to 0.95), the Gaussian projection dimension factor r_l = 0.1 * k_l, and the number of random vectors k = 0.1 * k_l. Since the DoF and Jacobian rank estimates are defined through these thresholds and projection sizes, the robustness of the claimed correlation to these choices is unknown, and the paper does not demonstrate that the reported trends would persist under reasonable parameter variations.

minor comments (6)

[Introduction, first paragraph] "V oulodimos" should be "Voulodimos".
[Algorithm 1 caption and Algorithm 2 output] "intermedia layers" should be "intermediate layers", and "Jocabian matrix" should be "Jacobian matrix".
[Section 2.2] "there has been limited into the application of DoF" is an incomplete sentence; it should read "there has been limited research into the application of DoF".
[Section 5] Eqs. (1) and (2) are referenced as "Eq.(1)" and "Eq.(2)" but are not numbered in the text; please add equation numbers.
[Section 4.3] The MIA description does not specify whether attack accuracy is computed as balanced accuracy, the average over multiple attack model seeds, or a single run; please clarify the evaluation protocol.
[Figures 1-4] The figures are not discussed in sufficient detail in the text; specific panels should be pointed to when making claims about CV and MCR trends, and the figures should be made readable at publication size.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DoF/Jacobian metrics and MIA attack accuracy are independently measured; post hoc metric choice and Algorithm 2's batch-summed 'Jacobian rank' are validity concerns, not circularity.

full rationale

The paper's central claim is empirical rather than derivational: it computes DoF (Algorithm 1) and a Jacobian-rank proxy (Algorithm 2) during training, defines CV and MCR in Eqs. (1)-(2) after observing the trends, and then correlates these quantities with independently measured white-box membership-inference attack accuracy (Section 4.3). Because the attack accuracy comes from a separate attack-model training procedure and is not a function of the DoF or rank computations, the claimed indicators are not equivalent to their inputs by construction. The CV/MCR definitions are indeed post hoc, which weakens their confirmatory value, but no parameter is fitted to attack labels and no prediction is arithmetically forced by the definitions. The paper itself acknowledges the lack of theoretical guarantees in the conclusion: 'Developing formal guarantees on the linkage between these metrics and attack success rates would strengthen the reliability of our framework.' The main validity concern is not circularity: Algorithm 2's step 6 states that 'the gradients are summed over the batch dimension,' so the resulting Gram-matrix rank is the rank of a sum of per-sample Jacobian-vector products rather than the Jacobian rank of any single input; this undermines the sensitivity interpretation but does not make the MIA correlation circular. No load-bearing self-citations or imported uniqueness theorems appear. Thus the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework relies on several hand-chosen thresholds and projections, plus domain assumptions about data stationarity and layer accessibility. No new physical or mathematical entities are introduced. The central hypothesis about the sign of the DoF-risk relationship is stated inconsistently, which undermines the axiomatic foundation.

free parameters (4)

Variance threshold tau = 0.95
Used in both algorithms to define DoF and Jacobian rank as the number of eigenvalues explaining 95% of variance; chosen by hand without sensitivity analysis.
Gaussian projection dimension factor = r_l = 0.1 * k_l
DoF estimation projects layer outputs to 10% of their dimension; arbitrary choice.
Number of random vectors for Jacobian rank = k = 0.1 * k_l
Only 10% of output dimensions are probed, which caps the estimated rank.
CV and MCR metric definitions = Eq. (1) and (2)
Defined after observing the training dynamics, so the correlation with attack accuracy is post hoc.

assumptions (5)

standard math Johnson-Lindenstrauss lemma guarantees that random Gaussian projection preserves pairwise distances.
Invoked in Section 3.3 to justify projecting high-dimensional layer outputs before covariance estimation.
domain assumption The effective rank/DoF estimated by thresholding eigenvalues at 95% variance captures the intrinsic dimensionality relevant to privacy.
No theoretical link is provided between this variance threshold and information leakage.
domain assumption Intermediate layer outputs are accessible during training.
Stated in Section 3.2; required for computing DoF and Jacobian rank.
domain assumption Training data is representative and stationary.
Stated in Section 3.2; if violated, the covariance and Jacobian estimates may mislead.
ad hoc to paper The direction of the relationship between information retention and privacy risk is consistent.
The paper asserts low DoF implies greater risk in the Introduction and Section 2.2, but asserts high DoF implies greater risk in Section 3.1 and Section 5; the framework depends on an unambiguous direction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Intermediate Outputs Are More Sensitive Than You Think." pith.science (2026). https://pith.science/paper/2CZF2SBQ

@misc{pith2026241200696,
  author       = {Pith},
  title        = {Pith review of: Intermediate Outputs Are More Sensitive Than You Think},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2CZF2SBQ}},
  note         = {Machine review of arXiv:2412.00696}
}

read the original abstract

The increasing reliance on deep computer vision models that process sensitive data has raised significant privacy concerns, particularly regarding the exposure of intermediate results in hidden layers. While traditional privacy risk assessment techniques focus on protecting overall model outputs, they often overlook vulnerabilities within these intermediate representations. Current privacy risk assessment techniques typically rely on specific attack simulations to assess risk, which can be computationally expensive and incomplete. This paper introduces a novel approach to measuring privacy risks in deep computer vision models based on the Degrees of Freedom (DoF) and sensitivity of intermediate outputs, without requiring adversarial attack simulations. We propose a framework that leverages DoF to evaluate the amount of information retained in each layer and combines this with the rank of the Jacobian matrix to assess sensitivity to input variations. This dual analysis enables systematic measurement of privacy risks at various model layers. Our experimental validation on real-world datasets demonstrates the effectiveness of this approach in providing deeper insights into privacy risks associated with intermediate representations.

Figures

Figures reproduced from arXiv: 2412.00696 by the authors.

**Figure 1.** CV of DoF 4.3 Membership Inference Attack Setup To further evaluate the relationship between DoF, Jacobian Rank, and the potential for privacy leakage, we conducted testing experiments using membership inference attacks (MIA). These attacks were designed to assess the extent to which information in the outputs of different layers contributes to the exposure of training data. We implemented a white-box membership inf… view at source ↗

**Figure 2.** CV of Rank The MIA setup is as follows. The batch size used during the attack model training was set to 64, and the Adam optimizer was employed with a learning rate of 0.0001. The attack model was trained for 100 epochs, with the best model selected based on its testing accuracy. The primary evaluation metric for the attack was the membership inference accuracy, which measures the attack model’s ability to determine… view at source ↗

**Figure 3.** MCR of DoF and Rank during this process. Layers closer to the output exhibit smaller reductions in DoF CV and larger reductions in Rank CV after reaching their peaks. These reductions are strongly correlated with the success rates of membership inference attacks, where layers with smaller reductions in DoF CV or larger reductions in Rank CV are more vulnerable. Additionally, MCR values generally increase during the … view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: MCR of Rank Our results suggest that both CV and MCR are effective indicators of privacy leakage risks. While Rank-based metrics provide more pronounced trends, they come with higher computational costs due to the need for additional differentiation steps. In contrast,…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Real-Time Privacy Risk Measurement with Privacy Tokens for Gradient Leakage
cs.LG 2025-02 reject novelty 4.0 of 10

The paper introduces privacy tokens, learned gradient embeddings, to estimate mutual information between training data and gradients for real-time privacy risk monitoring during model training.

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages · cited by 1 Pith paper

[1]

Deep learning with differential privacy

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318,

2016
[6]

Privacy in deep learning: A survey

Fatemehsadat Mireshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. Privacy in deep learning: A survey. arXiv preprint arXiv:2004.12254,

arXiv 2004
[9]

Sok: Security and privacy in machine learning

Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P Wellman. Sok: Security and privacy in machine learning. In 2018 IEEE European symposium on security and privacy (EuroS&P), pages 399–414. IEEE,

work page 2018
[10]

Membership inference attacks against machine learning models

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE,

2017
[13]

Enhanced membership inference attacks against machine learning models

Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093–3106,

work page 2022
[14]

Privacy risk in machine learning: Analyzing the connection to overfitting

16 Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE,

work page 2018
[1969]

Demystifying membership inference attacks in machine learning as a service

Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Demystifying membership inference attacks in machine learning as a service. IEEE transactions on services computing , 14(6):2073–2089,

work page
[1975]

Membership inference attacks from first principles

Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE,

2022

Show all 15 references

[2010]

Distillation as a defense to adversarial perturbations against deep neural networks

Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582–597. IEEE,

2016
[2015]

Deep learning for computer vision: A brief review

Athanasios V oulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018(1):7068349,

2018
[2016]

Towards better understanding of gradient-based attribution methods for deep neural networks

Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104,

arXiv
[2018]

Optimality of the johnson-lindenstrauss lemma

Kasper Green Larsen and Jelani Nelson. Optimality of the johnson-lindenstrauss lemma. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 633–638. IEEE,

2017
[2020]

Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning

Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pages 739–753. IEEE,

2019
[2021]

A survey on gradient inversion: Attacks, defenses and future directions

Rui Zhang, Song Guo, Junxiao Wang, Xin Xie, and Dacheng Tao. A survey on gradient inversion: Attacks, defenses and future directions. arXiv preprint arXiv:2206.07284,

arXiv
[2022]

Robust learning with jacobian regularization.arXiv preprint arXiv:1908.02729, 5(6):7,

Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization.arXiv preprint arXiv:1908.02729, 5(6):7,

1908 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Deep learning with differential privacy

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318,

2016

[2] [6]

Privacy in deep learning: A survey

Fatemehsadat Mireshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. Privacy in deep learning: A survey. arXiv preprint arXiv:2004.12254,

arXiv 2004

[3] [9]

Sok: Security and privacy in machine learning

Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P Wellman. Sok: Security and privacy in machine learning. In 2018 IEEE European symposium on security and privacy (EuroS&P), pages 399–414. IEEE,

work page 2018

[4] [10]

Membership inference attacks against machine learning models

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE,

2017

[5] [13]

Enhanced membership inference attacks against machine learning models

Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093–3106,

work page 2022

[6] [14]

Privacy risk in machine learning: Analyzing the connection to overfitting

16 Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE,

work page 2018

[7] [1969]

Demystifying membership inference attacks in machine learning as a service

Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Demystifying membership inference attacks in machine learning as a service. IEEE transactions on services computing , 14(6):2073–2089,

work page

[8] [1975]

Membership inference attacks from first principles

Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE,

2022