{"id":"e6587486-128a-439c-8a0f-f4695e558c22","arxiv_id":"2412.00696","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A deep network's intermediate layers can be ranked by privacy risk using estimated degrees of freedom and Jacobian rank, which the authors link to membership inference attack success.","lead":"This paper proposes a way to estimate privacy risk in each layer of a vision model by measuring how many independent directions the layer's outputs use (degrees of freedom) and how strongly those outputs respond to input changes (Jacobian rank). It claims these two numbers can flag which layers leak the most training-data information, without running expensive attack simulations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2 sums per-sample input gradients over the batch before forming the Gram matrix, so the reported 'Jacobian rank' is not the rank of any single input's Jacobian; the framework's sensitivity pillar is not measuring what it claims.","rationale":"I read the paper in good faith: it proposes two cheap metrics (DoF and Jacobian rank) to rank intermediate-layer privacy risk without attack simulations. The DoF estimator is a reasonable intrinsic-dimension proxy. The fatal soft spot is Algorithm 2. The text's own implementation procedure says the per-sample gradients are summed across the batch before the Gram matrix is formed, so the matrix whose rank is computed is not the Jacobian of any single input. The paper's central claim depends on the second metric measuring input sensitivity; if it does not, the framework's explanatory story is invalid. This is an internal inconsistency, not a disagreement with prior work. I also considered the alternative concern that Table 2 contains a counterexample (AlexNet Conv2d_4 has higher MCR(DoF) than Conv2d_5 but lower attack accuracy), which also undermines the empirical claim. But the batch-sum flaw is more load-bearing because it invalidates the metric's definition, whereas the counterexample only weakens the correlation. A single concrete check—recomputing ranks per-sample—would settle which effect drives the reported trends. The reader's weakest assumption identifies the same Algorithm 2 issue, so I agree with the reader's analysis. My stress test leaves the reader's REJECT verdict unchanged.","tokens_in":12513,"tokens_out":6118,"duration_ms":54539,"concrete_test":"Recompute the Jacobian-rank metric without batch summation: for each sample i in the batch, form the k×k Gram matrix G_i^{(l)} = (U_i^{(l)})^T U_i^{(l)} from the per-sample gradients u_{i,j}^{(l)} = ∇_x ⟨h_i^{(l)}, v_j^{(l)}⟩ (no summation over i), and define the rank via the same 95% eigenvalue threshold. Then compute CV and MCR from the per-sample ranks, averaged over the batch, and re-plot Figures 2 and 4 and re-build Table 2. If the per-sample rank trends lose the reported U-shape or the MCR-attack-accuracy correlation, the paper's results are an artifact of batch aggregation. If they reproduce, the batch summation is not the decisive flaw.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that CV and MCR computed from DoF and Jacobian rank indicate privacy leakage, and that higher MCR/specific CV trends mark layers more susceptible to membership inference attacks. The DoF estimator (Algorithm 1) is a standard PCA-based intrinsic-dimension estimate and is defensible. The second pillar, however, is not. In Section 3.3, the parallel computation procedure for Algorithm 2 explicitly states (step 6) that 'to obtain the final matrix U(l), the gradients are summed over the batch dimension, resulting in a matrix of shape [CHW, k].' This batch-summed U(l) is then used to form the Gram matrix G(l) = (U(l))^T U(l), whose eigenvalue threshold yields the claimed Jacobian rank. The rank of this Gram matrix is the rank of the sum of per-sample Jacobian-vector products, which is not the Jacobian rank of any individual input. It can be larger than or smaller than the per-sample ranks and depends on batch composition and gradient signs. Consequently, the metric labeled 'sensitivity to input variations' is not a per-sample sensitivity measure, and the paper's dual-analysis rationale collapses: the observed MIA correlations cannot be attributed to input-output sensitivity of intermediate representations. This is an internal methodological flaw, not a matter of disagreement with prior consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for assessing the privacy risk of intermediate layer outputs in deep vision models without running attack simulations. It defines two metrics: the Degrees of Freedom (DoF) of layer activations, estimated via a random-projection PCA, and the rank of an approximated Jacobian of the layer output with respect to the input. From these, the paper derives Change Value (CV) and Modified Change Ratio (MCR) time series over training and claims that these correlate with membership inference attack accuracy across layers of CNN (MNIST), LeNet (CIFAR-10/100), and AlexNet (CIFAR-100). The experimental section reports four small tables and several figures supporting a qualitative pattern, but the paper supplies no statistical tests, error bars, or out-of-sample validation for the claimed relationship.","tokens_in":12824,"tokens_out":6348,"duration_ms":54431,"significance":"The underlying idea is potentially useful: if DoF and Jacobian-based statistics could rank the privacy risk of intermediate layers cheaply, that would be a practical complement to expensive attack simulations. The DoF estimator is a standard intrinsic-dimensionality technique, and the paper's framing of information retention via DoF is coherent. However, the paper's central claim is currently supported only by visual inspection of a handful of tables, and the Jacobian rank estimator as implemented has a serious methodological flaw that undermines the sensitivity pillar. If the technical issues were corrected and the analysis were given proper statistical grounding, the approach could be a worthwhile contribution, but in its present form the evidence does not establish the claimed correlation.","major_comments":[{"comment":"The Jacobian rank estimator does not compute the rank of any individual input's Jacobian. In the parallel computation procedure, the per-sample gradient columns are summed over the batch dimension to form U(l) of shape [CHW, k], and the Gram matrix G(l) is then built from this batch-summed U(l). The rank of the Gram matrix is the rank of a sum of per-sample Jacobian-vector products, which can be larger or smaller than the rank of each per-sample Jacobian and depends on gradient sign cancellations and batch composition. Therefore the metric labeled \"sensitivity to input variations\" is not a per-sample sensitivity measure, and the paper's dual-analysis rationale breaks down.","section":"Section 3.3, Algorithm 2 and parallel computation description"},{"comment":"The global claim that \"layers with higher MCR values and specific CV trends are more susceptible to membership inference attacks\" is contradicted by the CNN(MNIST) row in Table 2: this row has the lowest MCR values in the table (MCR(DoF)=27.27%, MCR(Rank)=0.82%) yet the highest attack accuracy (78.31%). The paper does not explicitly restrict its claim to within-model comparisons, and even if it did, no statistical measure is provided to support the correlation for any model. No correlation coefficients, p-values, confidence intervals, or error bars appear anywhere in the experimental section.","section":"Section 5, Table 2"},{"comment":"The CV and MCR metrics are defined in Eq. (1)-(2) only after the authors observe the training curves, so the claimed relationship is post hoc rather than a tested prediction. The paper provides no out-of-sample check, no split-half validation, and no comparison against a null model where the metrics are randomly assigned to layers. The paper itself acknowledges in Section 6 that \"developing formal guarantees on the linkage between these metrics and attack success rates would strengthen the reliability of our framework,\" which is an admission that the central claim currently rests on observed pattern matching rather than on a validated indicator.","section":"Section 5, Eqs. (1)-(2)"},{"comment":"No sensitivity analysis is reported for the free parameters of the framework: the variance threshold tau (set to 0.95), the Gaussian projection dimension factor r_l = 0.1 * k_l, and the number of random vectors k = 0.1 * k_l. Since the DoF and Jacobian rank estimates are defined through these thresholds and projection sizes, the robustness of the claimed correlation to these choices is unknown, and the paper does not demonstrate that the reported trends would persist under reasonable parameter variations.","section":"Section 4.2"}],"minor_comments":[{"comment":"\"V oulodimos\" should be \"Voulodimos\".","section":"Introduction, first paragraph"},{"comment":"\"intermedia layers\" should be \"intermediate layers\", and \"Jocabian matrix\" should be \"Jacobian matrix\".","section":"Algorithm 1 caption and Algorithm 2 output"},{"comment":"\"there has been limited into the application of DoF\" is an incomplete sentence; it should read \"there has been limited research into the application of DoF\".","section":"Section 2.2"},{"comment":"Eqs. (1) and (2) are referenced as \"Eq.(1)\" and \"Eq.(2)\" but are not numbered in the text; please add equation numbers.","section":"Section 5"},{"comment":"The MIA description does not specify whether attack accuracy is computed as balanced accuracy, the average over multiple attack model seeds, or a single run; please clarify the evaluation protocol.","section":"Section 4.3"},{"comment":"The figures are not discussed in sufficient detail in the text; specific panels should be pointed to when making claims about CV and MCR trends, and the figures should be made readable at publication size.","section":"Figures 1-4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper has a promising high-level concept, but the batch-summing issue in Algorithm 2 is a fundamental flaw in the sensitivity metric as currently defined, and the experimental validation is too weak to support the central claim as stated. The authors would need to recompute the Jacobian rank per input (or provide a clear justification for the batch-summed quantity), add proper statistical testing, and address the counterexample in Table 2. This is substantial work but it is within the scope of a major revision rather than being beyond repair. The manuscript may also be better suited for a venue that accepts more exploratory empirical analyses, or for a longer version with more rigorous evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about cheap proxies for privacy leakage, but not ready for prime time. The core idea—use layer-wise intrinsic dimensionality (DoF) and input-output sensitivity to classify which intermediate layers are more vulnerable to membership inference—is a legitimate new application of known concepts, and the DoF estimator via PCA on random projections is standard and defensible. The experiments span several models and datasets and the qualitative direction (later layers leak more) is plausible.\n\nThe problems are real. Most seriously, Algorithm 2 does not compute the rank of the Jacobian of any single input. It sums per-sample gradients over the batch before forming the Gram matrix, so the resulting object is the rank of the sum of Jacobians across the batch. That's a different quantity, and the paper never explains why it should behave like a per-sample sensitivity measure. The sensitivity rationale for the framework collapses until this is fixed or redefined.\n\nThen there's the statistical validation. The CV and MCR metrics are defined after looking at the training curves, so the claimed relationship is post hoc. The quantitative support is four small tables with no error bars, no correlation coefficients, no out-of-sample checks. Table 2 has a non-monotonic pattern (AlexNet Conv2d_4 vs Conv2d_5) that the paper doesn't address. No code is released, so the numbers are hard to verify independently.\n\nThe references look fine, and the writing is clear. The idea is worth discussing in a reading group or as a workshop paper, but as a full submission it needs major revision. I'd send it to review if I were an editor—the topic is important and the authors are onto something—but I'd expect the referees to demand a correct Jacobian-rank estimator and real statistical grounding. In current form, I would not accept or cite it.","headline":"A promising attack-free privacy-risk heuristic, but the Jacobian-rank estimator is mislabeled and the evidence is too thin to support the central claim.","tokens_in":13289,"tokens_out":5269,"would_cite":false,"duration_ms":45833,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the privacy risk of a model's intermediate layers can be ranked by two cheap metrics—layer-output Degrees of Freedom and Jacobian rank—without running any attack simulation.","keywords":["privacy risk assessment","membership inference attacks","intermediate representations","Degrees of Freedom","Jacobian rank","computer vision","training-time monitoring","sensitivity analysis"],"falsifier":"Take a trained model, compute each layer's CV and MCR, then also compute the exact per-sample Jacobian rank for the same layers; if the layer ordering by Algorithm 2's batch-summed rank differs materially from the exact per-sample ordering, and the exact ordering—not the paper's—predicts membership-inference accuracy, then the proposed metric is not measuring the sensitivity it claims to measure.","tokens_in":12330,"feed_emoji":"🔒","tokens_out":7639,"duration_ms":67684,"temperature":0.7,"pith_summary":"The paper claims that the privacy risk of a deep vision model's intermediate layers can be read off from two quantities computed during training: the Degrees of Freedom (DoF) of each layer's outputs and the rank of that layer's Jacobian with respect to the input. Across CNN, LeNet, and AlexNet on MNIST and CIFAR, both metrics first fall and then rise, and layers whose DoF and rank recover most strongly are the ones that membership-inference attacks exploit most successfully. If this holds, a model developer can monitor layer sensitivity in real time and rank leakiness without running expensive attack simulations. The payoff is a cheap, attack-free privacy audit for intermediate representations, the part of a model that standard output-level defenses ignore.","feed_headline":"Privacy-risky layers found without an attack simulation","feed_subtitle":"Two training-time statistics, DoF and Jacobian rank, flag which hidden layers leak membership information.","key_machinery":"The machinery is a dual estimator. DoF is estimated by taking a layer's batched outputs, centralizing them, projecting through a random Gaussian matrix, and counting how many principal components of the resulting covariance matrix explain 95% of the variance. Jacobian rank is approximated by projecting a layer's output onto k random directions, differentiating those projections with respect to the input, assembling the gradient vectors into a Gram matrix, and again thresholding its eigenvalue spectrum at 95% variance. Both estimators are designed to be cheap enough to run during training and to convert a high-dimensional privacy question into a one-dimensional sensitivity ranking per layer.","core_discovery":"On its own terms, the paper's discovery is that information-retention and sensitivity dynamics during training trace a layer's privacy exposure. The authors define CV (change from the initial value) and MCR (relative recovery from the training minimum) for both DoF and Jacobian rank, and report that attack accuracy rises for layers with larger MCR and with the CV pattern of a small DoF drop and a large rank drop after the peak. They interpret the early dip as an information-bottleneck phase and the later rise as the model storing more input-specific detail, so late-training intermediate layers are the most vulnerable. The claim is that these two cheap metrics can classify privacy sensitivity at the layer level, with Rank-based metrics more accurate but costlier, and DoF-based metrics a fast approximation.","pith_inferences":["Beyond the paper, the dip-and-rise pattern could be used as a training-stopping or defense-trigger signal: if late-training layers are the leaky ones, early stopping or layer-specific noise could cut MIA success at the epochs the metrics flag.","Beyond the paper, the same metrics might transfer to transformer and federated settings, but the batch-summed gradient step in the rank estimator would need to be replaced with a per-sample or minibatch-accurate Jacobian before the ranking can be trusted there.","Beyond the paper, a testable extension is to obfuscate only the highest-MCR layers and measure whether attack accuracy drops almost as much as when all layers are obfuscated; that would turn the correlation into a usable defense."],"forward_implications":["Privacy risk can be monitored live during training, so a developer can identify both the vulnerable layers and the training epochs at which they become vulnerable.","Layers closer to the output are consistently more attack-vulnerable, matching the observed higher MCR and stronger rank recovery.","DoF-based monitoring can serve as a low-cost screening tool, while Rank-based monitoring provides a more accurate but more expensive assessment.","The metrics can guide where to apply privacy protections, such as which intermediate representations to mask or perturb to reduce membership-inference success."],"supporting_citations":[{"why":"Provides the white-box membership-inference setup and layer-vulnerability analysis used to validate the paper's metrics.","marker":"Nasr et al. [2019]"},{"why":"Defines the membership-inference attack framework whose success rates ground the paper's risk labels.","marker":"Shokri et al. [2017]"},{"why":"Connects overfitting to privacy risk, the premise that links low or volatile DoF to leakage.","marker":"Yeom et al. [2018]"},{"why":"Supplies the use of Degrees of Freedom for model generalization that the paper adapts to privacy assessment.","marker":"Zhang et al. [2021]"},{"why":"Justifies the random Gaussian projection in DoF estimation via the Johnson-Lindenstrauss lemma.","marker":"Frankl and Maehara [1988]"},{"why":"Supports the efficiency claim of the same random projection.","marker":"Larsen and Nelson [2017]"},{"why":"Represents the differential-privacy baseline that the paper positions against as output-focused.","marker":"Dwork [2006, 2008]"},{"why":"Documents representation-level privacy leakage that motivates auditing intermediate layers.","marker":"Sun et al. [2021]"}],"fun_headline_variants":["No attack simulation needed to find leaking layers","DoF and Jacobian rank expose hidden-layer leaks","Late-training layers store more input detail","Two cheap stats reveal which layers leak privacy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the batch-level rank computed by summing per-sample gradients still measures each input's sensitivity, even though the sum can cancel individual directions and therefore may not describe any single image.","fun_headline_variants_meta":{"raw":{"variants":["No attack simulation needed to find leaking layers","DoF and Jacobian rank expose hidden-layer leaks","Late-training layers store more input detail","Two cheap stats reveal which layers leak privacy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000482,"raw_usage":{"total_tokens":2336,"prompt_tokens":850,"completion_tokens":1486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":1439}},"tokens_in":466,"tokens_out":1486,"duration_ms":10578,"temperature":1.0,"reasoning_tokens":1439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:05:41.558213+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained model, compute each layer's CV and MCR, then also compute the exact per-sample Jacobian rank for the same layers; if the layer ordering by Algorithm 2's batch-summed rank differs materially from the exact per-sample ordering, and the exact ordering—not the paper's—predicts membership-inference accuracy, then the proposed metric is not measuring the sensitivity it claims to measure.","supporting_citations":[{"cited_title":"Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning","cited_arxiv_id":null,"evidence_quote":"Provides the white-box membership-inference setup and layer-vulnerability analysis used to validate the paper's metrics."},{"cited_title":"Privacy risk in machine learning: Analyzing the connection to overfitting","cited_arxiv_id":null,"evidence_quote":"Connects overfitting to privacy risk, the premise that links low or volatile DoF to leakage."},{"cited_title":"Optimality of the johnson-lindenstrauss lemma","cited_arxiv_id":null,"evidence_quote":"Supports the efficiency claim of the same random projection."}],"review_version":1}