Pith. sign in

REVIEW 4 major objections 4 minor 25 references

Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A near-OOD benchmark that trains on its 'OOD' class can invert AUROC from 0.326 to 0.911 when corrected.

desk verdict A well-supported leak correction and an honest negative result, but the fingerprint's representation-dependence is load-bearing and needs pinning down before I'd trust it in the wild. read the letter →

arxiv 2607.19393 v1 pith:JO55HTX4 submitted 2026-07-02 cs.LG cs.AIcs.CLcs.CV

Vishnu Bindu Balachandran This is my paper
classification cs.LGcs.AIcs.CLcs.CV
keywords out-of-distributiondetectionnear-OODevaluationbenchmarkcontaminationdataleakagemodelauditingperturbation-responseleave-class-outdecodability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a common way of building hard near-OOD benchmarks—designating a class the model has already been trained on as 'OOD'—silently contaminates the in-distribution fit set and can flip reported AUROC from 0.326 (below chance) to 0.911 once the class is removed by leave-class-out retraining. It distills the contamination into a retraining-free fingerprint: near-perfect supervised decodability (OracleLDA ≥ 0.95) together with unsupervised detection collapsed below 0.65. The fingerprint is validated on 52 controlled settings across four fine-tuned backbones (sensitivity 18/20, specificity 31/32) and on an in-the-wild audit of 24 standard OOD pairs, where it fires on only the intrinsically hard CIFAR-100→CIFAR-10 pair. A second result: after correcting the leak, perturbation-response OOD signals are 'decodable but not detectable'—a supervised linear reader recovers them (AUROC 0.87–1.00) while no unsupervised detector closes the gap, and the perturbation method does not beat plain Mahalanobis distance.

What carries the argument

The leak fingerprint: fired = (OracleLDA ≥ 0.95) ∧ (best-unsupervised AUROC < 0.65), where OracleLDA is a supervised linear (Fisher/LDA) probe measuring how perfectly a label-aware reader can separate the designated OOD class from the ID fit set, and best-unsupervised is the maximum of Mahalanobis and kNN AUROC on ID-only fit. The diagnostic works without backbone retraining by contrasting a supervised reader (which sees labels) against unsupervised detectors (which see only the ID fit set): a trained class is perfectly linearly separable from ID yet inside the one-class fit, so this combination fires; a genuinely novel class is imperfectly decodable and stays below the oracle gate. The leav

What would settle it

Run the paper's fingerprint on a deliberately leaked split using a model trained with a contrastive objective whose penultimate embeddings are not linearly class-separable; if the fingerprint fails to fire on known leakage or fires on clean far-OOD pairs, the representation-dependent premise is broken. Concretely, on a new architecture, construct the 20 leak / 32 clean controlled settings exactly as the paper does; if sensitivity is below about 0.8 or specificity below about 0.95, the thresholds or the representation are not transferable.

Watch

Extended reading notes

Core claim

The paper's central claim is that a near-OOD evaluation split becomes measurably contaminated, not merely conservative, when the designated 'OOD' class is one the model was trained on: its training examples sit in the ID fit set, so a one-class detector ranks them as familiar and is penalized for it, manufacturing below-chance scores (0.326) that vanish under a leave-class-out correction (0.911 on the same class, 35 models, two domains). The contamination has a tell that costs no retraining: a supervised Fisher reader almost perfectly separates the leaked class (OracleLDA ≈ 1) while every unsupervised detector collapses (best-unsup < 0.65). The paper validates this 'leak fingerprint' on 52 d

Load-bearing premise

The fingerprint's reliability hinges on penultimate-embedding space being a place where a trained class is unsupervisedly separable while a genuinely novel near-OOD class is not, with the specific thresholds 0.95 and 0.65 transferring; if another architecture or dataset has different feature geometry, the test will over- or under-fire.

Editorial extensions

If this is right

  • Benchmark auditors can cheaply flag a suspect near-OOD split using cached features and one LDA probe, instead of k full re-trainings.
  • Standard cross-dataset OOD benchmarks (CIFAR, SVHN, DTD, etc.) are clean of this specific fit-set leak; only the intrinsically hard CIFAR-100→CIFAR-10 pair trips the fingerprint, and it is a documented threshold-scope false positive, not a leak.
  • Any near-OOD result below chance should be treated as a possible benchmark construction error, not a method failure, until the fit-set membership of the 'OOD' class is checked.
  • Perturbation-response detectors (attention-mask displacement signatures) do not beat plain embedding distance under clean evaluation; the gap is explained by whitening misallocating budget to near-singular directions.
  • The corrected leave-class-out protocol restores normal AUROC (0.911 vs 0.326) across all unsupervised detectors, in documents and text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decodable-but-not-detectable gap likely extends beyond perturbation signatures: any near-OOD signal that is a mean shift in feature space but not aligned with low-variance directions will be invisible to ID-only scorers, so the same audit logic could apply to other proposed OOD statistics.
  • Because the fingerprint relies on penultimate-embedding geometry, models whose embeddings are not class-clustered (contrastive vs. softmax trained, or foundation models with different feature anisotropy) may need recalibrated thresholds; a systematic threshold-transfer study is a natural next test.
  • The paper's negative result on perturbation detection suggests that head-masking sensitivity is not a free lunch; future work on perturbation-based OOD should establish that the added complexity buys separation beyond Euclidean distance before proposing new variants.
  • The in-paper retraction of the circular severity correlation is itself a methodological point: independent re-measurement (here, MMD in a non-detector encoder) is what separates a real effect from a self-confirmatory metric, and the same discipline could audit other claimed relationships in OOD literature.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper documents and analyzes a specific benchmark-construction leak in near-OOD evaluation: a designated OOD class that was actually present in the model's training set and in the one-class fit set. On a Tobacco document split this produced an AUROC of 0.326, below chance; deleting the class and retraining leave-class-out models raises the same evaluation to 0.911. The paper proposes a retraining-free 'leak fingerprint' — near-perfect supervised decodability (OracleLDA ≥ 0.95) coupled with collapsed unsupervised detection (best-unsup < 0.65) — and validates it on a controlled battery of 52 settings across ResNet-50 and ViT-B/16 on CIFAR-10/100, reporting sensitivity 18/20 and specificity 31/32 in penultimate-embedding space, with perfect matched fit-set-exclusion controls. It also audits 24 standard OOD pairs, finding only the known hard CIFAR-100→CIFAR-10 pair fires, and re-evaluates the perturbation detector under the corrected protocol, concluding that perturbation signals are 'decodable but not detectable' and do not beat a plain Mahalanobis baseline. The paper retracts an earlier circular severity correlation.

Significance. If the fingerprint is reliable, the paper provides a practical, low-cost audit tool for a subtle but damaging form of benchmark contamination, and the corrected protocol (0.326→0.911 with 35 retrained models across two domains) is a clean, reproducible demonstration. The paper is unusually transparent: thresholds are fixed a priori from the real leak, per-setting tables are given, the matched fit-set-exclusion design is a strong confound control, the earlier circular correlation is retracted in-paper, and code/checkpoints are promised. The theoretical account of why an ID-only detector cannot use the discriminating direction and why whitening can hurt is a useful framing. The main risk is that the central diagnostic is representation- and threshold-contingent exactly on the motivating real leak; this is acknowledged but not fully resolved.

major comments (4)
  1. [§5.1, Appendix E, Table 15] The recommended embedding-space fingerprint is not reliable on the paper's motivating real leak. Tobacco/Memo has OracleLDA=0.997, best-unsup=0.639, and fires on 2/5 seeds in embedding space (Table 15); it is decisive only in the 77-d signature (best-unsup 0.33), where the cheap clean-excl control false-fires 3/5 (Appendix E, Table 13). Appendix E then recommends embedding space, but this is exactly where the real leak is marginal. A benchmark auditor following the recommended protocol would miss the motivating case on 3/5 seeds; using the sharp signature would flag clean controls. Since the fingerprint's first contribution is a retraining-free detector of fit-set leakage, this representation-dependence is load-bearing. Please either report a decision rule that works on Tobacco/Memo with acceptable per-seed reliability, or restrict the claim to 'validated on synthetic fit-set leaks; on t
  2. [§3, §4.4] There is a contradiction in the definition of OracleLDA. Section 3 says 'No statistic is ever fit on OOD data (enforced by a unit test)' while the diagnostic OracleLDA is a supervised linear probe fit on labeled ID/OOD examples (§4.4, §5.1: 'shrinkage-LDA supervised probe on a held-out split'). The audit therefore requires OOD labels and fits a classifier on them. Please state explicitly that the 'no fit on OOD' guarantee covers the unsupervised detectors only, and that the fingerprint's supervised conjunct is fit on the benchmark's OOD split. This matters for reproducibility and for understanding what an auditor must have in hand.
  3. [§5.2, Table 13, Appendix H] The 'decisive evidence' for the fingerprint is the matched fit-set-exclusion contrast, but it is validated only on the CIFAR battery (20/20, embedding space). For the real Tobacco/Memo leak, no embedding-space clean-excl result is presented; the correction shown is leave-class-out retraining. In the 77-d signature, the same clean-excl control false-fires on 3/5 settings (Table 13). The paper therefore does not demonstrate that the matched contrast 'isolates fit-set membership as the cause' in the deployed setting; it isolates it on synthetic group/class re-designations of four backbones. Please add the embedding-space clean-excl result for Tobacco/Memo, or qualify the claim.
  4. [§6, Appendix F, Appendix H] The in-the-wild audit tests only clean standard pairs (24 vision pairs plus 11 clean text/document settings), so it establishes specificity but not sensitivity on real leaked benchmarks. The only real leak included (Tobacco/Memo, Table 15) is borderline in the recommended representation and per-seed unstable, which further limits the evidence for field sensitivity. The paper should state this limitation explicitly in the abstract or conclusions; as written, readers may infer that the diagnostic has been shown to catch real leaks at the claimed reliability.
minor comments (4)
  1. [Abstract and §7] The abstract says 'no unsupervised detector does' while Section 7 correctly says 'no unsupervised detector we tried'. Please use the qualifier consistently, since the claim is necessarily over a finite method set.
  2. [Figures 2 and 3] The horizontal axes are cropped to OracleLDA≥0.92/0.94, which visually compresses the far-OOD points near 1.0. The text explains this, but the captions should state the crop explicitly.
  3. [Appendix E] The 77-d spot-check is run on a single backbone (ViT-B/16, CIFAR-10) with five leaked classes. The statement that 'the fingerprint detects leaks in either space' is therefore based on a small spot-check; please label it as such in the main text when referring to 'either space'.
  4. [Table 12] The per-setting table is very useful, but consider adding a column that explicitly labels the two false negatives and one false positive in the table itself, rather than only in the surrounding text, to make the error analysis easier to verify.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity in the active derivation; the one explicitly circular step is retracted, and the fingerprint thresholds are fixed a priori with matched and in-the-wild controls.

  1. self definitional [Section 8 (retraction); Table 11]
    "An earlier version of this work reported that Γ grows as OOD nears (Pearson −0.79). We retract it. The severity axis we used was itself an unsupervised detectability score, making the correlation circular by construction; recomputed against a detection-independent axis (squared MMD in a frozen non-detector encoder) it collapses to Pearson −0.035."

    Γ = OracleLDA − max(unsup) and the 'severity' axis was max(unsup), so the two quantities share the unsupervised term; a high gap is mechanically associated with a low unsupervised score, producing a spurious negative correlation. The paper explicitly retracts this claim and shows the correlation disappears on an independent MMD-based axis. Because the claim is retracted, it is not load-bearing for any current result.

full rationale

The paper's central empirical result — the same Memo class scoring 0.326 under the leaked construction and 0.911 under leave-class-out retraining — is computed in-paper, replicated across 35 fine-tunes in two domains, and does not depend on a fitted parameter being renamed as a prediction. The leak fingerprint uses thresholds (OracleLDA ≥ 0.95, best-unsup < 0.65) that the paper states were fixed a priori from the single real Tobacco leak, then validated on a controlled battery and an in-the-wild audit. The battery's leak/clean labels are defined by fit-set membership, which is a controlled validation rather than a circular derivation: the fingerprint could have failed (it misses 2/20 leaks and false-fires once), and the matched clean-excl contrast plus the 24-pair audit provide independent checks. Self-citations (Balachandran, 2026) are used for baseline reconciliation and protocol inheritance, but the key anomaly numbers and the leave-class-out correction are independently established in this paper; no uniqueness theorem or ansatz is imported from self-citation. The only explicitly circular step in the manuscript is the retracted Γ–severity correlation, which the authors identify, retract, and replace with an independent MMD-based measurement. The skeptic concern about representation-dependence (embedding-space borderline on the real leak, signature-space false-fires on clean-excl controls) is a robustness/correctness issue, not a circularity issue under the rubric. Overall, the active derivation chain is self-contained and transparent; no current result reduces to its own input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper rests on a small set of hand-set diagnostic thresholds, a simplifying Gaussian model in the theoretical account, and a representation choice. No invented physical or formal entities are introduced. The validation battery is carefully constructed, but settings are not independent draws within backbones, so the pooled sensitivity/specificity numbers overstate the effective sample size.

free parameters (2)
  • Fingerprint thresholds (OracleLDA gate, best-unsup cutoff) = 0.95, 0.65
    Hand-set from the single Tobacco leak before the validation battery. The paper shows the decision is stable across a band of cutoffs, but the operating point is a free choice that determines sensitivity/specificity.
  • best-unsup definition (max of Mahalanobis and kNN)
    Operationalizing the 'best unsupervised detector' as the maximum of two methods is a choice. Other unsupervised detectors (OC-SVM, Isolation Forest, LOF) are reported separately but not folded into the fingerprint rule, which affects the rule's specificity.
assumptions (4)
  • domain assumption ID and OOD signatures share covariance Sigma (Gaussian model)
    Used in Proposition 2 and Observation 3 to explain the decodability gap. Real embeddings are not exactly Gaussian with shared covariance, so the mechanism is a stylized account rather than a proof about real data.
  • domain assumption A trained class is near-perfectly linearly separable from the rest of ID in penultimate-embedding space
    Core premise of the fingerprint. Verified empirically on the leaked settings (OracleLDA ≈ 0.96–1.00), but assumed to generalize to other models and datasets.
  • domain assumption Removing a class from the one-class fit set without retraining ('clean-excl') is a valid negative control for the leak
    Used in Section 5 to isolate fit-set membership. The paper itself notes this is not a true leave-class-out, and in the 77-d perturbation signature this contrast loses specificity.
  • domain assumption Unsupervised detector statistics are fit only on ID-train data
    Stated as enforced by a unit test, but not independently verifiable without the code. If violated, the reported AUROC values would be unreliable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks." pith.science (2026). https://pith.science/paper/JO55HTX4

@misc{pith2026260719393,
  author       = {Pith},
  title        = {Pith review of: Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JO55HTX4}},
  note         = {Machine review of arXiv:2607.19393}
}
read the original abstract

While auditing a perturbation-based OOD detector on a document benchmark, we recorded an AUROC of 0.326 -- well below the 0.5 chance level. The cause is a benchmark leak: the designated "OOD" class is one the model was trained on, so its examples sit inside the in-distribution fit set and the detector is penalized for correctly ranking them as familiar. Deleting the class and retraining 35 models across two domains raises the score to 0.911. We distill the contamination into a leak fingerprint -- near-perfect supervised decodability (AUROC approximately 1) coupled with unsupervised detection collapsed below 0.65 -- and validate it on a controlled battery of 52 settings (20 leaked, 32 clean) across ResNet-50 and ViT-B/16 on CIFAR-10/100, achieving sensitivity 18/20 and specificity 31/32 in embedding space; the matched fit-set-exclusion controls are perfect at 20/20. An in-the-wild audit of 24 standard near/far OOD benchmark pairs fires on exactly one (the intrinsically hard CIFAR-100 vs CIFAR-10 pair) and on no far-OOD pair, confirming specificity and that standard cross-dataset construction is clean. Under the corrected protocol, perturbation signals are decodable but not detectable: a supervised reader recovers the OOD signal (AUROC 0.87-1.00) while no unsupervised detector does, and the perturbation method does not improve on plain Mahalanobis distance. We provide a theoretical account of why and, for transparency, retract an earlier circular correlation. The contributions are a corrected protocol and a validated leak diagnostic, not a new OOD method.

Figures

Figures reproduced from arXiv: 2607.19393 by the authors.

Figure 1
Figure 1. Why the leak inverts the score. Left: when a semantically similar trained class (Memo) is designated “OOD”, its examples lie inside the one-class ID fit set, so an unsupervised detector correctly ranks them as in-distribution and is penalized, scoring below chance (0.326). Right: removing the class from training (leave￾class-out) places a genuinely unseen class at the edge of the fit set (near-OOD: semantically clos… view at source ↗
Figure 2
Figure 2. Controlled validation of the leak fingerprint (52 settings, full 2 × 2). All settings have high supervised decodability (OracleLDA ≥ 0.94), so the horizontal axis is cropped to the populated range. Leaked settings (orange circles) fall in the fingerprint box (high decodability with unsupervised detection collapsed below 0.65), while both families of clean controls (fit-set exclusion, blue squares; novel dataset, gre… view at source ↗
Figure 3
Figure 3. In-the-wild audit: 24 standard near/far OOD benchmark pairs across four back￾bones, scored by the fingerprint in embedding space (the horizontal axis is cropped to the populated range, OracleLDA ≥ 0.94; the 20 far-OOD and near-STL pairs at OracleLDA ≈ 1 overlap near x=1). Every far-OOD pair (and every semantically￾near pair but one) sits outside the fingerprint box. The single fire is the intrinsically￾hard CIFAR-10… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Decodable but not detectable. For each setting, the supervised reader (OracleLDA, solid blue) recovers the OOD signal nearly perfectly, while the best unsupervised detector (hatched orange) lags; the gap Γ between them is signal that is present but unreadable without l…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 4 linked inside Pith

  1. [1]

    International Conference on Learning Representations (ICLR) , year =

    Dan Hendrycks and Kevin Gimpel , title =. International Conference on Learning Representations (ICLR) , year =

  2. [2]

    Srikant , title =

    Shiyu Liang and Yixuan Li and R. Srikant , title =. International Conference on Learning Representations (ICLR) , year =

  3. [3]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Kimin Lee and Kibok Lee and Honglak Lee and Jinwoo Shin , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  4. [4]

    Owens and Yixuan Li , title =

    Weitang Liu and Xiaoyun Wang and John D. Owens and Yixuan Li , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  5. [5]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Yiyou Sun and Chuan Guo and Yixuan Li , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  6. [6]

    International Conference on Machine Learning (ICML) , year =

    Yiyou Sun and Yifei Ming and Xiaojin Zhu and Yixuan Li , title =. International Conference on Machine Learning (ICML) , year =

  7. [7]

    arXiv preprint arXiv:2106.09022 , year =

    Jie Ren and Stanislav Fort and Jeremiah Liu and Abhijit Guha Roy and Shreyas Padhy and Balaji Lakshminarayanan , title =. arXiv preprint arXiv:2106.09022 , year =

    arXiv

  8. [8]

    Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track , year =

    Jingkang Yang and Pengyun Wang and Dejian Zou and Zitang Zhou and others , title =. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track , year =

Show all 25 references
  1. [9]

    ACM Transactions on Knowledge Discovery from Data , volume =

    Shachar Kaufman and Saharon Rosset and Claudia Perlich and Ori Stitelman , title =. ACM Transactions on Knowledge Discovery from Data , volume =

  2. [10]

    Patterns , volume =

    Sayash Kapoor and Arvind Narayanan , title =. Patterns , volume =

  3. [11]

    Journal of Multivariate Analysis , volume =

    Olivier Ledoit and Michael Wolf , title =. Journal of Multivariate Analysis , volume =

  4. [12]

    Scientific Reports , volume =

    Christos Constantinou and Georgios Ioannides and Aman Chadha and Aaron Elkins and Edwin Simpson , title =. Scientific Reports , volume =. 2026 , note =

    2026

  5. [13]

    Vishnu Bindu Balachandran , title =

  6. [14]

    Y. L. Li and D. Lu and P. Kirichenko and S. Qiu and T. G. J. Rudner and C. B. Bruss and A. G. Wilson , title =. arXiv preprint arXiv:2507.01831 , year =

    arXiv

  7. [15]

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun , title =. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  8. [16]

    International Conference on Learning Representations (ICLR) , year =

    Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and others , title =. International Conference on Learning Representations (ICLR) , year =

  9. [17]

    Alex Krizhevsky , title =

  10. [18]

    Ng , title =

    Yuval Netzer and Tao Wang and Adam Coates and Alessandro Bissacco and Bo Wu and Andrew Y. Ng , title =. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning , year =

  11. [19]

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    Mircea Cimpoi and Subhransu Maji and Iasonas Kokkinos and Sammy Mohamed and Andrea Vedaldi , title =. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  12. [20]

    Gradient-based learning applied to document recognition , journal =

    Yann LeCun and L. Gradient-based learning applied to document recognition , journal =

  13. [21]

    arXiv preprint arXiv:1708.07747 , year =

    Han Xiao and Kashif Rasul and Roland Vollgraf , title =. arXiv preprint arXiv:1708.07747 , year =

    arXiv

  14. [22]

    International Conference on Artificial Intelligence and Statistics (AISTATS) , year =

    Adam Coates and Andrew Ng and Honglak Lee , title =. International Conference on Artificial Intelligence and Statistics (AISTATS) , year =

  15. [23]

    In or out?

    Julian Bitterwolf and Maximilian M. In or out?. International Conference on Machine Learning (ICML) , year =

  16. [24]

    arXiv preprint arXiv:2606.02601 , year =

    Alejandro Ascarate and Leo Lebrat and Rodrigo Santa Cruz and Clinton Fookes and Olivier Salvado , title =. arXiv preprint arXiv:2606.02601 , year =

    arXiv

  17. [25]

    arXiv preprint arXiv:2510.15202 , year =

    Denis Janiak and Jakub Binkowski and Tomasz Kajdanowicz , title =. arXiv preprint arXiv:2510.15202 , year =

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.