Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

Modern neural networks can contain backdoors hidden as learned latent directions that are cryptographically undetectable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 21:53 UTC pith:E4LQU2RR

load-bearing objection The paper extends backdoor attacks to ResNet and ViT via latent directions but rests undetectability on an unproven conjecture about an intractable hypothesis test. the 2 major comments →

arxiv 2605.13214 v2 pith:E4LQU2RR submitted 2026-05-13 cs.CR cs.LG

Backdoor Channels Hidden in Latent Space: Cryptographic Undetectability in Modern Neural Networks

Marte Eggen , Eirik Reiestad , Kristian Gj{\o}steen , Inga Str\"umke This is my paper
classification cs.CR cs.LG
keywords backdoor attacksneural network securitylatent spacecryptographic undetectabilityhypothesis testingmachine learning securityResNetVision Transformer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper constructs a backdoor attack for modern architectures like ResNet and Vision Transformers by framing backdoor channels as learned latent directions rather than artificial additions. This reframing reduces the problem of undetectability to a hypothesis test between two unknown distributions over model parameters, which the authors conjecture is intractable in practice. If true, this means attackers can exploit the geometry the network already possesses instead of introducing foreign structure that might be detectable. The attack is shown to achieve high success rates with little impact on clean accuracy and to resist various post-training defenses.

Core claim

By identifying backdoor channels as learned latent directions, the question of undetectability reduces to a hypothesis test between two unknown distributions over model parameters, conjectured to be intractable, allowing the attack to succeed on state-of-the-art architectures trained on standard datasets while resisting defenses.

What carries the argument

Backdoor channels identified as learned latent directions in the latent space of the neural network, which allows the backdoored model to be statistically close to a clean model.

Load-bearing premise

The hypothesis test distinguishing the parameter distributions of backdoored and clean models is intractable in practice.

What would settle it

Discovery of an efficient algorithm that can reliably distinguish a backdoored model from a clean model by inspecting its parameters or behavior would falsify the claim of cryptographic undetectability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The attack works on ResNet and Vision Transformer architectures with high success rates and negligible clean accuracy degradation.
  • The backdoor resists a comprehensive suite of post-training defences without making the model unusable.
  • Cryptographic backdoors can be inherent properties of the geometry of learned representations rather than requiring exotic architectures.
  • Undetectability holds if no efficient algorithm can distinguish the backdoored model from a clean one based on parameter distributions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the hypothesis test is indeed intractable, then existing detection methods based on statistical anomalies may not suffice for these backdoors.
  • This suggests that the latent space geometry learned during standard training can be repurposed for malicious use without additional modifications.
  • Future work could explore whether training regularizations can prevent such latent directions from being exploitable.
  • Similar approaches might apply to other domains like language models where latent directions are studied.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper claims that backdoors can be embedded into modern neural networks (ResNet, Vision Transformer) trained on standard image datasets by treating them as latent directions already present in the learned parameter geometry. It reframes cryptographic undetectability as the intractability of distinguishing the parameter distribution of a clean model from that of a backdoored model via hypothesis testing, conjecturing this test to be intractable in practice. The construction is shown to achieve high attack success rates with negligible clean accuracy loss and to resist a suite of post-training defenses.

Significance. If the intractability conjecture is substantiated, the result would be significant for showing that cryptographic-style undetectability need not rely on stylized architectures but can arise from the geometry of representations learned by practical end-to-end models. The empirical demonstration on ResNet and ViT provides concrete evidence of attack viability against existing defenses.

major comments (2)
  1. [Abstract] Abstract: the central claim that undetectability 'reduces to a hypothesis test between two unknown distributions over model parameters, which we conjecture to be intractable in practice' supplies the intractability needed for the cryptographic guarantee but provides neither a reduction to a known hard problem nor a formal argument that the latent-direction construction forces computational indistinguishability; this is load-bearing for the main result.
  2. [Section 4, Section 5] Section 4 (construction) and Section 5 (experiments): resistance is demonstrated only against a finite list of post-training defenses; the manuscript does not argue why this list rules out the existence of an efficient statistical or learning-based distinguisher outside the tested suite, leaving the conjecture without direct empirical grounding.
minor comments (2)
  1. [Section 3] Notation for the two parameter distributions (clean vs. backdoored) is introduced without an explicit definition or reference to how they are sampled in the hypothesis test.
  2. [Table 2] Table 2 (attack success rates) lacks error bars or details on the number of random seeds used for the reported means.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful and constructive report. The comments correctly identify that our central claims rest on a conjecture of intractability rather than a formal reduction, and that the empirical evaluation covers a finite set of defenses. We address each point below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that undetectability 'reduces to a hypothesis test between two unknown distributions over model parameters, which we conjecture to be intractable in practice' supplies the intractability needed for the cryptographic guarantee but provides neither a reduction to a known hard problem nor a formal argument that the latent-direction construction forces computational indistinguishability; this is load-bearing for the main result.

    Authors: We agree that the intractability is presented as a conjecture rather than derived from a reduction to a known hard problem. The manuscript explicitly uses the word 'conjecture' and motivates the claim by showing that backdoor directions align with the geometry of naturally learned latent directions in ResNet and ViT parameter spaces. No formal proof of computational indistinguishability is provided. We will revise the abstract and the opening of Section 1 to state more explicitly that the cryptographic-style guarantee rests on this unproven but empirically supported conjecture, and we will add a short paragraph in the discussion section outlining why a reduction appears difficult given the data-dependent nature of the learned representations. revision: partial

  2. Referee: [Section 4, Section 5] Section 4 (construction) and Section 5 (experiments): resistance is demonstrated only against a finite list of post-training defenses; the manuscript does not argue why this list rules out the existence of an efficient statistical or learning-based distinguisher outside the tested suite, leaving the conjecture without direct empirical grounding.

    Authors: The observation is accurate: the experiments evaluate a concrete but finite collection of post-training defenses and statistical tests. We do not claim that passing these tests constitutes a proof that no efficient distinguisher exists. The conjecture draws its support from the construction itself—embedding the backdoor as a direction already present in the clean model's latent geometry—together with the consistent failure of the tested methods to isolate it without destroying utility. We will expand Section 5 with an explicit discussion acknowledging the limitation of any finite test suite and clarifying that the results are consistent with, but do not prove, the intractability conjecture. If space allows, we will also report results from two additional learning-based probes (e.g., membership-inference-style classifiers on parameter subsets) to further illustrate the difficulty. revision: yes

Circularity Check

0 steps flagged

No significant circularity in the derivation chain

full rationale

The paper constructs a backdoor mechanism by identifying latent directions and reduces the undetectability question to a hypothesis test between two parameter distributions, then explicitly states a conjecture of practical intractability. This conjecture is presented as an assumption rather than derived via any self-referential equation, fitted parameter renamed as prediction, or load-bearing self-citation chain. No quoted step equates the main claim to its inputs by construction, and the enumerated circularity patterns are not exhibited. The derivation therefore remains self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The load-bearing element is the unproven conjecture that the hypothesis test over parameter distributions is intractable; no free parameters or invented entities are described in the abstract.

axioms (1)
  • ad hoc to paper The hypothesis test between two unknown distributions over model parameters is intractable in practice
    Explicitly stated as a conjecture supporting the undetectability claim in the abstract.

pith-pipeline@v0.9.1-grok · 5776 in / 1389 out tokens · 35654 ms · 2026-06-30T21:53:01.958438+00:00 · methodology

0 comments
read the original abstract

Recent cryptographic results establish that neural networks can be backdoored such that no efficient algorithm can distinguish them from a clean model. These guarantees, however, have been confined to stylised architectures of limited practical relevance, leaving open whether comparable undetectability extends to modern, end-to-end trained networks. We construct such an attack mechanism for state-of-the-art architectures, closely aligned to the cryptographic notion of undetectability, by identifying backdoor channels as learned latent directions, and show that the question of undetectability reduces to a hypothesis test between two unknown distributions over model parameters, which we conjecture to be intractable in practice. The consequence of this reframing is significant: if exploitable channels within a network's latent space are statistically indistinguishable from naturally learned directions, an attacker need not introduce foreign structure but can instead exploit the geometry the network already possesses. Demonstrating the approach on ResNet and Vision Transformer architectures trained on standard image classification datasets, the attack achieves both consistently high success rates with negligible clean accuracy degradation, and resists a comprehensive suite of post-training defences, none of which neutralise the backdoor without rendering the model unusable. Our results establish that cryptographic backdoors need not be artefacts requiring exotic architectures or artificial constructions, but identifiable as latent properties inherent to the geometry of learned representations.

Figures

Figures reproduced from arXiv: 2605.13214 by Eirik Reiestad, Inga Str\"umke, Kristian Gj{\o}steen, Marte Eggen.

Figure 1
Figure 1. Figure 1: PCA visualisation of the activation space of a backdoored single-hidden-layer ReLU [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy over five epochs of (blue) fine-tuning and (green) fine-pruning defences for [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) The clean distribution follows a standard multivariate Gaussian, which is modified with [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy over five epochs of (blue) fine-tuning and (green) fine-pruning defences for [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Backdoor layer weight distributions of backdoored models compared to their clean counter [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Original source class (left), triggered input (middle), and secret key (right, shown on a [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Visibility of the trigger, quantified by the MMD score, as a function of the scaling [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Statistically Undetectable Backdoors in Deep Neural Networks

    cs.LG 2026-07 conditional novelty 7.0

    Trainers can plant statistically undetectable white-box backdoors in constrained DNNs that give exponential advantage for invariance-based adversarial examples, while outsiders cannot find any in poly-time under latti...

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages · cited by 1 Pith paper · 5 internal anchors

  1. [1]

    arXiv preprint arXiv:2509.07504 , year=

    Bilal Hussain Abbasi, Yanjun Zhang, Leo Zhang, and Shang Gao. Backdoor attacks and defenses in computer vision domain: A survey.arXiv preprint arXiv:2509.07504, 2025

    work page arXiv 2025

  2. [2]

    Complexity theoretic lower bounds for sparse principal component detection

    Quentin Berthet and Philippe Rigollet. Complexity theoretic lower bounds for sparse principal component detection. In Shai Shalev-Shwartz and Ingo Steinwart, editors,COLT 2013 - The 26th Annual Conference on Learning Theory, June 12-14, 2013, Princeton University, NJ, USA, JMLR Workshop and Conference Proceedings, pages 1046–1066. JMLR.org, 2013. URL http...

    work page 2013

  3. [3]

    Computational Lower Bounds for Sparse PCA

    Quentin Berthet and Philippe Rigollet. Computational lower bounds for sparse pca.arXiv preprint arXiv:1304.0828, 2013

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  4. [4]

    Brennan and Guy Bresler

    Matthew S. Brennan and Guy Bresler. Optimal average-case reductions to sparse PCA: from weak assumptions to strong hardness. In Alina Beygelzimer and Daniel Hsu, editors,Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA, Proceedings of Machine Learning Research, pages 469–470. PMLR, 2019. URL http://proceedings.mlr.press/ v99/bre...

    work page 2019

  5. [5]

    Data free backdoor attacks.Advances in Neural Information Processing Systems, 37:23881–23911, 2024

    Bochuan Cao, Jinyuan Jia, Chuxuan Hu, Wenbo Guo, Zhen Xiang, Jinghui Chen, Bo Li, and Dawn Song. Data free backdoor attacks.Advances in Neural Information Processing Systems, 37:23881–23911, 2024

    work page 2024

  6. [6]

    Wild patterns reloaded: A survey of machine learning security against training data poisoning

    Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, and Fabio Roli. Wild patterns reloaded: A survey of machine learning security against training data poisoning. ACM Computing Surveys, 55(13s):1–39, 2023

    work page 2023

  7. [7]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  8. [8]

    Unelicitable backdoors via cryptographic transformer circuits

    Andis Draguns, Andrew Gritsevskiy, Sumeet Ramesh Motwani, and Christian Schroeder de Witt. Unelicitable backdoors via cryptographic transformer circuits. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 53684–53709. Curran Associates, Inc., 202...

    work page doi:10.52202/079017-1700 2024

  9. [9]

    Di Luna and Giovanni Viglietta

    Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models : [extended abstract]. In2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pages 931–942, 2022. doi: 10.1109/ FOCS54457.2022.00092

    work page arXiv 2022

  10. [10]

    Kim, Vinod Vaikuntanathan, and Or Zamir

    Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models, 2024. URL https://arxiv.org/abs/2204.06974

    work page arXiv 2024

  11. [11]

    Borgwardt, Malte J

    Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test.Journal of Machine Learning Research, 13(25):723–773,

    work page

  12. [12]

    URLhttp://jmlr.org/papers/v13/gretton12a.html

    work page

  13. [13]

    Papakostas

    Ioannis Grigoriadis, Eleni Vrochidou, Iliana Tsiatsiou, and George A. Papakostas. Machine learning as a service (mlaas)—an enterprise perspective. In Mukesh Saraswat, Chandreyee Chowdhury, Chintan Kumar Mandal, and Amir H. Gandomi, editors,Proceedings of Interna- tional Conference on Data Science and Applications, pages 261–273, Singapore, 2023. Springer ...

    work page 2023

  14. [14]

    Concept backpropagation: An explainable ai approach for visualising learned concepts in neural network models.arXiv preprint arXiv:2307.12601, 2023

    Patrik Hammersborg and Inga Strümke. Concept backpropagation: An explainable ai approach for visualising learned concepts in neural network models.arXiv preprint arXiv:2307.12601, 2023. 10

    work page arXiv 2023

  15. [15]

    Survey on backdoor attacks on deep learning: Current trends, categorization, applications, research challenges, and future prospects.IEEE Access, 2025

    Muhammad Abdullah Hanif, Nandish Chattopadhyay, Bassem Ouni, and Muhammad Shafique. Survey on backdoor attacks on deep learning: Current trends, categorization, applications, research challenges, and future prospects.IEEE Access, 2025

    work page 2025

  16. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    work page 2016

  17. [17]

    Handcrafted backdoors in deep neural networks.Advances in Neural Information Processing Systems, 35:8068–8080, 2022

    Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. Handcrafted backdoors in deep neural networks.Advances in Neural Information Processing Systems, 35:8068–8080, 2022

    work page 2022

  18. [18]

    Injecting undetectable backdoors in obfuscated neural networks and language models.Advances in Neural Information Processing Systems, 37:21537–21571, 2024

    Alkis Kalavasis, Amin Karbasi, Argyris Oikonomou, Katerina Sotiraki, Grigoris Velegkas, and Manolis Zampetakis. Injecting undetectable backdoors in obfuscated neural networks and language models.Advances in Neural Information Processing Systems, 37:21537–21571, 2024

    work page 2024

  19. [19]

    Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). InInternational conference on machine learning, pages 2668–2677. PMLR, 2018

    work page 2018

  20. [20]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  21. [21]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    work page 2009

  22. [22]

    Analyzing and editing inner mechanisms of backdoored language models

    Max Lamparth and Anka Reuel. Analyzing and editing inner mechanisms of backdoored language models. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2362–2373, 2024

    work page 2024

  23. [23]

    Fine-pruning: Defending against backdooring attacks on deep neural networks

    Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. InInternational symposium on research in attacks, intrusions, and defenses, pages 273–294. Springer, 2018

    work page 2018

  24. [24]

    Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.Journal of Machine Learning Research, 22(165):1–73, 2021

    Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.Journal of Machine Learning Research, 22(165):1–73, 2021

    work page 2021

  25. [25]

    Random features for large-scale kernel machines.Advances in neural information processing systems, 20, 2007

    Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines.Advances in neural information processing systems, 20, 2007

    work page 2007

  26. [26]

    Mauro Ribeiro, Katarina Grolinger, and Miriam A.M. Capretz. Mlaas: Machine learning as a service. In2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 896–902, 2015. doi: 10.1109/ICMLA.2015.152

    work page doi:10.1109/icmla.2015.152 2015

  27. [27]

    Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

    Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond.arXiv preprint arXiv:1611.07476, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  28. [28]

    Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

    Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks.arXiv preprint arXiv:1706.04454, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  29. [29]

    Empirical analysis of the hessian of over-parametrized neural networks

    Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. InICLR 2018 Workshop, 2018. URL https://iclr.cc/virtual/2018/workshop/563

    work page 2018

  30. [30]

    Neural cleanse: Identifying and mitigating backdoor attacks in neural networks

    Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In2019 IEEE symposium on security and privacy (SP), pages 707–723. IEEE, 2019

    work page 2019

  31. [31]

    Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.Scientific data, 10(1):41, 2023

    Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.Scientific data, 10(1):41, 2023. 11 A Visualisation of spiked covariance Figure 3 illustrates the difference between clean and backdoored distributions, hig...

    work page 2023