Pith. sign in

REVIEW 3 major objections 6 minor 15 references

Natural backdoor absorption plus sparse verification drives attack success probability to zero in community training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 00:40 UTC pith:7BHLQFXG

load-bearing objection Clean DTMC of injection vs absorption plus lazy penalties that claims asymptotic collapse; the formal step from weight updates to lim p(t)=0 is incomplete, empirics are narrow but supportive. the 3 major comments →

arxiv 2607.06643 v1 pith:7BHLQFXG submitted 2026-07-07 cs.CR cs.LG

The Power of Backdoor Absorption in Community Training

Issam Seddik , Sami Souihi , Mohamed Tamaazousti , Sara Tucci Piergiovanni This is my paper
classification cs.CR cs.LG
keywords backdoor attackscommunity trainingbackdoor absorptionlazy verificationdiscrete-time Markov chainByzantine resiliencedecentralized AI training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When a resource-constrained owner must outsource model training to a community of trainers, some of whom may be malicious, full cryptographic re-checking of every step is too expensive. The paper shows that neural networks already "wash out" stealthy backdoors under a continuous stream of honest updates, and that this natural absorption can be turned into a defense. By modelling the tug-of-war between injection and absorption as a Markov chain, the authors prove that a light combination of randomized trainer selection and occasional verification (as little as 10 percent of steps) forces any bounded adversary's success probability to zero over time, without harming clean accuracy. The result matters because it offers a mathematically grounded way to secure decentralized training without the prohibitive cost of exhaustive audits.

Core claim

Under a composite defense that pairs the natural absorption of backdoors by honest updates with a randomized scheduler and a lazy verification oracle, the success probability of any adversary who controls only a fixed fraction of trainers collapses asymptotically to zero, even when the oracle is invoked on only a small fraction of training steps.

What carries the argument

A Discrete-Time Markov Chain whose states track consecutive malicious injection steps versus consecutive honest absorption steps; under time-inhomogeneous penalties the adversary's selection probability is driven to zero, so the long-run probability of residing in a viable backdoor state vanishes.

Load-bearing premise

The adversary never learns how many training steps remain before deployment and therefore cannot simply poison the final steps to bypass absorption.

What would settle it

Run the same community-training protocol while giving the adversary exact knowledge of the remaining horizon T; if a last-minute injection still yields high attack success rate despite 10 percent lazy verification and weight penalties, the asymptotic claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies backdoor resilience in a sequential community-training setting where a resource-constrained owner delegates steps to n trainers of which f are malicious. It models the competition between adversarial injection (budget τ consecutive malicious steps) and natural backdoor absorption (lifespan π honest steps) as a Discrete-Time Markov Chain with Progress-Preserving (PPI) and Progress-Resetting (PRI) injection topologies, derives expected hitting times and a static steady-state success probability via renewal reward, then proposes a composite defense of natural absorption, weight-based randomized scheduling, and lazy verification (audit probability v, detection rate d, penalty fraction S). The central claim is that progressive weight penalties drive the adversary’s selection probability to zero asymptotically (lim t→∞ p(t)=0), forcing attack success probability to collapse, with empirical support on ResNet-18/CIFAR-10/BadNet showing ASR reduced from ~99% to 7.2% at v=0.1 with stable clean accuracy.

Significance. If the asymptotic guarantee holds under the stated threat model, the work offers a practically useful and theoretically motivated alternative to exhaustive cryptographic auditing for decentralized/community training: security via sparse selective verification plus the network’s own absorption dynamics. Strengths include a clean closed-form DTMC treatment of PPI/PRI hitting times (Eqs. 2–4), an explicit renewal-reward baseline for static vulnerability (Eq. 7), numerical sensitivity analysis across (τ,π) and (p,q), and empirical confirmation that empirical ASR lies between the theoretical PPI upper and PRI lower bounds (Fig. 5) with only ~2.8% overhead at 10% lazy verification (Table 2). These elements make the contribution actionable for safety-critical community AI if the load-bearing asymptotic argument is completed.

major comments (3)
  1. §4.5, Eqs. (8)–(11): The paper asserts lim t→∞ Wmal(t)=0 ⇒ lim p(t)=0 and therefore asymptotic collapse of attack success, but supplies only the per-detection multiplicative rule Wi(t+1)=(1−S)·Wi(t) and an informal implication. There is no theorem establishing almost-sure (or high-probability) depletion of a fixed coalition of f trainers under sparse audits (v=0.1). Because only the caught trainer is penalized, detections are stochastic, and the adversary can still be scheduled and re-poison between detections, residual aggregate weight need not vanish; the ordered product of time-inhomogeneous matrices (Eq. 10) is not shown to drive absorption-state mass to zero. A rigorous argument (e.g., supermartingale / Borel–Cantelli on cumulative detections, or an explicit bound on limsup p(t)) is load-bearing for the abstract’s “we prove” claim and must be supplied or the claim weakened to a high
  2. §3.2 (blind-horizon assumption) and §4.4–4.5: The entire asymptotic analysis relies on the adversary never learning remaining steps T−t and therefore being unable to concentrate injection near deployment. The manuscript correctly notes that last-minute injection would bypass absorption, yet provides no robustness analysis (partial horizon leakage, noisy estimates of T, or adaptive adversaries that treat verification events as signals). Because this assumption is identified as critical to the defense and is not an artifact of the DTMC alone, the paper should either (i) prove that weight depletion still forces success probability below a target even under limited horizon knowledge, or (ii) explicitly scope the guarantee to blind adversaries and quantify failure under horizon leakage.
  3. §4.2–4.3 vs. §5: The formal bounds treat τ and π as fixed exogenous parameters of the attack/defense geometry, while the empirical evaluation calibrates τ=2, π=27 to match observed BadNet dynamics on ResNet-18/CIFAR-10 (Fig. 5 caption). The mapping from continuous ASR trajectories to discrete consecutive-step counters is not justified beyond calibration, nor is sensitivity to mis-estimated (τ,π) analyzed. Without a principled estimator or a demonstration that the collapse result is robust to moderate mis-specification of τ and π, the claimed bridge from DTMC theory to the empirical 7.2% ASR result remains incomplete.
minor comments (6)
  1. Abstract and §1 claim a “proof” that success probability collapses to zero; §4.5 presents only an informal limit implication. Align wording with the actual formal content once the depletion argument is completed.
  2. Figure 3 caption and panel labels use “<” / “>” without defining the τ-vs-π regimes in the figure itself; add explicit (τ,π) values in the legend for readability.
  3. Notation: p is used both for malicious selection probability and (in §5.1) for poisoning ratio; disambiguate (e.g., ρ for poisoning ratio as in §2.2).
  4. Table 1 reports extreme values (e.g., µ′_I ≈ 1.11×10^10) without stating the exact (τ,π) pairs used; include them in the table header or caption.
  5. Related work on backdoor absorption / catastrophic forgetting (Kemker et al., Du et al., Souly et al., Zhang et al.) is cited but the novelty relative to “absorption as defense” could be stated more sharply in §2.2–2.3.
  6. Empirical evaluation is limited to BadNet on ResNet-18/CIFAR-10; a short note on expected behavior under other triggers (e.g., clean-label or sample-specific) would strengthen §5 without requiring full re-runs.

Circularity Check

1 steps flagged

Minor non-load-bearing self-citation to authors' own PoTS oracle for empirical parameters; the DTMC hitting-time bounds, renewal-reward asymptotics, and weight-penalty collapse claim are derived independently of that citation and are not forced by construction from fitted inputs.

specific steps
  1. self citation load bearing [Section 5.1 Defense Configurations / Table 2]
    "While our defense framework is fundamentally oracle-agnostic, our evaluation models the verification oracle after PoTS (Seddik et al., 2025), which reduces per-step verification time to 30% while maintaining a 0.9 detection rate. Accordingly, we configure our simulated oracle with d=0.9."

    Authors cite their own concurrent PoTS paper for the concrete detection probability and overhead numbers used in the empirical evaluation of the composite defense. The citation is not load-bearing for the DTMC theorems or the asymptotic collapse claim (which hold for any d>0, v>0, S>0), but it is the only self-reference that supplies numerical parameters appearing in the reported results; scored as the minor non-load-bearing case that justifies score 2 rather than 0.

full rationale

The paper's central derivation chain (state-space partition into injection/absorption states, closed-form expected hitting times under PPI/PRI topologies via geometric waiting times and recursive expectations, renewal-reward steady-state probability, and the time-inhomogeneous product under dynamic weights) is self-contained Markov-chain analysis that does not reduce to its own inputs by definition. τ and π are free parameters of the threat model (calibrated only for the empirical plots); the asymptotic lim p(t)=0 is asserted from the multiplicative penalty rule Wi(t+1)=(1-S)Wi(t) whenever a detection occurs, not obtained by fitting a constant that is later re-labeled a prediction. The sole self-citation (to Seddik et al. 2025 PoTS) supplies only the concrete detection rate d=0.9 and 30% overhead used in the simulation tables; the text explicitly states the framework is oracle-agnostic, so the citation is not load-bearing for the claimed lim P(Success)=0. No uniqueness theorem, ansatz, or renaming of a known empirical pattern is imported. The incompleteness of the almost-sure depletion argument for a fixed-size coalition is a correctness gap, not circularity. Hence only the minor self-citation raises the score above 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central asymptotic claim rests on standard Markov-chain hitting-time formulas, the modeling choice that absorption requires exactly π+1 consecutive honest steps, the blind-horizon premise that prevents last-minute injection, and the operational assumption that a verification oracle with positive detection rate exists and that weight penalties permanently reduce future selection probability. Free parameters (v, S, d, α, calibrated τ/π) affect quantitative rates but not the qualitative collapse once they are strictly positive. No new physical entities are postulated; the invented constructs are purely modeling devices.

free parameters (5)
  • lazy verification probability v = 0.1
    Chosen as 0.1 in both theory plots and experiments; any v>0 yields asymptotic collapse, but the concrete 10% figure is a free design choice that determines practical overhead.
  • penalty fraction S = 0.2
    Set to 0.2; controls how fast malicious weight is depleted. Not derived from first principles.
  • detection rate d of the verification oracle = 0.9
    Set to 0.9 following the cited PoTS oracle; enters the effective transition probabilities p′=p(1−v d).
  • ASR viability threshold α = 0.5
    Defines the boundary between injection and absorption states; set to 0.5 for experiments.
  • injection lifespan τ and absorption lifespan π = e.g. τ=2, π=27 in Fig. 5
    Calibrated per attack configuration so that the DTMC matches observed consecutive-step behavior; free parameters of the model rather than measured constants.
axioms (5)
  • domain assumption Blind-horizon assumption: adversary never observes remaining steps T−t and therefore cannot time a last-minute injection.
    Stated in Section 3.2 as a fundamental reality of owner-orchestrated scheduling; load-bearing for the claim that absorption cannot be bypassed.
  • domain assumption Backdoor absorption occurs after exactly π+1 consecutive honest steps (and is fully reset by any malicious step).
    Defines the absorption-state chain length; idealizes continuous gradient dynamics into a pure consecutive-count process (Section 2.2 and 4.3).
  • standard math Standard Discrete-Time Markov Chain hitting-time and Renewal-Reward theorems apply to the constructed state space.
    Used to obtain closed forms for µI, µA and Psteady (Eqs. 2–7); classical results of Ross / Levin-Peres / Gallager.
  • domain assumption A verification oracle exists that, when invoked, detects a malicious update with fixed probability d and allows the owner to apply a permanent weight penalty.
    Section 4.5; the paper is oracle-agnostic but assumes such an oracle with d>0 is available (instantiated by PoTS).
  • ad hoc to paper Trainer selection probabilities remain proportional to current trust weights and are independent across steps given those weights.
    Defines the time-inhomogeneous transition matrices P(t) (Eq. 8–10); modeling choice of the defense.
invented entities (3)
  • Injection / Absorption state partitions (I0…Iτ−1, A0…Aπ) of the DTMC no independent evidence
    purpose: Track consecutive malicious progress and consecutive honest decay so that hitting times and asymptotic success probability become computable.
    Purely formal constructs introduced in Section 4.1; no independent physical existence claimed.
  • Progress-Preserving Injection (PPI) and Progress-Resetting Injection (PRI) topologies no independent evidence
    purpose: Provide adversary best-case and worst-case bounds on expected injection time.
    Modeling extremes introduced in Section 4.2; real training dynamics lie somewhere between them.
  • Absorption lifespan π independent evidence
    purpose: Quantify how many consecutive honest steps are required to drive ASR below α after a successful injection.
    Defined in Section 2.2 / 3.2 from prior absorption literature; treated as a fixed integer parameter of the chain.

pith-pipeline@v1.1.0-grok45 · 20274 in / 3833 out tokens · 50667 ms · 2026-07-11T00:40:25.555889+00:00 · methodology

0 comments
read the original abstract

Backdoor attacks severely threaten large-scale AI models. When model owners delegate training to external compute providers within a decentralized training paradigm, adversaries can craft stealthy, low-frequency triggers to inject malicious behavior while evading standard audits. Traditionally, detecting these attacks requires a full re-computation of the training steps--a prohibitive overhead that directly contradicts the owner's resource constraints. To address this, we investigate the resilience of continuous optimization dynamics under Byzantine perturbations, where adversaries are forced to compete against a continuous influx of honest updates. Under a threat model where an adversary compromises f out of n total trainers, we quantify the minimum auditing overhead required by the model owner to probabilistically bound the attack success rate. We formalize this injection-absorption dynamic as a Discrete-Time Markov Chain (DTMC). Using this framework, we prove that the success probability of any bounded adversary asymptotically collapses to zero under a defense strategy combining natural absorption, a randomized scheduler, and lazy verification oracle. Empirical results demonstrate significant backdoor suppression with zero utility degradation even when invoking the verification oracle on merely 10% of the total training steps. This approach yields a provably sound and computationally efficient defense for safety-critical AI.

Figures

Figures reproduced from arXiv: 2607.06643 by Issam Seddik, Mohamed Tamaazousti, Sami Souihi, Sara Tucci Piergiovanni.

Figure 1
Figure 1. Figure 1: Adversarial injection and honest absorption dynamics in Community Training. The Model Owner delegates [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: DTMC state transitions for the adversary’s injection phase, detailing (a) Progress-Preserving Injection (PPI) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Theoretical backdoor absorption dynamics. (a) Attack success probability versus honest fraction [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Attack Success Rate (ASR) and Clean Accuracy (CACC) evolution for a BadNet attack on ResNet-18 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Empirical ASR evolution versus theoretical bounds for a BadNet attack on ResNet-18 over 1,000 remaining [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 6 internal anchors

  1. [1]

    Zero-knowledge proofs of training for deep neural networks

    Kasra Abbaszadeh, Christodoulos Pappas, Jonathan Katz, and Dimitrios Papadopoulos. Zero-knowledge proofs of training for deep neural networks. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 4316–4330,

    work page 2024

  2. [2]

    Large-scale machine learning with stochastic gradient descent

    Léon Bottou. Large-scale machine learning with stochastic gradient descent. InProceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177–186. Springer,

    work page 2010

  3. [3]

    Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning

    Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning.arXiv preprint arXiv:1712.05526,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Decoupled DiLoCo for Resilient Distributed Pre-training

    Arthur Douillard, Keith Rush, Yani Donchev, Zachary Charles, Nova Fallen, Ayush Dubey, Ionel Gog, Josef Dean, Blake Woodworth, Zachary Garrett, et al. Decoupled diloco for resilient distributed pre-training.arXiv preprint arXiv:2604.21428,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Lifelong anomaly detection through unlearning

    Min Du, Zhi Chen, Chang Liu, Rajvardhan Oak, and Dawn Song. Lifelong anomaly detection through unlearning. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pages 1283–1297,

    work page 2019

  6. [6]

    BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

    http://www. deeplearningbook.org. 13 The Power of Backdoor Absorption in Community Training Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain.arXiv preprint arXiv:1708.06733,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

    Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive llms that persist through safety training.arXiv preprint arXiv:2401.05566,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Zkcnn: Zero knowledge proofs for convolutional neural network predictions and accuracy

    Tianyi Liu, Xiang Xie, and Yupeng Zhang. Zkcnn: Zero knowledge proofs for convolutional neural network predictions and accuracy. InProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 2968–2985,

    work page 2021

  9. [9]

    Eiffel: Ensuring integrity for federated learning

    Amrita Roy Chowdhury, Chuan Guo, Somesh Jha, and Laurens Van der Maaten. Eiffel: Ensuring integrity for federated learning. InProceedings of the 2022 ACM SIGSAC conference on computer and communications security, pages 2535–2549,

    work page 2022

  10. [10]

    Convergence of markov chains for constant step-size stochastic gradient descent with separable functions.SIAM Journal on Applied Dynamical Systems, 24(3):2005–2043,

    David Shirokoff and Philip Zaleski. Convergence of markov chains for constant step-size stochastic gradient descent with separable functions.SIAM Journal on Applied Dynamical Systems, 24(3):2005–2043,

    work page 2005

  11. [11]

    Poisoning attacks on llms require a near-constant number of poison samples.arXiv preprint arXiv:2510.07192,

    14 The Power of Backdoor Absorption in Community Training Alexandra Souly, Javier Rando, Ed Chapman, Xander Davies, Burak Hasircioglu, Ezzeldin Shereen, Carlos Mougan, Vasilios Mavroudis, Erik Jones, Chris Hicks, et al. Poisoning attacks on llms require a near-constant number of poison samples.arXiv preprint arXiv:2510.07192,

    work page arXiv

  12. [12]

    Can You Really Backdoor Federated Learning?

    Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning?arXiv preprint arXiv:1911.07963,

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  13. [13]

    Latent backdoor attacks on deep neural networks

    Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pages 2041–2055,

    work page 2019

  14. [14]

    Narcissus: A practical clean-label backdoor attack with limited information

    Yi Zeng, Minzhou Pan, Hoang Anh Just, Lingjuan Lyu, Meikang Qiu, and Ruoxi Jia. Narcissus: A practical clean-label backdoor attack with limited information. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 771–785,

    work page 2023

  15. [15]

    Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering

    Rui Zhu, Di Tang, Siyuan Tang, Guanhong Tao, Shiqing Ma, Xiaofeng Wang, and Haixu Tang. Gradient shaping: Enhancing backdoor attack against reverse engineering.arXiv preprint arXiv:2301.12318,

    work page internal anchor Pith review Pith/arXiv arXiv