REVIEW 3 major objections 4 minor 48 references
Tight Stability Bounds for Robust Distributed Learning: Byzantine Failures Hurt Generalization More than Data Poisoning
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Byzantine failures cause strictly worse generalization than data poisoning in robust distributed learning.
desk verdict First real theoretical separation between Byzantine and poisoning generalization, but the Byzantine lower-bound proof as printed has a threshold bug and the abstract claims more than the f≥n/3 theorem proves. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is SMEA (smallest maximum eigenvalue averaging), the robust aggregation rule that averages the $n-f$ worker updates whose empirical covariance has the smallest spectral norm. The argument is carried by uniform stability: replacing one honest sample must move the output loss by at most $\varepsilon$, and the paper bounds $\varepsilon$ by comparing the robust update to the honest-average update. Under data poisoning, every update—including the poisoned ones—is a gradient of a smooth loss, so the analysis can intersect the two SMEA-selected subsets and charge the instability to the $f$ workers that swap: an additive $\frac{f}{n-f}$ term. Under Byzantine failures, corrupted vectors have no regularity, forcing the comparison to go through the $(f,\kappa)$-robustness coefficient $\kappa$; since $\kappa \ge \frac{f}{n-2f}$, the additive term is $\sqrt{\frac{f}{n-2f}}$. SMEA also violates the loss's co-coercivity inequality because its selected subset changes with the parameters, and this violation is exactly where the additive instability enters.
What would settle it
On the paper's one-dimensional construction ($\ell(\theta;z)=z\theta$, workers split between $+C$ and $-C$, and Byzantine workers running the pseudo-code as printed), change one honest sample from $C$ to $0$ and run GD with SMEA for $T$ steps. Under the printed condition `if $g_1 \le C$`, the Byzantine workers return the huge value on both trajectories, so the perturbed run will not diverge at the rate $\gamma C T/((n-f)m)$; observing no such divergence would falsify the claimed $\Omega(\gamma C^2 T \sqrt{f/(n-2f)})$ lower bound.
Extended reading notes
Core claim
On its own terms, the paper claims a fundamental, unavoidable separation between the two threat models. Under data poisoning, the uniform stability of GD/SGD with SMEA is tightly characterized: $\Theta(\gamma C^2 T (\frac{f}{n-f} + \frac{1}{(n-f)m}))$ in the convex case and $\Theta(\frac{C^2}{\mu}(\frac{f}{n-f}+\frac{1}{(n-f)m}))$ in the strongly convex case. Under Byzantine failures for $n/3 \le f < n/2$, the paper constructs a linear-loss instance where SMEA discards a whole block of honest gradients and an adaptive Byzantine switch makes the two neighboring runs diverge, yielding a lower bound $\Omega(\gamma C^2 T(\sqrt{\frac{f}{n-2f}} + \frac{1}{(n-f)m}))$. The upper bounds match these rates up to constants, and Theorem 4.2 converts the stability gap into a generalization-error ratio. The author's claim is that this gap is structural: data poisoning is strictly less harmful to population risk than Byzantine failure, even when both are designed optimally.
Load-bearing premise
The Byzantine lower bound assumes the attacker can observe a single honest gradient and switch between a crafted value $\beta$ and an arbitrarily large value exactly when one sample changes; the paper's printed pseudo-code uses the trigger `if $g_1 \le C$`, which is satisfied in both neighboring runs, so the proof as written depends on that comparison being implemented as a strict inequality.
Editorial extensions
If this is right
- Algorithms with optimal optimization guarantees under both threat models will still show different population risk; test error, not just training error, must be the benchmark.
- Under data poisoning, the $\Theta(\frac{f}{n-f})$ stability degradation is unavoidable for SMEA, so any attempt to get better generalization under poisoning must change the aggregation rule.
- As $f$ approaches $n/2$, the Byzantine generalization gap grows without bound, while the poisoning penalty stays bounded by $\frac{f}{n-f}$, so the two threat models separate most sharply in the high-Byzantine regime.
- In the strongly convex case, both gaps appear as a one-time $O(1/\mu)$ term rather than accumulating over iterations, which predicts that the relative harm of Byzantine failures is largest in early or convex regimes.
Reading between the lines
- The proof identifies a design criterion for robust aggregation: preserving co-coercivity of the loss would eliminate the worker-swapping term entirely; the open question of whether such a rule exists becomes a concrete route to closing the gap.
- The same comparison—regular poisoned gradients vs arbitrary adaptive Byzantine vectors—should reappear in momentum, variance reduction, or multi-step local-update methods, since the recursions only rely on the boundedness and smoothness of the updates.
- A practical test on heterogeneous federated benchmarks: fix a strong Byzantine attack and a near-optimal poisoning attack, vary $f$, and measure test accuracy; Theorem 4.2 predicts the gap should scale roughly like $(n-f)/\sqrt{f(n-2f)}$ rather than staying constant.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the uniform algorithmic stability of robust distributed (S)GD with the SMEA aggregation rule under two threat models: data poisoning and Byzantine failures. It proves matching upper and lower bounds for data poisoning—Θ(γC²T(f/(n−f)+1/((n−f)m))) in the convex case and Θ(C²/μ(f/(n−f)+1/((n−f)m))) in the strongly convex case—and a Byzantine lower bound Ω(γC²T(√(f/(n−2f))+1+1/((n−f)m))) for n/3≤f<n/2. Theorem 4.2 converts these stability bounds into a generalization-error gap of order (n−f)/√(f(n−2f)) between Byzantine failures and data poisoning. The appendix contains detailed proofs of all results, and numerical experiments illustrate the gap.
Significance. If the results hold, this is the first theoretical explanation for the empirically observed worse generalization of Byzantine attacks compared with data poisoning in robust distributed learning, and the data-poisoning bounds are genuinely tight with explicit constructions. The stability framework for (f,κ)-robust aggregation rules and the proof technique of comparing robust updates with honest averaging are useful contributions that go beyond the specific SMEA case. The main caveats are that the Byzantine lower bound and the generalization-gap theorem are proved only for f≥n/3, and that the load-bearing Byzantine trigger construction in Section C.3 contains a localized but correctable defect.
major comments (3)
- [Section C.3, Algorithm 1 and Theorem 3.2] The Byzantine trigger condition 'if g_1 ≤ C' is satisfied in both the unperturbed run (where worker 1 sends g_1 = C) and the perturbed run (where g_1 = (m−1)/m C < C). Consequently the Byzantine workers always return the arbitrarily high value and never send β. The claimed SMEA selection of Select_{n−2f}(H+) ∪ B and the divergence θ_T = −γCT(1 + (1/2)√(f/(n−2f))) are therefore not consequences of the printed algorithm, and the lower bound reduces to the negligible O(γCT/((n−f)m)) term. The proof should use a strict inequality (e.g., 'if g_1 < C then return high, else return β') so that the trigger fires only in the perturbed run. This is a load-bearing defect because the Byzantine-versus-poisoning separation for f ≥ n/3 rests precisely on this trigger.
- [Abstract, Section 1, Theorems 3.2 and 4.2] The abstract states the Byzantine degradation is Ω(√(f/(n−2f))) without any restriction on f, and the conclusion claims a fundamental gap between the two threat models in unqualified form. However, Theorem 3.2 and Theorem 4.2 hold only for n/3 ≤ f < n/2. For f < n/3 the paper supplies only a weaker lower bound Ω(γC²T f/(n−2f)) (Section C.3, case iii) and numerical evidence; no order-of-magnitude separation is established in that regime. The abstract and conclusion should explicitly restrict the strong separation claim to f ≥ n/3, or state precisely what is known for smaller f.
- [Section 4, proof of Theorem 4.2 and Lemma 4.1] The proof asserts that a single choice of honest data distributions with pivot p₁ = (1/2)δ₀ + (1/2)δ₋C makes the two possible honest datasets align with both the worst-case Byzantine and data-poisoning lower-bound scenarios. But the Byzantine construction in Section C.3 uses a pivot whose two values are C and 0 (with m=1), while the data-poisoning construction in Section D.2.1 uses pivot values 0 and −C and different values for the other honest workers. These are different distributions, so the proof as written does not show that the two lower bounds can be realized on the same {pᵢ}. The theorem can be repaired by using the general two-point-pivot form of Lemma 4.1 (which holds for any support {a,b} with |a−b|=C) and applying the data-poisoning upper bound from Theorem 3.3 to the distribution chosen for the Byzantine construction, but that step needs to be made explicit.
minor comments (4)
- [Theorems 3.1 and 3.3] The statements contain a typo: 'sup_z E_A[|ℓ(A(S');z)−ℓ(A(S');z)|]' should read 'ℓ(A(S);z)−ℓ(A(S');z)' in both theorems; the proofs use the correct difference.
- [Section 4, Theorem 4.2] The phrase 'Consider the setting of Theorem 4.1' is confusing because no Theorem 4.1 appears in the paper; the reference should be to Lemma 4.1 or to the setting of Section 4 more generally.
- [Definition 2.4] The uniform stability inequality should include an absolute value, i.e., sup_z |E[ℓ(A(S);z)−ℓ(A(S');z)]| ≤ ε, to avoid sign ambiguity; the subsequent proofs and Proposition 2.1 use the absolute value implicitly.
- [Section D.2.1, dataset construction] The notation for the neighboring datasets is terse: for S, worker 1 has z^{(1,1)} = −C and all other samples 0, while for S' all samples of worker 1 are 0. The text should state clearly which dataset the primed quantities refer to, especially because the proof then writes g^{(1)}_t = −C/m.
Circularity Check
No circular derivation: the stability and generalization bounds are proven by explicit constructions; the self-cited Allouah et al. robustness facts are background lemmas, not the target result.
full rationale
The paper's central chain is self-contained. Theorem 3.1 upper bounds are derived directly from the (f,κ)-robustness definition with standard expansivity arguments; Theorem 3.2 builds an explicit linear-loss construction with SMEA subset selection; Theorems 3.3 and 3.4 give matching upper and lower bounds for data poisoning using subset-intersection structure; and Theorem 4.2 converts these into generalization ratios through Lemma 4.1's exact worst-case computation. No parameter is fitted to data and no result is a renamed input: the numerical section re-instantiates the paper's own lower-bound construction, which is a sanity check rather than an independent empirical claim. The only author-overlapping citations are to Allouah et al. 2023a,b for the (f,κ)-robust framework, SMEA, and κ bounds; these are prior mathematical lemmas with stated assumptions that do not include the target stability gap, and the paper does not reduce its main lower bounds to them. One appendix correctness concern (the printed Byzantine trigger 'if g1 <= C' never sends β in either neighboring run) is a proof bug, not a circularity, so it does not change the circularity verdict.
Assumptions & free parameters
assumptions (5)
- domain assumption Loss functions are C-Lipschitz and L-smooth, with convex or µ-strongly convex sublevel sets over the parameter domain.
- domain assumption Aggregation rule F is (f,κ)-robust in the sense of Definition 2.2, and SMEA outputs the average of the n-f gradients with minimum spectral norm covariance.
- standard math κ satisfies κ ≥ f/(n-2f) for any (f,κ)-robust rule, and for SMEA √κ ∈ O(√(f/(n-f))(1+f/(n-2f))).
- standard math Hardt et al. classical stability recursion and co-coercivity of convex L-smooth functions.
- domain assumption In the lower bounds, the adversary has full knowledge of honest data distributions and can observe all communication and adapt at each iteration.
Cite this review
Pith. "Pith review of Tight Stability Bounds for Robust Distributed Learning: Byzantine Failures Hurt Generalization More than Data Poisoning." pith.science (2026). https://pith.science/paper/LLBKMI6J
@misc{pith2026250618020,
author = {Pith},
title = {Pith review of: Tight Stability Bounds for Robust Distributed Learning: Byzantine Failures Hurt Generalization More than Data Poisoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/LLBKMI6J}},
note = {Machine review of arXiv:2506.18020}
}
abstract
Robust distributed learning algorithms aim to maintain reliable performance despite the presence of misbehaving workers. Such misbehaviors are commonly modeled as \textit{Byzantine failures}, allowing arbitrarily corrupted communication, or as \textit{data poisoning}, a weaker form of corruption restricted to local training data. While prior work shows similar optimization guarantees for both models, an important question remains: \textit{How do these threat models impact generalization?} We show, for the first time, a fundamental gap in generalization guarantees between the two threat models: Byzantine failures yield strictly worse rates than those achievable under data poisoning. Our findings are based upon a tight algorithmic stability analysis of robust distributed learning. Specifically, with $f$ out of $n$ workers misbehaving, we prove that: \textit{(i)} under data poisoning, the uniform algorithmic stability of a robust distributed learning algorithm
Figures
Reference graph
Works this paper leans on
-
[1]
K. Abbaszadeh, C. Pappas, J. Katz, and D. Papadopoulos. Zero-knowledge proofs of training for deep neural networks. In B. Luo, X. Liao, J. Xu, E. Kirda, and D. Lie, editors, CCS , 2024
work page 2024
-
[2]
D. Alistarh, Z. Allen-Zhu, and J. Li. Byzantine stochastic gradient descent. Advances in neural information processing systems , 31, 2018
work page 2018
-
[3]
Z. Allen-Zhu, F. Ebrahimian, J. Li, and D. Alistarh. Byzantine-resilient non-convex stochastic gradient descent. arXiv preprint arXiv:2012.14368 , 2020
arXiv 2012
-
[4]
Y. Allouah, S. Farhadkhani, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan. Fixing by mixing: A recipe for optimal byzantine ml under heterogeneity. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , page...
work page 2023
-
[5]
Y. Allouah, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan. On the privacy-robustness-utility trilemma in distributed learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 569--...
work page 2023
-
[6]
P. Alquier. User-friendly introduction to pac-bayes bounds. Foundations and Trends in Machine Learning , 17(2):174--303, 2024
work page 2024
-
[7]
F. Bach. Learning Theory from First Principles . Adaptive Computation and Machine Learning series. MIT Press, 2024
work page 2024
- [8]
Show all 48 references
-
[9]
Bottou and O
L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007
2007
-
[10]
Bousquet and A
O. Bousquet and A. Elisseeff. Stability and generalization. Journal of Machine Learning Research , 2:499--526, 2002
2002
-
[11]
Charles and D
Z. Charles and D. Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research ,...
2018
-
[12]
Devroye and T
L. Devroye and T. Wagner. Distribution-free performance bounds for potential function rules. Information Theory, IEEE Transactions on , IT-25:601 -- 604, 10 1979
1979
-
[13]
Elisseeff, T
A. Elisseeff, T. Evgeniou, and M. Pontil. Stability of randomized learning algorithms. Journal of Machine Learning Research , 6(3):55--79, 2005
2005
-
[14]
Farhadkhani, R
S. Farhadkhani, R. Guerraoui, N. Gupta, and R. Pinot. Brief announcement: a case for byzantine machine learning. In Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing , pages 131--134, 2024
2024
-
[15]
Farhadkhani, R
S. Farhadkhani, R. Guerraoui, N. Gupta, and R. Pinot. On the relevance of byzantine robust optimization against data poisoning. arXiv preprint arXiv:2405.00491 , 2024
2024 arXiv
-
[16]
Farhadkhani, R
S. Farhadkhani, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan. B yzantine machine learning made easy by resilient averaging of momentums. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on...
2022
-
[17]
Farhadkhani, R
S. Farhadkhani, R. Guerraoui, O. Villemaud, et al. An equivalence between data poisoning and byzantine gradient attacks. In International Conference on Machine Learning , pages 6284--6323. PMLR, 2022
2022
-
[18]
Goldwasser, S
S. Goldwasser, S. Micali, and C. Rackoff. The knowledge complexity of interactive proof systems. SIAM Journal on Computing , 18(1):186--208, 1989
1989
-
[19]
Gorbunov, S
E. Gorbunov, S. Horv \'a th, P. Richt \'a rik, and G. Gidel. Variance reduction is an antidote to byzantines: Better rates, weaker assumptions and communication compression as a cherry on the top. In The Eleventh International Conference on Learning Representations , 2023
2023
-
[20]
Guerraoui, N
R. Guerraoui, N. Gupta, and R. Pinot. Byzantine machine learning: A primer. ACM Comput. Surv. , 56(7), 2024
2024
-
[21]
Guerraoui, N
R. Guerraoui, N. Gupta, and R. Pinot. Robust Machine Learning: Distributed Methods for Safe AI . Machine Learning: Foundations, Methodologies, and Applications Series. Springer Nature Singapore, Imprint: Springer, 2024
2024
-
[22]
Guerraoui, S
R. Guerraoui, S. Rouault, et al. The hidden vulnerability of distributed learning in byzantium. In International conference on machine learning , pages 3521--3530. PMLR, 2018
2018
-
[23]
Hardt, B
M. Hardt, B. Recht, and Y. Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225--1234. PMLR, 2016
2016
-
[24]
Kairouz, H
P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. D’Oliveira, H. Eichner, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. He, ...
2021
-
[25]
S. P. Karimireddy, L. He, and M. Jaggi. Learning from history for byzantine robust optimization. In International Conference on Machine Learning , pages 5311--5319. PMLR, 2021
2021
-
[26]
S. P. Karimireddy, L. He, and M. Jaggi. Byzantine-robust learning on heterogeneous datasets via bucketing. In International Conference on Learning Representations , 2022
2022
-
[27]
Lamport, R
L. Lamport, R. Shostak, and M. Pease. The byzantine generals problem. ACM Transactions on Programming Languages and Systems , 4(3):382--401, 1982
1982
-
[28]
Le Bars, A
B. Le Bars, A. Bellet, M. Tommasi, K. Scaman, and G. Neglia. Improved stability and generalization guarantees of the decentralized SGD algorithm. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st I...
2024
-
[29]
Lei and Y
Y. Lei and Y. Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In International Conference on Machine Learning , pages 5809--5819. PMLR, 2020
2020
-
[30]
S. Liu, N. Gupta, and N. H. Vaidya. Approximate byzantine fault-tolerance in distributed optimization. In Proceedings of the 2021 ACM Symposium on Principles of Distributed Computing , pages 379--389, 2021
2021
-
[31]
Lugosi and G
G. Lugosi and G. Neu. Generalization bounds via convex analysis. In Conference on Learning Theory , pages 3524--3546. PMLR, 2022
2022
-
[32]
D. A. McAllester. Pac-bayesian model averaging. Machine Learning , 37(3):275--299, 1999
1999
-
[33]
Rogers and T
W. Rogers and T. Wagner. A finite sample distribution-free performance bound for local discrimination rules. The Annals of Statistics , 6, 05 1978
1978
-
[34]
Russo and J
D. Russo and J. Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory , 66(1):302--323, 2019
2019
-
[35]
Sabater, A
C. Sabater, A. Bellet, and J. Ramon. A n A ccurate, S calable and V erifiable P rotocol for F ederated D ifferentially P rivate A veraging. Machine Learning , 111:4249--4293, 2022
2022
-
[36]
Shalev-Shwartz, O
S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research , 11(90):2635--2670, 2010
2010
-
[37]
A. S. Shamsabadi, G. Tan, T. I. Cebere, A. Bellet, H. Haddadi, N. Papernot, X. Wang, and A. Weller. C onfidential- DP proof: C onfidential P roof of D ifferentially P rivate T raining. In ICLR , 2024
2024
-
[38]
Shejwalkar and A
V. Shejwalkar and A. Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In NDSS , 2021
2021
-
[39]
Z. Sun, X. Niu, and E. Wei. Understanding generalization of federated learning via stability: Heterogeneity matters. In International Conference on Artificial Intelligence and Statistics , pages 676--684. PMLR, 2024
2024
-
[40]
J. Thaler. Proofs, arguments, and zero-knowledge. Found. Trends Priv. Secur. , 4(2-4):117--660, 2022
2022
-
[41]
Vapnik and A
V. Vapnik and A. Chervonenkis. Theory of Pattern Recognition [in Russian] . Nauka, Moscow, 1974. (German Translation: W. Wapnik & A. Tscherwonenkis, Theorie der Zeichenerkennung , Akademie--Verlag, Berlin, 1979)
1974
-
[42]
V. N. Vapnik. Statistical learning theory . Wiley-interscience, 1998
1998
-
[43]
C. Xie, O. Koyejo, and I. Gupta. Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation. In Uncertainty in Artificial Intelligence , pages 261--270. PMLR, 2020
2020
-
[44]
Ye and Q
H. Ye and Q. Ling. Generalization error matters in decentralized learning under byzantine attacks. IEEE Transactions on Signal Processing , 73:843--857, 2025
2025
-
[45]
H. Ye, T. Sun, and Q. Ling. Generalization guarantee of decentralized learning with heterogeneous data. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1--5, 2025
2025
-
[46]
H. Ye, H. Zhu, and Q. Ling. On the tradeoff between privacy preservation and byzantine-robustness in decentralized learning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 9336--9340, 2024
2024
-
[47]
D. Yin, Y. Chen, R. Kannan, and P. Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International conference on machine learning , pages 5650--5659. Pmlr, 2018
2018
-
[48]
B. Zhu, L. Wang, Q. Pang, S. Wang, J. Jiao, D. Song, and M. I. Jordan. Byzantine-robust federated learning with optimal statistical rates and privacy guarantees, 2023
2023
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.