REVIEW 3 major objections 6 minor 39 references
Robust Score Matching
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Score matching can be made robust to corrupted rows by replacing means with a geometric median of means.
desk verdict Solid robust score matching paper with a real gap: consistency is proved only for sublinear contamination, while the experiments and tuning heuristic target a fixed proportion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The geometric median of means (GMoM) partitions the sample into $K$ blocks, takes the mean of each block, and then computes the geometric median of those block means. The geometric median is a convex combination of its arguments, so applying the GMoM to positive semidefinite matrices keeps the resulting matrix positive semidefinite; this preserves convexity of the score matching objective and guarantees the existence of the estimator. The method inherits a breakdown point near $1/2$, and the block-size parameter $K$ tunes the trade-off between robustness, bias, and variance.
What would settle it
Simulate data from a square root graphical model with a fixed 5% of rows replaced by arbitrary outliers and compute the GMoM score matching estimator at increasing sample sizes $n = 10^3, 10^4, 10^5$. If the squared error to the true parameter does not shrink to zero as $n$ grows, the consistency claim does not hold in the fixed-proportion contamination regime.
Extended reading notes
Core claim
The central claim is that replacing the score matching statistics $\Gamma(X)$ and $g(X)$ by their geometric-median-of-means versions, $\hat\Gamma_K(X)$ and $\hat g_K(X)$, yields an estimator $\hat\theta(K) = \hat\Gamma_K(X)^{-1}\hat g_K(X)$ that converges in probability to the true parameter $\theta_0$ when at most $n_c = o(n)$ observations are arbitrarily corrupted. With an $\ell^1$ penalty and under an irrepresentability condition, the regularized estimator has support contained in the true support and satisfies $\|\hat\theta(K, \beta, \lambda) - \theta_0\|_\infty \leq c_{\Gamma_0}(2-\alpha)\lambda$ with probability at least $1 - 2\delta$.
Load-bearing premise
The proof only allows the number of corrupted rows to grow like $O(\log(1/\delta))$, so the contamination must be a vanishing fraction of the sample; a fixed positive fraction of corrupted rows is not covered by the consistency theorem.
Editorial extensions
If this is right
- Score matching can be extended to contaminated data while keeping the convenience of a convex objective in exponential family models.
- The estimator is consistent when the number of corrupted rows grows slower than the sample size, even if the corruption is adversarial.
- For sparse graphical models, the regularized robust estimator provably recovers the true conditional independence structure under contamination.
- In numerical experiments, the robust estimator matches the standard one on clean data and substantially outperforms it when data are contaminated, including on a real precipitation network.
Reading between the lines
- The formal consistency result only covers a vanishing fraction of corrupted rows, while the experiments use a fixed 5% or 10% contamination; the fixed-proportion regime is supported empirically but not by the main theorem.
- The same GMoM construction could robustify other moment-based estimators that require averaging positive semidefinite matrices, since the convex-combination property preserves definiteness.
- A natural next test is whether a minimax optimal choice of block size $K$ can be derived for fixed contamination proportions, rather than the heuristic $K = 4\varepsilon n$ used in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a robustified score matching estimator for exponential families by replacing the empirical moment quantities Γ(X) and g(X) with geometric median of means (GMoM) aggregates. The resulting objective remains convex in exponential family models. The main theoretical results are a concentration inequality for the GMoM under rowwise corruption (Theorem 3.1), consistency of the robust score matching estimator when the number of corrupted samples is sublinear (Corollary 3.2), and an ℓ1-regularized support recovery guarantee for pairwise interaction models (Theorem 4.3). Numerical experiments on square root graphical models and a precipitation dataset show improved support recovery relative to classical score matching under 5–20% contamination.
Significance. If the advertised claims held for the contamination model in Section 3.1, this would be a valuable extension of score matching to adversarial rowwise corruption: the retention of convexity is a real advantage over naive robustification, and support recovery in non-Gaussian graphical models is of substantial interest. The paper is carefully written, with detailed proofs, a thoughtful discussion of multivariate median properties, and a useful empirical study including a non-synthetic application. The main caveat is that the theoretical guarantees are established only for sublinearly many corrupted samples, whereas the paper's problem statement and experiments use a fixed contamination proportion; the contribution is therefore more modest than the abstract suggests unless that gap is closed.
major comments (3)
- [3.1, Theorem 3.1, Corollary 3.2] The contamination model in Section 3.1 allows a fixed proportion ε of rows to be altered arbitrarily, and the simulations in Section 5.1 use ε=5%. However, Theorem 3.1 permits only up to (⌊17 log(1/δ)⌋+1)τ corrupted samples, which for fixed confidence δ is O(1), and Corollary 3.2 obtains consistency only in the regime n_c=o(n). If one sets n_c=ε n with ε>0 fixed, the proof's construction δ=exp(−(⌈n_c/τ⌉−1)/17) yields log(1/δ)=Θ(n), so the error bound in (11) is Θ(1) and no consistency in probability follows; additionally, the required K grows linearly in n, whereas the theorem's framework for fixed δ has K=O(log(1/δ)). The abstract's claim of consistent estimation under contaminated data is therefore not established for the paper's own contamination model. The authors should either restrict the consistency claim to n_c=o(n) and state this limitation prominently, or supply a fixed-ε concentration argument, which would need to address the fact that the GMoM is biased for the mean when K is large.
- [3.4, 5.1] The block-size rule K:=4εn used in all experiments is linear in n, while the theoretical results require K=O(log(1/δ)) for a fixed confidence level δ (and K≤n/2). Consequently the empirical configuration K=n/5 for ε=0.05 is outside the scope of Theorem 3.1 and Theorem 4.3. The heuristic is motivated only by breakdown-point considerations, not by the concentration theory; the paper should explicitly state that the experiments are not covered by the theoretical guarantees.
- [Theorem 4.3] The support recovery guarantee has the same sublinear-corruption limitation. The statement allows n_c:=τ(⌊17 log(1/δ)⌋+1) corrupted samples, so for a fixed confidence δ this number is O(1) and independent of n. In the fixed-proportion regime n_c=ε n, the theorem can be applied only with δ exponentially small in n; then the lower bound on λ in (16) is of constant order, and the ℓ∞ error bound (17) does not vanish. The discussion following the theorem presents this as a consistency-style guarantee, but for fixed ε the conclusion reduces to a constant-error containment statement rather than convergence to θ0.
minor comments (6)
- [3.4] The sentence 'the breakdown point is exceeded when K ≥ 2ε' is dimensionally inconsistent; it should read K ≥ 2ε n.
- [Appendix A] The positive-definiteness argument for Γ̂K uses the convex-combination representation (8), which is valid only when the geometric median does not coincide with one of the block means; the K=1 case and the coincident case should be handled separately for completeness.
- [2.2] There is a typo: 'subleties' should be 'subtleties'.
- [Appendix B] In the simplification of the exponent, the phrase 'First term of ψ negative' is cryptic; the argument would be clearer if the sign of the relevant term were shown explicitly.
- [Appendix E] In the statement of Theorem E.2, 'he minimizer' should be 'the minimizer'.
- [Figure 1] The caption states that pointwise uncertainty in TPR is at most ±0.75% based on 100–500 simulations; please clarify whether this is a 95% bootstrap band for the average ROC and why the reported uncertainty is so small.
Circularity Check
No significant circularity: robust consistency comes from an external median-of-means lemma; self-citations to Yu et al. (2019) are prior published theorems used as building blocks, not as the source of the robustness claim.
full rationale
The paper's main robustness claim is not circular. The estimator in (10) is defined by replacing the empirical matrix Gamma(X) and vector g(X) in the classical score matching loss with their GMoM counterparts in (9); no parameter is fitted to the quantity that is later 'predicted' (the true theta_0). Theorem 3.1 is proved in Appendix B from Minsker's geometric median concentration lemma (Lemma B.1), an external result, and Corollary 3.2 follows by choosing delta(n_c) = exp(-(ceil(n_c/tau_0)-1)/17) and invoking continuity of matrix inversion together with the standard score matching identifiability result (Yu et al. 2019, Prop. 2). The support recovery theorem (Theorem 4.3) is modular: Appendix E shows concentration of the robust Gamma and g estimates (Lemma E.1 plus Theorem 3.1) and then applies Yu et al. (2019, Theorem E.2) as a black box. That citation is a self-citation (M. Drton is a coauthor of Yu et al. 2019), but it is independent support: the cited theorem is a peer-reviewed, parameter-free statement whose assumptions (irrepresentability and closeness of Gamma and g estimates) do not include the GMoM estimator, so the robust claim does not reduce by definition to the citation. The choice K = 4*epsilon*n is explicitly presented as a heuristic validated by simulations (Sections 3.4 and Appendix D), not as a theoretical prediction. The main weakness is a scope gap, not circularity: Section 3.1 and the experiments use a fixed contamination proportion epsilon, while Theorem 3.1 and Corollary 3.2 require n_c = o(n) corrupted rows; this mismatch affects whether the consistency guarantee applies to the paper's own simulation regime, but it is not a circular derivation. For these reasons no step in the derivation is equivalent to its own input.
Assumptions & free parameters
free parameters (3)
- K (number of GMoM blocks) =
K = 4*epsilon*n (heuristic; in simulations 200 for n=1000, 5% contamination)
- beta (diagonal multiplier) =
0 for n=1000; about 0.01 for n=80 (upper bound of Theorem 4.3 with Gamma_0 estimated from uncorrupted data)
- lambda (l1 penalty) =
Varied over a grid to produce ROC curves
assumptions (6)
- domain assumption Exponential family regularity conditions (A1) and (A2) from Appendix A, plus the conditions of Proposition 2 in Yu et al. (2019) for the score matching loss to identify theta_0.
- domain assumption Finite second moments of Gamma(x) and g(x) under theta_0.
- domain assumption Irrepresentability condition with incoherence parameter alpha in (0,1] for Gamma_0.
- domain assumption The contamination model allows at most tau*(floor(17 log(1/delta)) + 1) rows corrupted arbitrarily, with tau < 1/2.
- ad hoc to paper Known contamination proportion epsilon (or a bound on the number of corruptions) to choose K.
- standard math Minsker's geometric median concentration lemma (Lemma B.1).
Cite this review
Pith. "Pith review of Robust Score Matching." pith.science (2026). https://pith.science/paper/OODNS4NS
@misc{pith2026250105105,
author = {Pith},
title = {Pith review of: Robust Score Matching},
year = {2026},
howpublished = {\url{https://pith.science/paper/OODNS4NS}},
note = {Machine review of arXiv:2501.05105}
}
read the original abstract
Proposed in Hyv\"arinen (2005), score matching is a parameter estimation procedure that does not require computation of distributional normalizing constants. In this work we utilize the geometric median of means to develop a robust score matching procedure that yields consistent parameter estimates in settings where the observed data has been contaminated. A special appeal of the proposed method is that it retains convexity in exponential family models. The new method is therefore particularly attractive for non-Gaussian, exponential family graphical models where evaluation of normalizing constants is intractable. Support recovery guarantees for such models when contamination is present are provided. Additionally, support recovery is studied in numerical experiments and on a precipitation dataset. We demonstrate that the proposed robust score matching estimator performs comparably to the standard score matching estimator when no contamination is present but greatly outperforms this estimator in a setting with contamination.
Reference graph
Works this paper leans on
-
[1]
barticle Alqallaf , F. , Van Aelst , S. , Yohai , V.J. , Zamar , R.H. : Propagation of outliers in multivariate data . Ann. Statist. 37 ( 1 ), 311 -- 331 ( 2009 ) barticle
work page 2009
-
[2]
bchapter Bhatt , S. , Fang , G. , Li , P. , Samorodnitsky , G. : Minimax m-estimation under adversarial contamination . In: Proceedings of the 39th International Conference on Machine Learning . Proceedings of Machine Learning Research , vol. 162 , pp. 1906 -- 1924 . PMLR , New York, USA ( 2022 ) bchapter
work page 1906
- [3]
-
[4]
: Statistical uses of the spatial median
barticle Brown , B.M. : Statistical uses of the spatial median . J. Roy. Statist. Soc. Ser. B 45 ( 1 ), 25 -- 30 ( 1983 ) barticle
work page 1983
-
[5]
barticle Chu , J.-h. , Weiss , S.T. , Carey , V.J. , Raby , B.A. : A graphical model approach for inferring large-scale networks integrating gene expression and genetic polymorphism . BMC Systems Biology 3 , 1 -- 9 ( 2009 ) barticle
work page 2009
-
[6]
barticle Donoho , D.L. , Gasko , M. : Breakdown properties of location estimates based on halfspace depth and projected outlyingness . Ann. Statist. 20 ( 4 ), 1803 -- 1827 ( 1992 ) barticle
work page 1992
-
[7]
bbook Diakonikolas , I. , Kane , D.M. : Algorithmic High-Dimensional Robust Statistics . Cambridge University Press , Cambridge ( 2023 ) bbook
work page 2023
-
[8]
barticle Devroye , L. , Lerasle , M. , Lugosi , G. , Oliveira , R.I. : Sub- G aussian mean estimators . Ann. Statist. 44 ( 6 ), 2695 -- 2725 ( 2016 ) barticle
work page 2016
Show all 39 references
-
[9]
, Perlman , M.D
barticle Eaton , M.L. , Perlman , M.D. : The non-singularity of generalized sample covariance matrices . Ann. Statist. 1 , 710 -- 717 ( 1973 ) barticle
1973
-
[10]
: An introduction to ROC analysis
barticle Fawcett , T. : An introduction to ROC analysis . Pattern Recognition Letters 27 ( 8 ), 861 -- 874 ( 2006 ) barticle
2006
-
[11]
, Hastie , T
barticle Friedman , J. , Hastie , T. , H\" o fling , H. , Tibshirani , R. : Pathwise coordinate optimization . Ann. Appl. Stat. 1 ( 2 ), 302 -- 332 ( 2007 ) barticle
2007
-
[12]
, Hastie , T
barticle Friedman , J. , Hastie , T. , Tibshirani , R. : Sparse inverse covariance estimation with the graphical lasso . Biostatistics 9 ( 3 ), 432 -- 441 ( 2007 ) barticle
2007
-
[13]
: Estimation of non-normalized statistical models by score matching
barticle Hyv\" a rinen , A. : Estimation of non-normalized statistical models by score matching . J. Mach. Learn. Res. 6 , 695 -- 709 ( 2005 ) barticle
2005
-
[14]
: Some extensions of score matching
barticle Hyv\" a rinen , A. : Some extensions of score matching . Comput. Statist. Data Anal. 51 ( 5 ), 2499 -- 2512 ( 2007 ) barticle
2007
-
[15]
, Ravikumar , P
bchapter Inouye , D. , Ravikumar , P. , Dhillon , I. : Square root graphical models: Multivariate generalizations of univariate exponential families that permit positive dependencies . In: Proceedings of The 33rd International Conference on Machine Learning . Proceedings of Ma...
2016
-
[16]
, Shafiee , S
botherref Kuhn , D. , Shafiee , S. , Wiesemann , W. : Distributionally Robust Optimization (2024). https://arxiv.org/abs/2411.02549 botherref
2024 arXiv
-
[17]
: Graphical Models
bbook Lauritzen , S.L. : Graphical Models . Oxford Statistical Science Series , vol. 17 , p. 298 . The Clarendon Press, Oxford University Press , New York, USA ( 1996 ) bbook
1996
-
[18]
, Drton , M
barticle Lin , L. , Drton , M. , Shojaie , A. : Estimation of high-dimensional graphical models using regularized score matching . Electron. J. Stat. 10 ( 1 ), 806 -- 854 ( 2016 ) barticle
2016
-
[19]
, Mendelson , S
barticle Lugosi , G. , Mendelson , S. : Mean estimation and regression under heavy-tailed distributions: A survey . Foundations of Computational Mathematics 19 ( 5 ), 1145 -- 1190 ( 2019 ) barticle
2019
-
[20]
, Mosler , K
barticle Liu , X. , Mosler , K. , Mozharovskyi , P. : Fast computation of T ukey trimmed regions and median in dimension p>2 . J. Comput. Graph. Statist. 28 ( 3 ), 682 -- 697 ( 2019 ) barticle
2019
-
[21]
, Rousseeuw , P.J
barticle Lopuha\" a , H.P. , Rousseeuw , P.J. : Breakdown points of affine equivariant estimators of multivariate location and covariance matrices . Ann. Statist. 19 ( 1 ), 229 -- 248 ( 1991 ) barticle
1991
-
[22]
, Staerman , G
bchapter Laforgue , P. , Staerman , G. , Cl \'e men c on , S. : Generalization bounds in the presence of outliers: a median-of-means study . In: Proceedings of the 38th International Conference on Machine Learning . Proceedings of Machine Learning Research , vol. 139 , pp. 593...
2021
-
[23]
, Tan , X.L
barticle Loh , P.-L. , Tan , X.L. : High-dimensional robust precision matrix estimation: Cellwise corruption under -contamination . Electronic Journal of Statistics 12 ( 1 ), 1429 -- 1467 ( 2018 ) barticle
2018
-
[24]
, Drton , M
bbook Maathuis , M. , Drton , M. , Lauritzen , S. , Wainwright , M. (eds.): Handbook of Graphical Models . Chapman & Hall/CRC Handbooks of Modern Statistical Methods , p. 536 . CRC Press , Boca Raton, FL ( 2019 ) bbook
2019
-
[25]
: Geometric median and robust estimation in B anach spaces
barticle Minsker , S. : Geometric median and robust estimation in B anach spaces . Bernoulli 21 ( 4 ), 2308 -- 2335 ( 2015 ) barticle
2015
-
[26]
: Distributed statistical estimation and rates of convergence in normal approximation
barticle Minsker , S. : Distributed statistical estimation and rates of convergence in normal approximation . Electron. J. Stat. 13 ( 2 ), 5213 -- 5252 ( 2019 ) barticle
2019
-
[27]
, Martin , R.D
bbook Maronna , R.A. , Martin , R.D. , Yohai , V.J. , Salibi\' a n-Barrera , M. : Robust Statistics , 2 nd edn. Wiley Series in Probability and Statistics , p. 430 . John Wiley & Sons, Inc. , Hoboken, NJ ( 2019 ) bbook
2019
-
[28]
, Oja , H
barticle Niinimaa , A. , Oja , H. , Tableman , M. : The finite-sample breakdown point of the O ja bivariate median and of the corresponding half-samples version . Statist. Probab. Lett. 10 ( 4 ), 325 -- 328 ( 1990 ) barticle
1990
-
[29]
, Deasy , J.O
barticle Oh , J.H. , Deasy , J.O. : Inference of radio-responsive gene regulatory networks using the graphical lasso algorithm . BMC Bioinformatics 15 , 1 -- 8 ( 2014 ) barticle
2014
-
[30]
, Dunson , D.B
barticle Roy , A. , Dunson , D.B. : Nonparametric graphical model for counts . Journal of Machine Learning Research 21 ( 229 ), 1 -- 21 ( 2020 ) barticle
2020
-
[31]
, Oja , H
bchapter Ronkainen , T. , Oja , H. , Orponen , P. : Computation of the multivariate O ja median . In: Developments in Robust Statistics ( V orau, 2001) , pp. 344 -- 359 . Physica , Heidelberg ( 2003 ) bchapter
2001
-
[32]
: Multivariate symmetry and asymmetry
barticle Serfling , R.J. : Multivariate symmetry and asymmetry . Encyclopedia of Statistical Sciences 8 , 5338 -- 5345 ( 2006 ) barticle
2006
-
[33]
, Kolar , M
bchapter Sun , S. , Kolar , M. , Xu , J. : Learning structured densities via infinite dimensional exponential families . In: Advances in Neural Information Processing Systems , vol. 28 . Curran Associates, Inc. , New York, USA ( 2015 ) bchapter
2015
-
[34]
: A survey of multidimensional medians
barticle Small , C.G. : A survey of multidimensional medians . International Statistical Review / Revue Internationale de Statistique 58 ( 3 ), 263 -- 277 ( 1990 ) barticle
1990
-
[35]
barticle Tank , K. , A.M.G. , Coauthors : Daily dataset of 20th-century surface air temperature and precipitation series for the european climate assessment . International Journal of Climatology 22 , 1441 -- 1453 ( 2002 ) barticle
2002
-
[36]
, Zhang , C.-H
barticle Vardi , Y. , Zhang , C.-H. : A modified W eiszfeld algorithm for the F ermat- W eber location problem . Math. Program. 90 ( 3 ), 559 -- 566 ( 2001 ) barticle
2001
-
[37]
, Drton , M
barticle Yu , S. , Drton , M. , Shojaie , A. : Generalized score matching for non-negative data . J. Mach. Learn. Res. 20 ( 1 ), 2779 -- 2848 ( 2019 ) barticle
2019
-
[38]
, Kolar , M
bchapter Yu , M. , Kolar , M. , Gupta , V. : Statistical inference for pairwise graphical models using score matching . In: Advances in Neural Information Processing Systems , vol. 29 . Curran Associates, Inc. , New York, USA ( 2016 ) bchapter
2016
-
[39]
write newline
" write newline "" before.all 'output.state := FUNCTION string.to.integer 't := t text.length 'k := #1 'char.num := t char.num #1 substring 's := s is.num s "." = or char.num k = not and char.num #1 + 'char.num := while char.num #1 - 'char.num := t #1 char.num substring FUNCTI...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.