Pith. sign in

REVIEW 4 major objections 7 minor 70 references

Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization

T0 review · 4 major / 7 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dense SymNMF becomes practical at a million rows once n-by-n intermediates are removed, and five AdaGrad-family solvers alone still converge there.

desk verdict Solid large-scale SymNMF systems paper: five AdaGrad-family methods really do reach n=10^6 under a shared KKT gate, but the "fastest tracks the spectrum" rule is confounded and should not be the takeaway you remember. read the letter →

arxiv 2607.24518 v1 pith:I4YNRAG4 submitted 2026-07-27 cs.LG cs.NAmath.NA

Lavinia Ghita , Dhruv Desai , Jake Goldberg , Roman Yokunda Enzmann This is my paper · ORCID
classification cs.LGcs.NAmath.NA
keywords symmetricNMFAdaGradtailpairwisedependencecorrelationmatricesGPUscalingriskfactorssphericalK-meansprojectedgradient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Symmetric non-negative matrix factorization recovers soft group structure from a dependence matrix, but the usual dense objective has kept it at moderate size. This paper shows that a simple trace identity removes every n-by-n temporary, so one GPU reaches about 100,000 instruments and multi-node runs reach a million. Across seven algorithm families on absolute correlation and extreme-value tail-dependence matrices, only a handful of adaptive first-order methods remain reliable at that scale: plain AdaGrad, RMSprop, and three engineered variants. Which one is fastest depends on the spectrum—full-batch AdaGrad on short, gapped correlation problems, and a block stochastic growing-batch method on long, flat tail problems. Hard spherical K-means is cheaper when angular clusters exist, but collapses when a single common factor dominates, so the soft factorization stays necessary for risk-factor work on large portfolios.

What carries the argument

The trace-identity loss ||S−HHᵀ||_F² = ||S||_F² − 2 tr(HᵀSH) + ||HᵀH||_F², which evaluates the objective and gradient using only n×k and k×k products and thereby doubles single-GPU capacity and enables multi-node scaling.

What would settle it

Rerun the n=10^6 Phase-2 bake-off on real absolute-correlation and empirical tail-dependence matrices from a large traded universe whose effective rank and spectral gaps do not match the synthetic generator; if the AdaGrad-family frontier or the spectrum-dependent winner flips, the central scaling claim fails to transfer.

Watch

Extended reading notes

Core claim

With the trace-reformulated loss, five AdaGrad-family methods converge under a uniform projected-gradient KKT gate at n=10^6. At that scale the winner tracks the matrix spectrum: Block-SVRG AdaptGrow is fastest on flat, ill-conditioned tail-dependence matrices, while full-batch AdaGrad is fastest on dominant-low-rank correlation matrices.

Load-bearing premise

The large-scale ranking rests on synthetic block factor matrices with a fixed sample size and a fixed tail exceedance count, so the hard tail spectra partly reflect that artificial rank ceiling rather than real growing-sample portfolios.

Editorial extensions

If this is right

  • Portfolio risk-factor models that need soft loadings on ~10^6 names can use dense SymNMF on multi-GPU clusters instead of being limited to moderate n.
  • Solver choice can be spectrum-driven: full-batch AdaGrad for short gapped correlation runs, Block-SVRG AdaptGrow for long flat tail runs.
  • When only hard labels are required and angular structure is present, spherical K-means on the rows of S is the cheaper default; once effective rank nears one, soft factorization is required.
  • Long-memory adaptive methods should stay in projected (clamped) space at scale; softplus helps mainly short-memory or small-n second-order methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same trace reformulation and AdaGrad-family stack likely transfers to other dense symmetric Gram-style objectives (kernel NMF, graph Laplacians) whose bottleneck is an n×n residual.
  • Streaming or rolling-window SymNMF for live risk systems is a direct next engineering step once the batch n=10^6 path is stable.
  • If real market tail matrices keep growing effective rank with sample size, the TPDM-side advantage of cheap block steps may shrink and full-batch AdaGrad may win more often.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents a large-scale GPU study of symmetric non-negative matrix factorization (SymNMF) on two dependence-matrix types — absolute Pearson correlation and tail pairwise dependence matrices (TPDM) from extreme value theory — from n=10^2 to n=10^6. A trace-identity reformulation eliminates all n×n intermediates, roughly doubling single-GPU capacity; seven solver families (30+ configurations) are screened under a uniform three-criterion KKT gate (loss gate, per-element projected-gradient residual, stagnation). Five AdaGrad-family methods converge at n=10^6, three of them introduced here (Piecewise AdaGrad, Row-Stochastic SVRG, Block-SVRG AdaptGrow). The headline finding is that at n=10^6 the fastest solver tracks the matrix spectrum: Block-SVRG AdaptGrow wins on the flat TPDM spectrum while full-batch AdaGrad wins on the gapped correlation spectrum. The paper also gives clean algebraic equivalences between orthogonality-constrained SymNMF and (spherical) K-means (Theorems 6.1–6.2), and proves a spectral degeneracy bound (Eq. 14) showing spherical K-means collapses as effective rank approaches one.

Significance. If the results hold, this is a useful empirical contribution: it is, to my knowledge, the first systematic SymNMF solver comparison at n=10^6 with controlled ground truth, and it ships several genuinely valuable elements — a uniform externally-defined convergence certificate (projected-gradient KKT residual, not a quantity derived from the fitted H), explicit spectral diagnostics (Table 3) tied to optimization difficulty, multi-seed runs, matched-hardware wall-clock comparisons, and an unusually honest accounting of failed approaches (Appendix D on entry sampling, sketching, dual averaging). The trace-identity reformulation is simple but the capacity analysis (Table 1) and the fp32/TF32 precision discussion are careful. The spectral-selection rule, if validated, would be a practical and falsifiable guideline. The main limitation is that the three new optimizers are empirical with no rate or regret analysis (the authors state this plainly), and the central spectral-selection claim rests on a confounded comparison, detailed in the major comments.

major comments (4)
  1. [§5.3, Table 6 (also Abstract and §8)] The headline claim — 'at n=10^6 the fastest solver tracks the matrix spectrum' — rests on a single cell pair (CORR vs TPDM at n=10^6) in which spectrum is confounded with at least three other variables. (i) Rank: CORR uses k=1000, TPDM uses k=400 (§5.3), which the paper itself calls 'distinct problems.' (ii) Problem nature: the empirical TPDM is a normalized Gram matrix with rank ≤ n_exc=400 (§3.4), so at k=400 an essentially exact factorization exists (E_t→0) and the Hessian is degenerate along zero-eigenvalue directions; every other cell in the paper is an approximation problem with a positive residual floor. The 'long factorization where cheap per-iteration cost decides' therefore occurs in an exact-fit regime, not in a generic flat-spectrum approximation problem. (iii) Certification: the TPDM n=10^6 cell alone is certified at τ_g=10^-3 rather than 10^-4 (§4.1). Any of these could pla
  2. [§4.1 (Criterion 2) and §2.1] The relaxation of the KKT gate to τ_g=10^-3 for TPDM at n=10^6 is justified as a 'numerical floor, not a weaker standard,' but the floor is asserted, not measured. The fp32 cancellation analysis of §2.1 is developed for the loss (relative error ~2ε/E_t in the three-term trace identity); the gradient is formed as 4(H(H⊤H)−SH), whose cancellation structure near exact fit is different — H(H⊤H) and SH both approach SH and nearly cancel entrywise — and is never quantified. A one-off fp64 (or fp32-with-fp64-accumulation) reference run at n=10^6 TPDM, or even at n=10^5 with k=n_exc, would directly measure the achievable projected-gradient floor and either substantiate or revise the 10^-3 choice. This is load-bearing because the relaxed gate applies only to the cell on which the headline AdaptGrow win is reported.
  3. [§3.4, §4.2, Appendix B] The TPDM benchmark fixes n_sample=40,000 and q=0.01, so n_exc=400 regardless of n. The paper is transparent that the resulting rank ceiling is 'an artifact of the fixed-exceedance generator, not of empirical TPDMs whose rank grows with the sample' (§4.2), but this acknowledgment does not propagate to the claims: the Abstract and §8 present the Phase-2 ranking and the spectrum-selection rule without qualification, and these are precisely the cells where the artifact bites (at n=10^6, k=1000 > n_exc forces the over-parameterized exact-fit regime; Table 3 footnote ‡). At minimum the abstract/conclusion should state that large-n TPDM hardness here mixes spectral flattening with a fixed-rank ceiling. Ideally, one experiment with n_exc growing with n (e.g., n_exc ∝ √n or a larger q at n=10^5) would show whether the TPDM-side ranking survives when the rank ceiling is removed.
  4. [§5.3, Table 7] The per-cell hyperparameters in Table 7 vary enormously across cells (e.g., AdaGrad η=96 at n=10^5 CORR vs η=384 at n=10^6 CORR; Piecewise η drops from 128 to 0.75), and the tuning protocol is described only as 'best local sweep around the Phase 1 anchor.' The sweep grid sizes, the selection criterion (fastest converged? best E_t?), and the sensitivity of the Table 6 orderings to this tuning are not reported. Since the headline n=10^6 ordering is a 4.0 vs 4.8 minute margin (~20%), and RMSprop requires α=0.999 specifically in that cell, the robustness of the ordering to equal tuning effort across solvers should be demonstrated — e.g., a small sensitivity table showing wall-clock over the local η grid for the two leading solvers in each n=10^6 cell.
minor comments (7)
  1. [§1 and §2.2.3] The phrase 'retaining universal convergence' (§1) for the stochastic AdaGrad variants overstates what is shown: these are explicitly 'empirical engineering methods' with no convergence proof (§2.2.3). 'Universal' here means 'converged on all benchmark runs'; please qualify.
  2. [Footnote 2 and §3.2–3.3] Notation collision: H denotes both the spectral (angular) measure (dH(w), H_X, H_AZ throughout §3.2–3.3 and Appendix A) and the SymNMF factor. Footnote 2 says the intention was to reserve H for the factor, but the angular measure is still written H. Consider renaming the angular measure (e.g., Υ or Γ) for clarity.
  3. [§4.1, Eq. (10)] The scaled threshold τ_g(n) relaxes the KKT gate to 10^-2 at n=10^2 and 10^-3 at n=10^3, so 'convergence' in Table 4 means different things across sizes. This is disclosed but should be flagged in the Table 4 caption, since convergence rates at n=10^2–10^3 are not directly comparable to those at n=10^4.
  4. [§5.2] The memory-length ordering claim (softplus corrupts long-memory accumulators) is supported by the Table 5 pattern but, as the authors note, no controlled β2/α sweep was run. A one-dimensional sweep of Adam's β2 at n=10^4 would substantially strengthen this mechanistic claim at low cost.
  5. [§7] The decision not to run a wall-clock head-to-head against PLANC/MPI-FAUN on CPU is reasonable, but a rough cross-check at one scale (e.g., n=10^4 or 10^5 against published PLANC timings) would help readers place the GPU results; at minimum, state explicitly that no such comparison was attempted.
  6. [Table 4] Please state the iteration/time budget that defines a failed run, and note that wall-clock means are taken over converged runs only (survivorship is mild here given 18/18 rates, but MU/APG rows mix 15/18 convergence with mean times).
  7. [§6.2, Table 8] The K-means comparison uses a weaker stopping rule (objective stagnation only), which the text acknowledges; consider also reporting the silhouette of the SymNMF-argmax partition at n=10^6 TPDM to make the 'soft factorization remains necessary' claim quantitative rather than only spectral (Eq. 14).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical solver study with external KKT certification and restated known clustering equivalences.

full rationale

This paper is a large-scale GPU benchmarking and engineering study of SymNMF solvers, not a first-principles derivation that claims to predict observables from fitted constants. Convergence is certified by a projected-gradient KKT residual, a loss gate, and objective stagnation (Eqs. 8–12), none of which is defined from the fitted factor H in a self-referential way. The trace-identity loss (Eq. 2) is an algebraic rewrite of the Frobenius objective, not a circular definition. The three introduced AdaGrad variants are empirical engineering methods with openly stated stagnation/reset and batch-growth heuristics; the paper does not claim regret or rate theorems that would smuggle those heuristics in as predictions. Theorems 6.1–6.2 restate classical links between orthogonality-constrained SymNMF and (spherical) K-means, with proofs in the appendix and citations to Ding/Kuang et al.; that is standard baseline theory, not renaming a fitted pattern as a new law. Self-citations are absent as load-bearing uniqueness claims. Synthetic factor-model inputs and the spectrum-dependent ranking at n=10^6 raise external-validity and experimental-design concerns, but those are not circularity: nothing reduces by construction to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

Load-bearing content is mostly standard optimization and EVT plus synthetic data design choices. Free parameters are the many optimizer hyperparameters and the fixed generator settings that define the benchmark spectra. Invented entities are the three engineering solver variants; they are algorithms, not physical postulates, and are tested only inside this paper’s suite.

free parameters (5)
  • AdaGrad / Piecewise / SVRG / AdaptGrow learning rates and related knobs (η, c, c_r, |I|/n, ϕ_0, snapshot period, RMSprop = e.g. AdaGrad η=2 (Phase 1), η up to 384 at n=10^6 correlation; many others in Table 7
    Phase-1 and Phase-2 winners depend on local hyperparameter sweeps around anchors (Tables 4, 7); rankings are conditional on these tuned values.
  • Factorization rank schedule k=⌊√n⌋ (and TPDM k=n_exc=400 at n=10^6) = k=⌊√n⌋; TPDM n=10^6 uses k=400
    Sets problem difficulty and over/under-parameterization relative to generative rank k+1; primary reported regime.
  • TPDM exceedance fraction q and fixed n_sample = q=0.01, n_sample=40,000 → n_exc=400
    Fixes n_exc=400 for all n, creating the large-n rank ceiling and flat spectrum that drive the TPDM hardness narrative.
  • Convergence thresholds τ_E, τ_g(n), stagnation τ = τ_E=0.1; τ_g^∞=1e-4; TPDM n=10^6 τ_g=1e-3
    Define who ‘converges’; τ_g is relaxed at small n and for TPDM n=10^6.
  • Synthetic mixing-matrix loadings and noise scales = Appendix B defaults
    Common-factor mean 0.8, group loadings [0.6,1.8], noise levels, block-size jitter ±40% shape both spectra.
assumptions (5)
  • domain assumption SymNMF objective min_{H≥0} ||S−HH^T||_F^2 is a valid soft clustering / factor model for absolute correlation and unit-diagonal TPDM inputs
    Stated in §1–§3; standard in SymNMF clustering literature [4,5] and TPDM factorization [1].
  • domain assumption Multivariate regular variation with α=2 and the Cooley–Thibaud TPDM construction yield a symmetric nonnegative Gram-like matrix suitable for SymNMF
    §3.2–3.3 and Appendix A cite Resnick/EVT standard results; α=2 chosen so Σ=AA^T after normalization.
  • ad hoc to paper Projected-gradient KKT residual (with loss gate and stagnation) is the right uniform stopping rule across solver families
    §4.1 combines Lin-style projected gradient with authors’ τ choices; different gates could reorder slow methods.
  • standard math Trace expansion ||S−HH^T||_F^2 = ||S||_F^2 − 2 tr(H^T S H) + ||H^T H||_F^2 is algebraically exact and numerically adequate in fp32 at target E_t
    §2.1; authors quantify cancellation ~2ε/E_t and fall back to gradient gate when loss is precision-limited.
  • domain assumption Synthetic block factor model with shared A is a fair proxy for empirical risk-factor estimation on large portfolios
    §4.4 / Abstract call matrices ‘proxies’; no real-market S is factored.
invented entities (3)
  • Piecewise AdaGrad
    purpose: Reset/recalibrate AdaGrad accumulators on stagnation to avoid long-run saturation while keeping full-history scaling between resets
    Introduced in §2.2.3 and §5.1; empirical only, no regret proof; independent_evidence false outside this benchmark.
  • Row-Stochastic SVRG (for SymNMF/AdaGrad)
    purpose: Cut per-step cost with row subsampling while keeping accumulator growth uniform via full-gradient snapshots
    §5.1 adaptation of SVRG to row-sampled AdaGrad on SymNMF; validated only in this suite.
  • Block-SVRG AdaptGrow
    purpose: Dense block entry sampling with adaptive batch growth plus SVRG hybrid to restore cuBLAS throughput and fix per-row accumulator heterogeneity
    §5.1; wins TPDM n=10^6 in their tables; no external replication yet.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization." pith.science (2026). https://pith.science/paper/I4YNRAG4

@misc{pith2026260724518,
  author       = {Pith},
  title        = {Pith review of: Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I4YNRAG4}},
  note         = {Machine review of arXiv:2607.24518}
}
abstract

Symmetric non-negative matrix factorization (SymNMF) recovers latent group structure from a dependence matrix, but its dense, quadratic-memory objective has confined prior work to moderate sizes. We present a large-scale GPU study of seven algorithm families (over 30 configurations) on absolute Pearson correlation and tail pairwise dependence matrices from Extreme Value Theory, two proxies for empirical risk-factor estimation on large portfolios. A trace-identity reformulation eliminates all $n \times n$ intermediates, so a single GPU reaches $n \approx 10^5$ and multi-node distribution scales to $n = 10^6$ and beyond. Under a two-phase protocol, eleven methods converge at moderate scale; six remain efficient enough at $n = 10^5$ (five AdaGrad-family plus ADMM), and five AdaGrad-family methods still converge at $n = 10^6$: AdaGrad, RMSprop, and three we introduce (Piecewise AdaGrad, Row-Stochastic SVRG, Block-SVRG AdaptGrow). At $n = 10^6$ the fastest solver tracks the matrix spectrum: Block-SVRG AdaptGrow wins on the flat, ill-conditioned tail-dependence spectrum, where its lower per-iteration cost decides a long factorization, and full-batch AdaGrad wins on the dominant-low-rank correlation spectrum, where the run is short. We also benchmark spherical K-means as a hard-label baseline: cheaper when angular cluster structure is present, yet provably degenerate once the matrix collapses toward a single common factor, where the soft factorization remains necessary.

Figures

Figures reproduced from arXiv: 2607.24518 by the authors.

Figure 1
Figure 1. SymNMF on a 100 × 100 benchmark dependence matrix (k = 10). Left: input S; right: reconstruction HH⊤ via H ∈ R 100×10 + . The block structure of S is recovered in the columns of H (relative Frobenius residual in this example ∥S − HH ⊤∥F /∥S∥F = 1.6%). Identifiability. The SymNMF factorization is not unique: if H∗ is a solution, then so is H∗Π for any k × k permutation matrix Π, since (H∗Π)(H∗Π)⊤ = H∗ΠΠ⊤H∗⊤ = H∗H∗⊤. … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 5 linked inside Pith

  1. [1]

    Decompositions of dependence for high-dimensional ex- tremes.Biometrika, 106(3):587–604, 06 2019

    Daniel Cooley and Emeric Thibaud. Decompositions of dependence for high-dimensional ex- tremes.Biometrika, 106(3):587–604, 06 2019

    2019

  2. [2]

    Springer-Verlag, 2001

    Stuart Coles.An Introduction to Statistical Modeling of Extreme Values. Springer-Verlag, 2001

    2001

  3. [3]

    Resnick.Heavy-Tail Phenomena: Probabilistic and Statistical Modeling

    Sidney I. Resnick.Heavy-Tail Phenomena: Probabilistic and Statistical Modeling. Springer Series in Operations Research and Financial Engineering. Springer New York, 2010

    2010

  4. [4]

    Symmetric nonnegative matrix factorization for graph clustering

    Da Kuang, Chris Ding, and Haesun Park. Symmetric nonnegative matrix factorization for graph clustering. InProceedings of the 2012 SIAM International Conference on Data Mining, pages 106–117. SIAM, 2012

    2012

  5. [5]

    Symnmf: Nonnegative low-rank approximation of a similarity matrix for graph clustering.Journal of Global Optimization, 62(3):545–574, 2015

    Da Kuang, Sangwoon Yun, and Haesun Park. Symnmf: Nonnegative low-rank approximation of a similarity matrix for graph clustering.Journal of Global Optimization, 62(3):545–574, 2015

    2015

  6. [6]

    Adaptive subgradient methods for online learning and stochastic optimization.Journal of Machine Learning Research, 12:2121–2159, 2011

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization.Journal of Machine Learning Research, 12:2121–2159, 2011

    2011

  7. [7]

    Chris Ding, Xiaofeng He, and Horst D. Simon. On the equivalence of nonnegative matrix factorization and spectral clustering. InProc. SIAM Int. Conf. Data Mining, pages 606–610, 2005

    2005

  8. [8]

    Lee and H

    Daniel D. Lee and H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization.Nature, 401(6755):788–791, 1999

    1999

Show all 70 references
  1. [9]

    Lee and H

    Daniel D. Lee and H. Sebastian Seung. Algorithms for non-negative matrix factorization. In Advances in Neural Information Processing Systems 13 (NIPS 2000), pages 556–562. MIT Press, 2001

    2000

  2. [10]

    Fundamentals of Algorithms

    Nicolas Gillis.Nonnegative Matrix Factorization. Fundamentals of Algorithms. SIAM, 2020

    2020

  3. [11]

    Distributed optimization and statistical learning via the alternating direction method of multipliers.Foun- dations and Trends in Machine Learning, 3(1):1–122, 2011

    Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers.Foun- dations and Trends in Machine Learning, 3(1):1–122, 2011

    2011

  4. [12]

    Symmetric non- negative matrix factorization: Algorithms and applications to probabilistic clustering.IEEE Transactions on Neural Networks, 22(12):2117–2131, 2011

    Zhaoshui He, Shengli Xie, Rafal Zdunek, Guoxu Zhou, and Andrzej Cichocki. Symmetric non- negative matrix factorization: Algorithms and applications to probabilistic clustering.IEEE Transactions on Neural Networks, 22(12):2117–2131, 2011

    2011

  5. [13]

    Nemirovsky and David B

    Arkadi S. Nemirovsky and David B. Yudin.Problem Complexity and Method Efficiency in Optimization. Wiley, 1983

    1983

  6. [14]

    Minimization of functions having Lipschitz continuous first partial derivatives

    Larry Armijo. Minimization of functions having Lipschitz continuous first partial derivatives. Pacific Journal of Mathematics, 16(1):1–3, 1966

    1966

  7. [15]

    Some methods of speeding up the convergence of iteration methods.USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964

    Boris T Polyak. Some methods of speeding up the convergence of iteration methods.USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964. 36

    1964

  8. [16]

    A fast iterative shrinkage-thresholding algorithm for linear inverse problems.SIAM Journal on Imaging Sciences, 2(1):183–202, 2009

    Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems.SIAM Journal on Imaging Sciences, 2(1):183–202, 2009

    2009

  9. [17]

    Adaptive restart for accelerated gradient schemes.Foundations of Computational Mathematics, 15(3):715–732, 2015

    Brendan O’Donoghue and Emmanuel Cand` es. Adaptive restart for accelerated gradient schemes.Foundations of Computational Mathematics, 15(3):715–732, 2015

    2015

  10. [18]

    Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude

    Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude. Neural Networks for Machine Learning, Coursera Lecture 6e, 2012

    2012

  11. [19]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  12. [20]

    Incorporating Nesterov momentum into Adam.ICLR Workshop, 2016

    Timothy Dozat. Incorporating Nesterov momentum into Adam.ICLR Workshop, 2016

    2016

  13. [21]

    Adan: Adaptive nes- terov momentum algorithm for faster optimizing deep models.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):1791–1804, 2024

    Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nes- terov momentum algorithm for faster optimizing deep models.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):1791–1804, 2024

    2024

  14. [22]

    Accelerating stochastic gradient descent using predictive vari- ance reduction.Advances in Neural Information Processing Systems, 26, 2013

    Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive vari- ance reduction.Advances in Neural Information Processing Systems, 26, 2013

    2013

  15. [23]

    Friedlander and Mark Schmidt

    Michael P. Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting.SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012

    2012

  16. [24]

    Byrd, Gillian M

    Richard H. Byrd, Gillian M. Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning.Mathematical Programming, 134(1):127–155, 2012

    2012

  17. [25]

    Hyunsoo Kim and Haesun Park. Nonnegative matrix factorization based on alternating non- negativity constrained least squares and active set method.SIAM Journal on Matrix Analysis and Applications, 30(2):713–730, 2008

    2008

  18. [26]

    Algorithms for nonnegative matrix and tensor factorizations: A unified view based on block coordinate descent framework.Journal of Global Optimization, 58(2):285–319, 2014

    Jingu Kim, Yunlong He, and Haesun Park. Algorithms for nonnegative matrix and tensor factorizations: A unified view based on block coordinate descent framework.Journal of Global Optimization, 58(2):285–319, 2014

    2014

  19. [27]

    A method for the solution of certain non-linear problems in least squares

    Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2(2):164–168, 1944

    1944

  20. [28]

    Marquardt

    Donald W. Marquardt. An algorithm for least-squares estimation of nonlinear parameters. Journal of the Society for Industrial and Applied Mathematics, 11(2):431–441, 1963

    1963

  21. [29]

    Wright.Numerical Optimization

    Jorge Nocedal and Stephen J. Wright.Numerical Optimization. Springer Series in Operations Research. Springer, 2nd edition, 2006

    2006

  22. [30]

    Distributed-memory parallel symmetric nonnegative matrix factorization

    Srinivas Eswar, Koby Hayashi, Grey Ballard, Ramakrishnan Kannan, Richard Vuduc, and Haesun Park. Distributed-memory parallel symmetric nonnegative matrix factorization. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–...

    2020

  23. [31]

    Matheson, and Haesun Park

    Srinivas Eswar, Koby Hayashi, Grey Ballard, Ramakrishnan Kannan, Michael A. Matheson, and Haesun Park. PLANC: Parallel low-rank approximation with nonnegativity constraints. ACM Transactions on Mathematical Software, 47(3):1–37, 2021

    2021

  24. [32]

    Aksoy, Grey Ballard, and Haesun Park

    Koby Hayashi, Sinan G. Aksoy, Grey Ballard, and Haesun Park. Randomized algorithms for symmetric nonnegative matrix factorization.arXiv preprint arXiv:2402.08134, 2024

    2024 arXiv

  25. [33]

    Symnmf-net for the symmetric nmf problem

    Mingjie Li, Haoyang Kong, and Zhouchen Lin. Symnmf-net for the symmetric nmf problem. arXiv preprint arXiv:2205.13214, 2022

    2022 arXiv

  26. [34]

    Re- thinking non-negative matrix factorization with implicit neural representations

    Deepak Subramani, Takumi Imoto, Toshihisa Sato, Nobutaka Ono, and Shoji Miyazaki. Re- thinking non-negative matrix factorization with implicit neural representations. InProc. In- terspeech, 2024

    2024

  27. [35]

    Springer Science & Business Media, 2013

    Paul Embrechts, Claudia Kl¨ uppelberg, and Thomas Mikosch.Modelling extremal events: for insurance and finance, volume 33 ofSpringer Science & Business Media. Springer Science & Business Media, 2013

    2013

  28. [36]

    Springer Series in Operations Research and Financial Engineering

    Laurens de Haan and Ana Ferreira.Extreme Value Theory: An Introduction. Springer Series in Operations Research and Financial Engineering. Springer New York, 2010

    2010

  29. [37]

    Resnick.Extreme Values, Regular Variation and Point Processes

    Sidney I. Resnick.Extreme Values, Regular Variation and Point Processes. Springer, 2014

    2014

  30. [38]

    Basel iii: A global regulatory framework for more resilient banks and banking systems, 2011

    Basel Committee on Banking Supervision. Basel iii: A global regulatory framework for more resilient banks and banking systems, 2011

    2011

  31. [39]

    A tail dependence-based dissimilarity measure for financial time series clustering.Adv Data Anal Classif, 5:323–340, 2011

    Giovanna De Luca and Paolo Zuccolotto. A tail dependence-based dissimilarity measure for financial time series clustering.Adv Data Anal Classif, 5:323–340, 2011

    2011

  32. [40]

    Hierarchical risk parity: Accounting for tail dependencies in multi-asset multi-factor allocations

    Harald Lohre, Carsten Rother, and Kilian Axel Sch¨ afer. Hierarchical risk parity: Accounting for tail dependencies in multi-asset multi-factor allocations. In Emmanuel Jurczenko, editor, Machine Learning and Asset Management, pages 332–368. Iste and Wiley, 2020

    2020

  33. [41]

    Bivariate extreme statistics.Annals of the Institute of Statistical Mathematics, 11(2):195–210, 1960

    Masaaki Sibuya. Bivariate extreme statistics.Annals of the Institute of Statistical Mathematics, 11(2):195–210, 1960

    1960

  34. [42]

    Martin Schlather and Jonathan A. Tawn. A dependence measure for multivariate and spatial extreme values: Properties and inference.Biometrika, 90(1):139–156, 2003

    2003

  35. [43]

    Extremal behavior of stochastic integrals driven by regularly varying l´ evy processes.The Annals of Probability, 35(1):225–264, jan 2007

    Henrik Hult and Filip Lindskog. Extremal behavior of stochastic integrals driven by regularly varying l´ evy processes.The Annals of Probability, 35(1):225–264, jan 2007

    2007

  36. [44]

    Konstantinides, and Dingding Wang

    Meng Cheng, Dimitrios G. Konstantinides, and Dingding Wang. Multivariate regularly varying insurance and financial risks in multidimensional risk models.Journal of Applied Probability, 61(4):1319–1342, 2024

    2024

  37. [45]

    Bertsekas.Nonlinear Programming

    Dimitri P. Bertsekas.Nonlinear Programming. Athena Scientific, Belmont, MA, 2nd edition, 1999

    1999

  38. [46]

    Projected gradient methods for nonnegative matrix factorization.Neural Com- putation, 19(10):2756–2779, 2007

    Chih-Jen Lin. Projected gradient methods for nonnegative matrix factorization.Neural Com- putation, 19(10):2756–2779, 2007. 38

    2007

  39. [47]

    A well-conditioned estimator for large-dimensional covariance matrices.Journal of Multivariate Analysis, 88(2):365–411, 2004

    Olivier Ledoit and Michael Wolf. A well-conditioned estimator for large-dimensional covariance matrices.Journal of Multivariate Analysis, 88(2):365–411, 2004

    2004

  40. [48]

    Marchenko and Leonid A

    Vladimir A. Marchenko and Leonid A. Pastur. Distribution of eigenvalues for some sets of random matrices.Matematicheskii Sbornik, 114(4):507–536, 1967

    1967

  41. [49]

    Consistency of spectral clustering in stochastic block models

    Jing Lei and Alessandro Rinaldo. Consistency of spectral clustering in stochastic block models. The Annals of Statistics, 43(1):215–237, 2015

    2015

  42. [50]

    Jose Menchero, D. J. Orr, and Jun Wang. The Barra US equity model (USE4).MSCI Barra Research Notes, 2011

    2011

  43. [51]

    Svd based initialization: A head start for nonnegative matrix factorization.Pattern Recognition, 41(4):1350–1362, 2008

    Christos Boutsidis and Efstratios Gallopoulos. Svd based initialization: A head start for nonnegative matrix factorization.Pattern Recognition, 41(4):1350–1362, 2008

    2008

  44. [52]

    Polyak and Anatoli B

    Boris T. Polyak and Anatoli B. Juditsky. Acceleration of stochastic approximation by averag- ing.SIAM Journal on Control and Optimization, 30(4):838–855, 1992

    1992

  45. [53]

    Efficient estimations from a slowly convergent Robbins–Monro process.Tech- nical Report, Cornell University, 1988

    David Ruppert. Efficient estimations from a slowly convergent Robbins–Monro process.Tech- nical Report, Cornell University, 1988

    1988

  46. [54]

    Dual averaging methods for regularized stochastic learning and online optimization

    Lin Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research, 11:2543–2596, 2010

    2010

  47. [55]

    Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.SIAM Review, 53(2):217–288, 2011

    2011

  48. [56]

    Woodruff

    David P. Woodruff. Sketching as a tool for numerical linear algebra.Foundations and Trends in Theoretical Computer Science, 10(1–2):1–157, 2014

    2014

  49. [57]

    Some methods for classification and analysis of multivariate observations

    James MacQueen. Some methods for classification and analysis of multivariate observations. InProc. 5th Berkeley Symp. Math. Statist. Prob., pages 281–297, 1967

    1967

  50. [58]

    Dhillon and Dharmendra S

    Inderjit S. Dhillon and Dharmendra S. Modha. Concept decompositions for large sparse text data using clustering.Machine Learning, 42:143–175, 2001

    2001

  51. [59]

    Spherical k-means clustering

    Kurt Hornik, Ingo Feinerer, Martin Kober, and Christian Buchta. Spherical k-means clustering. Journal of Statistical Software, 50(10):1–25, 2012

    2012

  52. [60]

    Nonconvex admm for sparse+sparse matrix decom- position.arXiv preprint arXiv:1212.0437, 2012

    Wei Deng, Ming-Jun Lai, and Wotao Yin. Nonconvex admm for sparse+sparse matrix decom- position.arXiv preprint arXiv:1212.0437, 2012

    2012 arXiv

  53. [61]

    A stochastic nonconvex splitting method for symmetric nonnegative matrix factorization

    Songtao Lu, Mingyi Hong, and Zhengdao Wang. A stochastic nonconvex splitting method for symmetric nonnegative matrix factorization. InInternational Conference on Machine Learning (ICML), pages 2232–2241, 2017

    2017

  54. [62]

    Convergence analysis of alternating direction method of multipliers for nonconvex problems.Journal of Optimization Theory and Applications, 172(3):33–64, 2016

    Mingyi Hong, Zhi-Quan Luo, and Meisam Razaviyayn. Convergence analysis of alternating direction method of multipliers for nonconvex problems.Journal of Optimization Theory and Applications, 172(3):33–64, 2016. 39

    2016

  55. [63]

    MPI-F AUN: An MPI-based frame- work for alternating-updating nonnegative matrix factorization.IEEE Transactions on Knowl- edge and Data Engineering, 30(3):544–558, 2018

    Ramakrishnan Kannan, Grey Ballard, and Haesun Park. MPI-F AUN: An MPI-based frame- work for alternating-updating nonnegative matrix factorization.IEEE Transactions on Knowl- edge and Data Engineering, 30(3):544–558, 2018

    2018

  56. [64]

    Ehsan Amid and Manfred K. Warmuth. Winnowing with gradient descent. InProc. Conference on Learning Theory (COLT), 2020

    2020

  57. [65]

    Variants of RMSProp and Adagrad with logarithmic regret bounds

    Mahesh Chandra Mukkamala and Matthias Hein. Variants of RMSProp and Adagrad with logarithmic regret bounds. InProc. ICML, 2017

    2017

  58. [66]

    Disentangling adaptive gradient methods from learning rates

    Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. 2020

    2020

  59. [67]

    AdagradDecay optimizer.https://deeprec.readthedocs.io/en/latest/ AdagradDecay-Optimizer.html, 2023

    Alibaba Group. AdagradDecay optimizer.https://deeprec.readthedocs.io/en/latest/ AdagradDecay-Optimizer.html, 2023. DeepRec framework for recommendation and search

    2023

  60. [68]

    SGDR: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proc. ICLR, 2017

    2017

  61. [69]

    Grad-gradagrad? a non-monotone adaptive stochastic gradient method.arXiv preprint arXiv:2206.06900, 2022

    Aaron Defazio, Baoyu Zhou, and Lin Xiao. Grad-gradagrad? a non-monotone adaptive stochastic gradient method.arXiv preprint arXiv:2206.06900, 2022

    2022 arXiv

  62. [70]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. InProceedings of the 35th International Conference on Machine Learning, pages 1842–1850. PMLR, 2018. 40

    2018

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.