Pith. sign in

REVIEW 6 major objections 6 minor 24 references

A new R package for non-negative matrix factorization, nnmf, claims to deliver lower reconstruction error and less run-to-run variability than the fast RcppML package on most real-world datasets, at the cost of longer runtimes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:48 UTC pith:UEBM5KO4

load-bearing objection Useful NMF package benchmark, but the stability claim is built on unexplained near-zero variance and under-documented benchmark settings. the 6 major comments →

arxiv 2607.20084 v1 pith:UEBM5KO4 submitted 2026-07-22 stat.ML cs.LG

Non--negative matrix factorization using the textit{R} package textsf{nnmf}

Volkan Sevin\c{c} , Nikolas Kontemeniotis , Theodoros Perdikis , Michail Tsagris This is my paper
classification stat.ML cs.LG
keywords non-negative matrix factorizationR packagealternating non-negative least squaresquadratic programmingexponentiated gradientbenchmark studyreconstruction errorrun-to-run stability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a new R package, nnmf, that solves non-negative matrix factorization by alternating non-negative least squares: quadratic programming when samples outnumber variables, and a hybrid quadratic-programming/exponentiated-gradient scheme when variables outnumber samples. Using 26 real datasets and two factorization ranks, the paper compares nnmf against two established R implementations, NMF and RcppML, on reconstruction error, variability across repeated runs, and wall-clock time. The central finding is that nnmf most often produces the smallest reconstruction error at rank k=10 and the smallest spread of solutions across repeated runs, while RcppML is consistently the fastest. The paper argues that this gives researchers objective guidance when choosing among NMF packages: choose nnmf for stability and high-rank accuracy, RcppML for speed, and NMF for low-rank reconstruction.

Core claim

The paper's central claim is that its nnmf implementation is not just another NMF solver but a competitive one: on most of the 26 real datasets, nnmf achieves lower or comparable mean squared reconstruction error than RcppML, with a markedly smaller range of errors over 50 repeated runs, especially at rank k=10. At k=3, the established NMF package tends to give the lowest SSE, but nnmf is faster than NMF and more stable than either competitor. The paper concludes that RcppML is the speed winner, nnmf is the stability winner and the preferred choice for higher ranks, and NMF remains preferable for low-rank reconstruction.

What carries the argument

The core algorithm is alternating non-negative least squares (ANLS) with two regimes. When n>p, each row of W is updated by solving a small quadratic program, and H is updated in one joint quadratic program whose Hessian is the Kronecker product I_p ⊗ W^T W; the reconstruction error is read off directly from the optimal QP value via ||X − WH||_F^2 = ||X||_F^2 + 2f*. When n<p, the H-update switches to an exponentiated-gradient multiplicative step with learning rate η_H, avoiding the large kp × kp Hessian. Starting values come from k-means (or sparse k-means in the wide case), and a small ridge term guards against rank-deficient Hessians.

Load-bearing premise

The benchmark conclusions assume the three packages were run under genuinely comparable conditions — same initialization, tolerance, and stopping rules — but the paper does not document those settings, so the observed gaps could reflect configuration choices rather than intrinsic implementation quality.

What would settle it

Run the 26-dataset benchmark with all three packages started from identical W,H matrices and identical convergence tolerance, then count how often nnmf's MSE beats RcppML's; if RcppML wins the majority, the accuracy claim collapses. Also re-examine datasets 10 and 11, whose identical nnmf range values in the k=3 table suggest the data-preparation step differed from the other datasets.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For applications that value reproducibility, nnmf is the stability choice: across the 26 datasets its run-to-run error range is smaller than RcppML's in 84.6–92.3% of cases (depending on rank), and often far smaller.
  • At rank k=10, nnmf produces the lowest reconstruction error more often than either competitor, making it a candidate for high-rank factorizations; at k=3, the NMF package remains the accuracy leader.
  • RcppML is faster than nnmf in essentially every case tested, so users with hard time or memory budgets will still prefer it; nnmf's runtime advantage over NMF shrinks at higher ranks.
  • nnmf's additional functions (sparse-data solver, Manhattan-distance NMF, cross-validated rank selection, prediction, and covariate-aware factorization) mean the package can serve entire workflows that would otherwise require assembling multiple packages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the stability advantage survives controlled initialization and stopping criteria, the likely mechanism is the joint QP update for H: instead of a single multiplicative step per iteration, nnmf solves the full nonnegative least-squares subproblem, which should make each iteration more accurate and less sensitive to starting points. The paper does not isolate this mechanism.
  • The same joint-QP machinery could support exact rank selection or missing-data NMF by adding constraints to the QP; the paper's nnmfqp.cv() only does cross-validation, so this is untested.
  • A direct test is to rerun the benchmark with identical starting W,H and identical convergence tolerance across all three packages. If RcppML then matches or beats nnmf's MSE on most datasets, the reported accuracy gap is configuration-dependent rather than algorithmic.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces an R package, nnmf, that implements non-negative matrix factorization (NMF) via alternating non-negative least squares, using quadratic programming (QP) when n>p and a hybrid QP–exponentiated-gradient algorithm when n<p. The authors benchmark nnmf against two existing R packages, NMF and RcppML, on 26 real-world datasets for ranks k=3 and k=10. They report minimum MSE, the range of MSE over 50 repeated runs, and computation time. The central claims are that nnmf often yields lower MSE (especially for k=10) and much lower variability across runs, while RcppML is generally the fastest; NMF is said to be competitive at low rank. The paper also describes the package functions, initialization strategies, and algorithmic details.

Significance. If the empirical claims are substantiated, the paper provides a useful practical comparison of NMF implementations in R and introduces a new package with publicly available code. Strengths include the use of real-world data rather than simulations, the explicit algorithmic pseudocode in Algorithms 1–2, and the attention to several evaluation criteria (MSE, variability, runtime, memory). However, the current evidence base is not yet reliable: the comparability of package settings is undocumented, several tables contain suspicious values, and the variability results are reported with no uncertainty quantification. The practical guidance offered in Sections 4–5 therefore needs stronger experimental support.

major comments (6)
  1. [§4, Tables 3–6] The comparison settings for NMF and RcppML are not reported. Section 4 says only 'comparable parameter settings' are used; Section 3.4 details nnmf defaults (tol=1e-6, maxiter=1000, k_means=TRUE), but no function calls, initialization seeds, or convergence tolerances are given for the other two packages. The conclusions in Section 5—e.g., that nnmf 'provides smaller MSE values ... compared to RcppML'—rest on the assumption that the runs are aligned. Please provide complete, version-pinned replication code and exact settings for all packages.
  2. [Table 5] Datasets 10 and 11 have different dimensions (7,650×57 vs. 13,027×65), yet Table 5 reports for nnmf the identical MSE (1.174×10^11) and identical range (64.000) for both; the RcppML and NMF entries are also nearly identical. This is strong evidence of a data-preparation or table-copying error. The affected rows must be re-computed and the aggregate percentages (Tables 3–4) updated before the empirical claims can be assessed.
  3. [Tables 3 and 5] Many nnmf ranges in Table 5 are at machine precision (e.g., datasets 3, 17, 18, 21, 24, 26 have ranges ≤1e-8), although the experiment repeats each dataset 50 times and Section 3.2 describes random or K-means–based initialization. If a fixed seed or a deterministic initialization path is used, the 'range' comparison in Tables 3–4 is not a measure of algorithmic variability; if initialization is truly random, the mechanism producing 50 identical solutions must be explained. Report the seeds and initialization state for every run.
  4. [§5 vs Tables 3–4] Section 5 concludes that RcppML is 'a little faster' than nnmf, and Section 4 states their computational-cost performance is 'almost similar.' The time ratios in Tables 3–4 contradict this: RcppML/nnmf ratios are as low as 0.001 (dataset 10) for k=3 and 5.05×10^-4 (dataset 16) for k=10, i.e., RcppML is hundreds to thousands of times faster on those datasets. The runtime comparison should be stated quantitatively and by dataset.
  5. [§2.2, Eq. (1)–(2)] Equation (1) in §2.1 defines the objective as ∥X−WH^T∥² with X≈WH^T, while Eq. (2) and all subsequent derivations in §3.1 and Algorithms 1–2 use X≈WH with ∥X−WH∥². This notational inconsistency affects the derivation of the gradient and the QP formulations. Please make the model definition consistent.
  6. [Tables 3–5] The benchmark summary is purely descriptive: percentages such as '84.6% of cases' (Tables 3–4) are presented without standard errors, confidence intervals, or paired tests across the 26 datasets. In a comparison whose central quantity is run-to-run variability (range of MSE), the absence of any uncertainty quantification makes claims of 'significant' superiority (Section 5) unsupported. Report the distribution of the comparison metrics and, ideally, paired tests.
minor comments (6)
  1. [Table 2] Row 11: '1,3027' should read '13,027'.
  2. [Algorithms 1–2, §3.1.2] The EG update uses 'D' in 'k×D'; define D=p or replace with p.
  3. [§3.2] 'highted' is a typo for 'highlighted'.
  4. [§3.1.1] The sentence beginning 'This joint formulation is only feasible because...' is grammatically unclear and should be rewritten.
  5. [References] The reference for Tsagris et al. (2026) lists GitHub URL 'shabbychef/rnnmf' for the package 'nnmf'; verify the repository name.
  6. [Tables 5–6] 'Runing time' should be 'Running time'.

Circularity Check

0 steps flagged

No circular derivation: benchmark claims rest on external comparisons; the only self-citation (the package itself) is descriptive, not load-bearing.

full rationale

The paper is a software benchmark, not a derivation. Its central claims—that nnmf is competitive in MSE, has small solution ranges, and is slower than RcppML—are empirical summaries of Tables 3-6, computed from external datasets and runs of three independently developed packages. No fitted parameter is renamed as a prediction: Section 3.3 uses an already-fixed H to project new observations into W*, which is standard out-of-sample folding-in and is not part of the headline benchmark. The only self-citation is the reference to the authors' own nnmf package (Tsagris et al., 2026) in Section 1, used to identify the software rather than to justify a mathematical conclusion; no uniqueness theorem or optimality claim is imported from prior work by the same authors. Possible concerns about the benchmark—Section 4 says 'comparable parameter settings' without documenting exact initialization, tolerances, or function calls for the competitor packages, and Tables 5-6 contain very similar or identical entries for datasets 10 and 11—are experimental-control or correctness risks, not circularity, because they do not make any reported result equal its input by construction. No circular step can be exhibited from the text, so the circularity score is low.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The central claim rests on a benchmark protocol with several hand-set hyperparameters and unverified assumptions about comparability and data preparation. No new theoretical entities are introduced.

free parameters (7)
  • rank k = 3 and 10
    Factorization rank selected by the authors for the benchmark; strongly affects MSE and runtime.
  • ridge lambda = 1e-8
    Added to HH^T in the QP Hessian to guard rank deficiency; affects the W update.
  • EG learning rate eta_H = 0.1
    Default learning rate for the n < p hybrid update; controls H convergence speed and final SSE.
  • convergence tolerance tau = 1e-6
    Stopping rule based on SSE change; different packages may use different criteria, affecting reported time and accuracy.
  • max iterations T = 1000
    Iteration cap; may bind on harder datasets and affect final reconstruction error.
  • number of runs B = 50
    Each model is run 50 times and the minimum SSE is used; affects the MSE and range statistics.
  • K-means initialization max iterations = 10
    Default setting for Hartigan-Wong K-means used to initialize H; influences solution stability.
axioms (5)
  • domain assumption Alternating nonnegative least squares converges to a stationary point of the nonconvex NMF objective, and local solutions are comparable across packages.
    Algorithms 1-2 alternate between W and H updates; no global optimality guarantee, so the comparison assumes local optima are representative.
  • domain assumption The Bro and De Jong QP solver and the OSQP solver solve the nonnegative quadratic subproblems accurately.
    Section 3.1 relies on exact QP solves for W and H; solver tolerances are not reported.
  • ad hoc to paper The EG update with eta_H=0.1 produces an acceptable SSE trajectory for n < p cases.
    Algorithm 2 uses a fixed learning rate chosen by default; no line search or convergence guarantee is provided.
  • domain assumption Frobenius reconstruction error is the appropriate quality measure, and the range over 50 runs is a valid stability measure.
    Section 4 evaluates only MSE/range/time; alternative metrics or distributional summaries would change conclusions.
  • domain assumption Datasets in Table 2 were preprocessed consistently and are representative of real-world NMF applications.
    No preprocessing details or code are given; datasets 10 and 11 show identical reported values, suggesting a data error.

pith-pipeline@v1.3.0-alltime-deepseek · 15029 in / 14408 out tokens · 135573 ms · 2026-08-01T10:48:32.997858+00:00 · methodology

0 comments
read the original abstract

Non--negative matrix factorization (NMF) has become an established dimensionality reduction technique for extracting latent structures from non--negative data and has found widespread applications in fields such as bioinformatics, text mining, image analysis, and recommender systems. As the popularity of NMF has increased, numerous \textit{R} packages implementing different optimization strategies and computational frameworks have been developed. Despite their widespread availability, comprehensive evaluations of these implementations under real--world data conditions remain limited. Consequently, researchers often lack objective guidance when selecting an appropriate package for practical applications. This study introduces a new \textit{R} package for NMF and offers asystematic performance comparison with two widely available \textit{R} packages for NMF analysis. Rather than relying on simulated datasets, the evaluation is conducted using real--world data to better reflect the complexity, heterogeneity, and noise characteristics encountered in practical analytical settings. The packages are assessed using a consistent experimental framework, with emphasis on computational efficiency, convergence behavior, reconstruction accuracy, memory utilization, and the stability of the resulting matrix factorization.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references

  1. [1]

    and Mesirov, Jill P

    Brunet, Jean-Philippe and Tamayo, Pablo and Golub, Todd R. and Mesirov, Jill P. , title =. Proceedings of the National Academy of Sciences , year =

  2. [2]

    Statistics and Computing , year =

    Carbonetto, Peter and Sarkar, Abhishek and Wang, Zihao and Stephens, Matthew , title =. Statistics and Computing , year =

  3. [3]

    , title =

    Chalise, Prabhakar and Fridley, Brooke L. , title =. PLOS ONE , year =

  4. [4]

    2026 , note =

    DeBruine, Zach , title =. 2026 , note =

    2026

  5. [5]

    Rcpp: Seamless

    Eddelbuettel, Dirk and Fran. Rcpp: Seamless. Journal of Statistical Software , year =

  6. [6]

    2015 , howpublished =

    Gaujoux, Renaud , title =. 2015 , howpublished =

    2015

  7. [7]

    BMC Bioinformatics , year =

    Gaujoux, Renaud and Seoighe, Cathal , title =. BMC Bioinformatics , year =

  8. [8]

    SIAM Journal on Matrix Analysis and Applications , year =

    Kim, Hyunsoo and Park, Haesun , title =. SIAM Journal on Matrix Analysis and Applications , year =

  9. [9]

    and Seung, H

    Lee, Daniel D. and Seung, H. Sebastian , title =. Nature , year =

  10. [10]

    Sebastian , title =

    Lee, Daniel and Seung, H. Sebastian , title =. Advances in Neural Information Processing Systems , year =

  11. [11]

    , title =

    Lin, Xihui and Boutros, Paul C. , title =. BMC Bioinformatics , year =

  12. [12]

    2010 , howpublished =

    Liu, Qiang , title =. 2010 , howpublished =

    2010

  13. [13]

    Environmetrics , year =

    Paatero, Pentti and Tapper, Unto , title =. Environmetrics , year =

  14. [14]

    Biology Methods and Protocols , year =

    Quintero, Andres and H. Biology Methods and Protocols , year =

  15. [15]

    2026 , note =

    Tsagris, Michail and Kontemeniotis, Nikolas and Adam, Christiana , title =. 2026 , note =

    2026

  16. [16]

    Journal of Open Source Software , year =

    Tsuyuzaki, Koki and Nikaido, Itoshi , title =. Journal of Open Source Software , year =

  17. [17]

    IEEE Transactions on Knowledge and Data Engineering , year =

    Wang, Yu-Xiong and Zhang, Yu-Jin , title =. IEEE Transactions on Knowledge and Data Engineering , year =

  18. [18]

    2023 , doi =

    Carbonetto, Peter and Luo, Kevin and Sarkar, Abhishek and Hung, Anthony and Tayeb, Kaixuan and Pott, Sebastian and Stephens, Matthew , journal =. 2023 , doi =

    2023

  19. [19]

    SIAM Journal on Matrix Analysis and Applications , volume=

    Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method , author=. SIAM Journal on Matrix Analysis and Applications , volume=. 2008 , publisher=

    2008

  20. [20]

    2014 , publisher=

    Kim, Jingu and He, Yunlong and Park, Haesun , journal=. 2014 , publisher=

    2014

  21. [21]

    Journal of Chemometrics: A Journal of the Chemometrics Society , volume=

    A fast non-negativity-constrained least squares algorithm , author=. Journal of Chemometrics: A Journal of the Chemometrics Society , volume=. 1997 , publisher=

    1997

  22. [22]

    1979 , publisher=

    Hartigan, John A and Wong, Manchek A , journal=. 1979 , publisher=

    1979

  23. [23]

    Journal of the American Statistical Association , volume=

    A framework for feature selection in clustering , author=. Journal of the American Statistical Association , volume=. 2010 , publisher=

    2010

  24. [24]

    Bartolomeo Stellato and Goran Banjac and Paul Goulart and Stephen Boyd and Vineet Bansal , year =