REVIEW 4 major objections 4 minor 35 references
Nonnegative matrix factorization and the principle of the common cause
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The principle of the common cause picks the rank where NMF features become stable.
desk verdict A promising rank-selection heuristic for NMF, but the evidence doesn't support the 'resolves nonidentifiability' claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is the predictability inequality (16), proved from exact conditional independence: for each pair of pixel and image there exist basis images b1 and b2 such that p(π|b1) ≤ p(π|i) ≤ p(π|b2). The smallest rank at which this inequality holds for essentially all pixels defines the effective rank Rc. Stability is then assessed by comparing two sets of basis images with cosine distances and optimal one-to-one matching; the regime around Rc is where matched distances stay small, while for much larger ranks the basis images become unstable.
What would settle it
Run the same stability protocol on a synthetic dataset with known ground-truth nonnegative rank and known generating parts: if basis images around Rc fail to recover those parts under the paper's noise model, or if the matched cosine distances are no better than distances obtained by matching random basis-image pairs, the rank/stability claim would be refuted.
Extended reading notes
Core claim
The paper's central claim is that a predictive inequality derived from the principle of the common cause—for every pixel and image, some basis image predicts the pixel at least as strongly as the image itself, in both directions—gives a workable way to choose the effective rank of a nonnegative matrix factorization. Around the resulting rank Rc, the basis images found by local optimization are stable under bit-flip noise and under changes of the random seed, so the nonidentifiability of NMF does not prevent reliable feature extraction. The paper also claims that NMF implements an approximate form of the common-cause principle: the higher and more positively correlated joint probabilities are
Load-bearing premise
The stability conclusion rests on treating the few computed local minima (one seed pairing or one noise draw, matched optimally) as representative of all NMF solutions; if those minima are unrepresentative, the claim that nonidentifiability is effectively resolved does not follow.
Editorial extensions
If this is right
- Around the common-cause rank Rc, NMF basis images can be treated as interpretable features despite the factorization's non-uniqueness, because they survive noise and seed changes.
- The PCC-based rank estimate remains sensible under weak noise in cases where Bayesian Information Criterion based estimates fail to show a minimum.
- NMF gives an approximate common-cause principle: larger, positively correlated probabilities are explained better, which can ground clustering by shared causes.
- NMF denoises when the rank lies in a middle interval [R1, R2]; below it no reliable features form, and above it the model overfits noise.
- Reconstructed images have per-image entropy at least as large as the original conditional distributions, and basis images become sparser than images once the rank exceeds a small threshold.
Reading between the lines
- Beyond the paper: if the stability result transfers to other nonnegative data matrices (not just gray-scale images), the same rank-selection rule could serve as a default hyperparameter choice for NMF in applications where basis interpretability matters.
- Beyond the paper: the approximate common-cause principle suggests a testable design criterion for NMF objectives—explicitly upweighting large positive joint probabilities should reproduce or improve the observed anticorrelation between approximation error and probability size.
- Beyond the paper: the nonmonotonic distance distribution seen when changing seeds at different ranks is an open clue; a systematic study with many seeds and chance-level baselines could show whether the apparent stability around Rc is a true geometric property or a selection effect.
- Beyond the paper: because the rank estimate is derived from a conditional-independence condition, it may also apply to other latent-variable models beyond matrix factorization, such as topic models or nonnegative tensor factorizations.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a conceptual and methodological connection between nonnegative matrix factorization (NMF) and Reichenbach's principle of the common cause (PCC). It derives a predictability inequality from exact NMF/PCC and uses its empirical satisfaction to define an effective rank R_c, claiming this estimate is stable against weak noise in contrast to BIC. The paper further claims that NMF at ranks around R_c produces basis images that are stable against noise and against the seed of local optimization, thereby effectively resolving the NMF nonidentifiability problem. Additional contributions include an approximate-PCC interpretation with correlation inequalities and entropy relations, a clustering method based on common-cause weights, and denoising experiments comparing NMF with PCA.
Significance. If the central claims were fully substantiated, the paper would offer a practical rank-selection criterion and a resolution of NMF nonidentifiability, which are longstanding practical problems. The exact derivation of inequality (16) is clean, and the conceptual link between NMF and PCC is interesting and potentially fruitful. The BIC comparison and the denoising experiments, including the random-accuracy baseline, are useful additions. However, the current evidence for the strongest claims is incomplete: the noise-stability experiment is confounded by a data split, no chance baseline is provided for the matched cosine distances, and the paper itself acknowledges missing semantic analysis. The significance is therefore conditional on substantial revision.
major comments (4)
- [Section VI] The noise-stability experiment is confounded: the clean and noisy NMF runs are performed on two different halves of UTK, so any distance between the matched basis images reflects not only noise but also the difference between two independent subsets of data. This cannot establish 'stability against noise'. In addition, Table V reports maximum matched distances of 0.9175–0.9901 at R=36, and no null model is given for what the distance distribution would be under random local minima or random matching. The paper's own Section IX states that semantic analysis tools are lacking. The abstract's claim that nonidentifiability is 'effectively resolved' is therefore not supported by the presented evidence.
- [Section IV-A2 and Table II] There is an internal inconsistency in the noise-stability claim for the rank estimator. The text states 'Rc(0.05) = 46 > Rc(0) = 30' for UTK, but Table II reports for UTK with ξ=0.05 the values R=22 (τ=10^{-4}) and R=28 (τ=10^{-5}). Since the relation between Rc(ξ) and R(ξ,τ) is not defined, the reader cannot verify whether Rc increases or decreases with noise. The definition of Rc for noisy data needs to be made precise, and the seed variability of Rc should be reported.
- [Section V-B, Eq. (23)] The inequality S_i ≤ S-hat_i is stated as a result but no proof or empirical support is given. This inequality is not true for arbitrary approximations; for a mixture, conditional entropy can increase or decrease depending on the component distributions and weights. If it is meant to hold for NMF local minima, a proof or a precise statement of the conditions is needed. If it is an empirical observation, the supporting experiments should be shown. As written, it is an unsupported formal claim in the section that motivates the approximate-PCC interpretation.
- [Section IX] The paper itself limits the central claim: it states that different local minima 'do not necessarily mean that these different basis images refer to different semantic features' and that semantic analysis tools are still under development. This directly contradicts the abstract and Section VI, which present the nonidentifiability problem as effectively resolved. To support 'effective resolution', the authors need either a semantic evaluation of the outlier basis-image pairs or a quantitative bound on the proportion of unstable basis images. The current mean/median matched distances are not sufficient.
minor comments (4)
- [Table II caption] The dataset dimensions are inconsistent with Section III-A: Swimmer is given as (M=256, N=132) although images are 13x13=169 pixels; Olivetti is written as (M=400, M=642) with a duplicated M; UTK is given as (M=1500, N=502) whereas the text says 50x50=2500 pixels. These notation errors should be corrected.
- [Figure 4 caption vs Table V] The caption of Figure 4 reports mean and median matched cosine distances of 0.0236 and 0.0615 for R=36, while Table V for Fig. 5 at R=36 gives mean 0.1056 and median 0.0638. The captions also differ in whether the two halves were trained with the same or different seeds. This discrepancy needs clarification.
- [Section IV-A] The phrase 'the fraction of valid inequalities converges to 1' is vague; the threshold used to define R_c should be stated explicitly. The notation R_c, R(ξ,τ), and R_c(ξ) should be unified across the text and tables.
- [Section VI] The histograms in Figures 5 and 6 and the summary statistics in Table V would be more informative if the fraction of pairs with distance above a threshold (e.g., 0.5) were reported, together with a null distribution obtained by matching basis images from unrelated NMF runs or from random nonnegative matrices.
Circularity Check
No significant circularity: the rank criterion is an empirical threshold derived from an independent inequality, and the stability conclusions are measured rather than constructed.
full rationale
The central rank estimate Rc is not a fitted parameter that is then renamed as a prediction. It is an empirical threshold obtained from inequality (16), which the paper proves from the exact factorization (12); the NMF objective (3)/(6) does not optimize (16), so the threshold is not forced by construction. The stability analysis in Section VI compares basis images from independent data halves, noise realizations, or different seeds, and the reported distances are measured outcomes, not parameters chosen to make the conclusion true. The approximate-PCC correlations in Tables III and IV are empirical properties of the fitted NMF, not quantities fitted to match the anticorrelation claim. Self-citations (refs. [15], [21]) are contextual and non-load-bearing: they are not used to justify the rank criterion, the stability result, or the approximate-PCC findings. Section IX openly acknowledges the absence of semantic analysis tools, which is an evidential limitation but not a circularity. No equation in the paper reduces a predicted result to its own input.
Assumptions & free parameters
free parameters (3)
- tau tolerance for noisy rank =
e.g. 1e-4 and 1e-5 for UTK/Olivetti; 1e-2 and 1e-3 for Swimmer
- rank R=36 for UTK stability experiments =
36
- number of cluster images per basis (top-5) =
5
assumptions (4)
- domain assumption Alternating least-squares local minima found by scikit-learn are sufficiently representative of NMF solutions for the studied data.
- domain assumption The marginal-sum conservation (7) holds accurately enough for Frobenius NMF to justify the probabilistic interpretation of the factors.
- ad hoc to paper Entropy inequality S_i <= Shat_i for every image i.
- domain assumption Gray-scale images normalized to joint probabilities adequately model the common-cause scenario.
Cite this review
Pith. "Pith review of Nonnegative matrix factorization and the principle of the common cause." pith.science (2026). https://pith.science/paper/Z3RKUJNJ
@misc{pith2026250903652,
author = {Pith},
title = {Pith review of: Nonnegative matrix factorization and the principle of the common cause},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z3RKUJNJ}},
note = {Machine review of arXiv:2509.03652}
}
read the original abstract
Nonnegative matrix factorization (NMF) is a known unsupervised data-reduction method. The principle of the common cause (PCC) is a basic methodological approach in probabilistic causality, which seeks an independent mixture model for the joint probability of two dependent random variables. It turns out that these two concepts are closely related. This relationship is explored reciprocally for several datasets of gray-scale images, which are conveniently mapped into probability models. On one hand, PCC provides a predictability tool that leads to a robust estimation of the effective rank of NMF. Unlike other estimates (e.g., those based on the Bayesian Information Criteria), our estimate of the rank is stable against weak noise. We show that NMF implemented around this rank produces features (basis images) that are also stable against noise and against seeds of local optimization, thereby effectively resolving the NMF nonidentifiability problem. On the other hand, NMF provides an interesting possibility of implementing PCC in an approximate way, where larger and positively correlated joint probabilities tend to be explained better via the independent mixture model. We work out a clustering method, where data points with the same common cause are grouped into the same cluster. We also show how NMF can be employed for data denoising.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
P. Paatero and U. Tapper, “Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values,” Environmetrics, vol. 5, no. 2, pp. 111–126, 1994
work page 1994
-
[2]
Learning the parts of objects by non- negative matrix factorization,
D. D. Lee and H. S. Seung, “Learning the parts of objects by non- negative matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791, 1999
1999
-
[3]
Gillis, Nonnegative Matrix Factorization
N. Gillis, Nonnegative Matrix Factorization. SIAM, 2021
work page 2021
-
[4]
On the complexity of nonnegative matrix factorization,
S. A. Vavasis, “On the complexity of nonnegative matrix factorization,” SIAM journal on optimization , vol. 20, no. 3, pp. 1364–1377, 2010
work page 2010
-
[5]
Assessing methods for evaluating the number of components in non-negative matrix factorization,
J. M. Maisog, A. T. DeMarco, K. Devarajan, S. Young, P. Fogel, and G. Luta, “Assessing methods for evaluating the number of components in non-negative matrix factorization,” Mathematics, vol. 9, no. 22, p. 2840, 2021
work page 2021
-
[6]
Rank selection in nonnegative matrix factorization using minimum description length,
S. Squires, A. Pr ¨ugel-Bennett, and M. Niranjan, “Rank selection in nonnegative matrix factorization using minimum description length,” Neural computation, vol. 29, no. 8, pp. 2164–2176, 2017
work page 2017
-
[7]
Reichenbach, The direction of time
H. Reichenbach, The direction of time . University of California Press, 1956, vol. 65
work page 1956
-
[8]
Suppes, A probabilistic theory of causality
P. Suppes, A probabilistic theory of causality . North-Holland, Amster- dam, 1970
work page 1970
Show all 35 references
-
[9]
Peters, D
J. Peters, D. Janzing, and B. Sch ¨olkopf, Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017
2017
-
[10]
Colloquium: Bell’s theorem and locally mediated reformulations of quantum mechanics,
K. B. Wharton and N. Argaman, “Colloquium: Bell’s theorem and locally mediated reformulations of quantum mechanics,” Reviews of Modern Physics, vol. 92, no. 2, p. 021002, 2020
2020
-
[11]
The direction of time,
O. Penrose and I. C. Percival, “The direction of time,” Proceedings of the Physical Society , vol. 79, no. 3, p. 605, 1962
1962
-
[12]
A causal-model theory of conceptual representation and cat- egorization
B. Rehder, “A causal-model theory of conceptual representation and cat- egorization.” Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 29, no. 6, p. 1141, 2003
2003
-
[13]
Nonnegative matrix factorization: an analytical and interpretive tool in computational biology,
K. Devarajan, “Nonnegative matrix factorization: an analytical and interpretive tool in computational biology,”PLoS computational biology, vol. 4, no. 7, p. e1000029, 2008
2008
-
[14]
Common cause explanation,
E. Sober, “Common cause explanation,” Philosophy of Science, vol. 51, no. 2, pp. 212–241, 1984
1984
-
[15]
Resolution of simpson’s para- dox via the common cause principle,
A. Hovhannisyan and A. Allahverdyan, “Resolution of simpson’s para- dox via the common cause principle,” arXiv preprint arXiv:2403.00957, 2024
2024 arXiv
-
[16]
Image denoising review: From classical to state-of-the-art approaches,
B. Goyal, A. Dogra, S. Agrawal, B. S. Sohi, and A. Sharma, “Image denoising review: From classical to state-of-the-art approaches,” Infor- mation fusion, vol. 55, pp. 220–244, 2020
2020
-
[17]
Non-negative matrix factorization with fixed row and column sums,
N.-D. Ho and P. Van Dooren, “Non-negative matrix factorization with fixed row and column sums,” Linear Algebra and its Applications , vol. 429, no. 5-6, pp. 1020–1025, 2008
2008
-
[18]
Scikit-learn: Machine learning in python,
F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and ´E. Duch- esnay, “Scikit-learn: Machine learning in python,” Journal of Machine...
2011
-
[19]
Nonnegative ranks, decompositions, and factorizations of nonnegative matrices,
J. E. Cohen and U. G. Rothblum, “Nonnegative ranks, decompositions, and factorizations of nonnegative matrices,” Linear Algebra and its Applications, vol. 190, pp. 149–168, 1993
1993
-
[20]
When does non-negative matrix factor- ization give a correct decomposition into parts?
D. Donoho and V . Stodden, “When does non-negative matrix factor- ization give a correct decomposition into parts?” Advances in neural information processing systems , vol. 16, 2003
2003
-
[21]
The most likely common cause,
A. Hovhannisyan and A. E. Allahverdyan, “The most likely common cause,” International Journal of Approximate Reasoning , vol. 173, p. 109264, 2024
2024
-
[22]
Causal feature selection,
I. Guyon, C. Aliferis et al., “Causal feature selection,” in Computational methods of feature selection . Chapman and Hall/CRC, 2007, pp. 79– 102
2007
-
[23]
A unified view of causal and non-causal feature selection,
K. Yu, L. Liu, and J. Li, “A unified view of causal and non-causal feature selection,” ACM Transactions on Knowledge Discovery from Data (TKDD), vol. 15, no. 4, pp. 1–46, 2021
2021
-
[24]
Generalised reichenbachian common cause systems,
C. Mazzola, “Generalised reichenbachian common cause systems,” Synthese, vol. 196, no. 10, pp. 4185–4209, 2019
2019
-
[25]
Horwich, Asymmetries in time: Problems in the philosophy of science
P. Horwich, Asymmetries in time: Problems in the philosophy of science. MIT press, 1987
1987
-
[26]
The principle of the common cause faces the bernstein paradox,
J. Uffink, “The principle of the common cause faces the bernstein paradox,” Philosophy of Science, vol. 66, no. S3, pp. S512–S525, 1999
1999
-
[27]
Imprecise bayesian networks as causal models,
D. Kinney, “Imprecise bayesian networks as causal models,” Informa- tion, vol. 9, no. 9, p. 211, 2018
2018
-
[28]
The database of faces,
AT&T Laboratories Cambridge, “The database of faces,” https://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html, 1994
1994
-
[29]
Age progression/regression by condi- tional adversarial autoencoder,
Z. Zhang, Y . Song, and H. Qi, “Age progression/regression by condi- tional adversarial autoencoder,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2017
2017
-
[30]
Utkface,
Jangedoo, “Utkface,” https://www.kaggle.com/datasets/jangedoo/utkface- new
-
[31]
Model-order selection: a review of information criterion rules,
P. Stoica and Y . Selen, “Model-order selection: a review of information criterion rules,” IEEE signal processing magazine , vol. 21, no. 4, pp. 36–47, 2004
2004
-
[32]
On the derivation of the bayesian information criterion,
H. S. Bhat and N. Kumar, “On the derivation of the bayesian information criterion,” School of Natural Sciences, University of California , vol. 99, no. 4, 2010
2010
-
[33]
Weight of evidence: A brief survey,
I. Good, “Weight of evidence: A brief survey,” Bayesian statistics, vol. 2, pp. 249–270, 1985
1985
-
[34]
Scipy 1.0: Fundamental algorithms for scientific computing in python,
P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright et al., “Scipy 1.0: Fundamental algorithms for scientific computing in python,” Nature Methods, vol. 17, no. 3, pp. 261–272, 2020
2020
-
[35]
Causal feature learning: an overview,
K. Chalupka, F. Eberhardt, and P. Perona, “Causal feature learning: an overview,” Behaviormetrika, vol. 44, pp. 137–164, 2017
2017
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.