Pith. sign in

REVIEW 4 major objections 4 minor 35 references

Nonnegative matrix factorization and the principle of the common cause

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read The principle of the common cause picks the rank where NMF features become stable.

desk verdict A promising rank-selection heuristic for NMF, but the evidence doesn't support the 'resolves nonidentifiability' claim. read the letter →

arxiv 2509.03652 v1 pith:Z3RKUJNJ submitted 2025-09-03 cs.LG cs.AIphysics.data-anstat.ML

classification cs.LGcs.AIphysics.data-anstat.ML
keywords nonnegativematrixfactorizationprincipleofthecommoncauseeffectiverankselectionbasisimagestabilitynonidentifiabilityconditionalindependenceclusteringdenoising
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Nonnegative matrix factorization and the principle of the common cause turn out to describe the same operation: explaining a matrix of probabilities through a small set of hidden causes. The paper uses a predictability inequality inherited from the common-cause principle to choose the effective rank of NMF, and shows that basis images learned at that rank are stable under noise and under changes of the random seed—so the usual nonidentifiability of NMF does not block interpretable feature extraction. Conversely, NMF supplies an approximate version of the common-cause principle: larger and positively correlated joint probabilities are explained better by the independent mixture, which yields a clustering rule and a denoising procedure. If the stability claim holds, the practical takeaway is that a data-driven rank choice can make NMF features meaningful despite the factorization's known non-uniqueness.

What carries the argument

The engine is the predictability inequality (16), proved from exact conditional independence: for each pair of pixel and image there exist basis images b1 and b2 such that p(π|b1) ≤ p(π|i) ≤ p(π|b2). The smallest rank at which this inequality holds for essentially all pixels defines the effective rank Rc. Stability is then assessed by comparing two sets of basis images with cosine distances and optimal one-to-one matching; the regime around Rc is where matched distances stay small, while for much larger ranks the basis images become unstable.

What would settle it

Run the same stability protocol on a synthetic dataset with known ground-truth nonnegative rank and known generating parts: if basis images around Rc fail to recover those parts under the paper's noise model, or if the matched cosine distances are no better than distances obtained by matching random basis-image pairs, the rank/stability claim would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a predictive inequality derived from the principle of the common cause—for every pixel and image, some basis image predicts the pixel at least as strongly as the image itself, in both directions—gives a workable way to choose the effective rank of a nonnegative matrix factorization. Around the resulting rank Rc, the basis images found by local optimization are stable under bit-flip noise and under changes of the random seed, so the nonidentifiability of NMF does not prevent reliable feature extraction. The paper also claims that NMF implements an approximate form of the common-cause principle: the higher and more positively correlated joint probabilities are

Load-bearing premise

The stability conclusion rests on treating the few computed local minima (one seed pairing or one noise draw, matched optimally) as representative of all NMF solutions; if those minima are unrepresentative, the claim that nonidentifiability is effectively resolved does not follow.

Editorial extensions

If this is right

  • Around the common-cause rank Rc, NMF basis images can be treated as interpretable features despite the factorization's non-uniqueness, because they survive noise and seed changes.
  • The PCC-based rank estimate remains sensible under weak noise in cases where Bayesian Information Criterion based estimates fail to show a minimum.
  • NMF gives an approximate common-cause principle: larger, positively correlated probabilities are explained better, which can ground clustering by shared causes.
  • NMF denoises when the rank lies in a middle interval [R1, R2]; below it no reliable features form, and above it the model overfits noise.
  • Reconstructed images have per-image entropy at least as large as the original conditional distributions, and basis images become sparser than images once the rank exceeds a small threshold.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the stability result transfers to other nonnegative data matrices (not just gray-scale images), the same rank-selection rule could serve as a default hyperparameter choice for NMF in applications where basis interpretability matters.
  • Beyond the paper: the approximate common-cause principle suggests a testable design criterion for NMF objectives—explicitly upweighting large positive joint probabilities should reproduce or improve the observed anticorrelation between approximation error and probability size.
  • Beyond the paper: the nonmonotonic distance distribution seen when changing seeds at different ranks is an open clue; a systematic study with many seeds and chance-level baselines could show whether the apparent stability around Rc is a true geometric property or a selection effect.
  • Beyond the paper: because the rank estimate is derived from a conditional-independence condition, it may also apply to other latent-variable models beyond matrix factorization, such as topic models or nonnegative tensor factorizations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a conceptual and methodological connection between nonnegative matrix factorization (NMF) and Reichenbach's principle of the common cause (PCC). It derives a predictability inequality from exact NMF/PCC and uses its empirical satisfaction to define an effective rank R_c, claiming this estimate is stable against weak noise in contrast to BIC. The paper further claims that NMF at ranks around R_c produces basis images that are stable against noise and against the seed of local optimization, thereby effectively resolving the NMF nonidentifiability problem. Additional contributions include an approximate-PCC interpretation with correlation inequalities and entropy relations, a clustering method based on common-cause weights, and denoising experiments comparing NMF with PCA.

Significance. If the central claims were fully substantiated, the paper would offer a practical rank-selection criterion and a resolution of NMF nonidentifiability, which are longstanding practical problems. The exact derivation of inequality (16) is clean, and the conceptual link between NMF and PCC is interesting and potentially fruitful. The BIC comparison and the denoising experiments, including the random-accuracy baseline, are useful additions. However, the current evidence for the strongest claims is incomplete: the noise-stability experiment is confounded by a data split, no chance baseline is provided for the matched cosine distances, and the paper itself acknowledges missing semantic analysis. The significance is therefore conditional on substantial revision.

major comments (4)
  1. [Section VI] The noise-stability experiment is confounded: the clean and noisy NMF runs are performed on two different halves of UTK, so any distance between the matched basis images reflects not only noise but also the difference between two independent subsets of data. This cannot establish 'stability against noise'. In addition, Table V reports maximum matched distances of 0.9175–0.9901 at R=36, and no null model is given for what the distance distribution would be under random local minima or random matching. The paper's own Section IX states that semantic analysis tools are lacking. The abstract's claim that nonidentifiability is 'effectively resolved' is therefore not supported by the presented evidence.
  2. [Section IV-A2 and Table II] There is an internal inconsistency in the noise-stability claim for the rank estimator. The text states 'Rc(0.05) = 46 > Rc(0) = 30' for UTK, but Table II reports for UTK with ξ=0.05 the values R=22 (τ=10^{-4}) and R=28 (τ=10^{-5}). Since the relation between Rc(ξ) and R(ξ,τ) is not defined, the reader cannot verify whether Rc increases or decreases with noise. The definition of Rc for noisy data needs to be made precise, and the seed variability of Rc should be reported.
  3. [Section V-B, Eq. (23)] The inequality S_i ≤ S-hat_i is stated as a result but no proof or empirical support is given. This inequality is not true for arbitrary approximations; for a mixture, conditional entropy can increase or decrease depending on the component distributions and weights. If it is meant to hold for NMF local minima, a proof or a precise statement of the conditions is needed. If it is an empirical observation, the supporting experiments should be shown. As written, it is an unsupported formal claim in the section that motivates the approximate-PCC interpretation.
  4. [Section IX] The paper itself limits the central claim: it states that different local minima 'do not necessarily mean that these different basis images refer to different semantic features' and that semantic analysis tools are still under development. This directly contradicts the abstract and Section VI, which present the nonidentifiability problem as effectively resolved. To support 'effective resolution', the authors need either a semantic evaluation of the outlier basis-image pairs or a quantitative bound on the proportion of unstable basis images. The current mean/median matched distances are not sufficient.
minor comments (4)
  1. [Table II caption] The dataset dimensions are inconsistent with Section III-A: Swimmer is given as (M=256, N=132) although images are 13x13=169 pixels; Olivetti is written as (M=400, M=642) with a duplicated M; UTK is given as (M=1500, N=502) whereas the text says 50x50=2500 pixels. These notation errors should be corrected.
  2. [Figure 4 caption vs Table V] The caption of Figure 4 reports mean and median matched cosine distances of 0.0236 and 0.0615 for R=36, while Table V for Fig. 5 at R=36 gives mean 0.1056 and median 0.0638. The captions also differ in whether the two halves were trained with the same or different seeds. This discrepancy needs clarification.
  3. [Section IV-A] The phrase 'the fraction of valid inequalities converges to 1' is vague; the threshold used to define R_c should be stated explicitly. The notation R_c, R(ξ,τ), and R_c(ξ) should be unified across the text and tables.
  4. [Section VI] The histograms in Figures 5 and 6 and the summary statistics in Table V would be more informative if the fraction of pairs with distance above a threshold (e.g., 0.5) were reported, together with a null distribution obtained by matching basis images from unrelated NMF runs or from random nonnegative matrices.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the rank criterion is an empirical threshold derived from an independent inequality, and the stability conclusions are measured rather than constructed.

full rationale

The central rank estimate Rc is not a fitted parameter that is then renamed as a prediction. It is an empirical threshold obtained from inequality (16), which the paper proves from the exact factorization (12); the NMF objective (3)/(6) does not optimize (16), so the threshold is not forced by construction. The stability analysis in Section VI compares basis images from independent data halves, noise realizations, or different seeds, and the reported distances are measured outcomes, not parameters chosen to make the conclusion true. The approximate-PCC correlations in Tables III and IV are empirical properties of the fitted NMF, not quantities fitted to match the anticorrelation claim. Self-citations (refs. [15], [21]) are contextual and non-load-bearing: they are not used to justify the rank criterion, the stability result, or the approximate-PCC findings. Section IX openly acknowledges the absence of semantic analysis tools, which is an evidential limitation but not a circularity. No equation in the paper reduces a predicted result to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are postulated. The main ledger items are experimental conventions (tau, chosen ranks, top-5 clustering) and implicit or unproved regularities (representative local minima, marginal-sum accuracy, entropy inequality). The central conceptual apparatus, common cause and conditional independence, is inherited from prior literature.

free parameters (3)
  • tau tolerance for noisy rank = e.g. 1e-4 and 1e-5 for UTK/Olivetti; 1e-2 and 1e-3 for Swimmer
    R(xi, tau) depends strongly on tau; the choice of threshold is user-specified and changes the reported effective rank.
  • rank R=36 for UTK stability experiments = 36
    Chosen as close to Rc(0)=30 and Rc(0.25)=46; results differ at other ranks (72 and 108).
  • number of cluster images per basis (top-5) = 5
    Cluster visualization in Fig. 7 uses the top-5 values of p(i|b), an arbitrary choice; the clustering demonstration depends partly on it.
assumptions (4)
  • domain assumption Alternating least-squares local minima found by scikit-learn are sufficiently representative of NMF solutions for the studied data.
    All rank and stability conclusions are based on these local minima; no global optimality or comprehensive multi-restart analysis is given. Used throughout Sections IV and VI.
  • domain assumption The marginal-sum conservation (7) holds accurately enough for Frobenius NMF to justify the probabilistic interpretation of the factors.
    Section II-A states this with 'a precision much larger' but does not quantify; the probabilistic mapping (9)-(11) depends on it.
  • ad hoc to paper Entropy inequality S_i <= Shat_i for every image i.
    Section V-B states this without proof; it is not implied by the mixture representation for arbitrary approximations and appears potentially false in general.
  • domain assumption Gray-scale images normalized to joint probabilities adequately model the common-cause scenario.
    Section III maps images to probabilities; the causal interpretation of basis images as common causes is asserted, not inferred from interventions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nonnegative matrix factorization and the principle of the common cause." pith.science (2026). https://pith.science/paper/Z3RKUJNJ

@misc{pith2026250903652,
  author       = {Pith},
  title        = {Pith review of: Nonnegative matrix factorization and the principle of the common cause},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3RKUJNJ}},
  note         = {Machine review of arXiv:2509.03652}
}
read the original abstract

Nonnegative matrix factorization (NMF) is a known unsupervised data-reduction method. The principle of the common cause (PCC) is a basic methodological approach in probabilistic causality, which seeks an independent mixture model for the joint probability of two dependent random variables. It turns out that these two concepts are closely related. This relationship is explored reciprocally for several datasets of gray-scale images, which are conveniently mapped into probability models. On one hand, PCC provides a predictability tool that leads to a robust estimation of the effective rank of NMF. Unlike other estimates (e.g., those based on the Bayesian Information Criteria), our estimate of the rank is stable against weak noise. We show that NMF implemented around this rank produces features (basis images) that are also stable against noise and against seeds of local optimization, thereby effectively resolving the NMF nonidentifiability problem. On the other hand, NMF provides an interesting possibility of implementing PCC in an approximate way, where larger and positively correlated joint probabilities tend to be explained better via the independent mixture model. We work out a clustering method, where data points with the same common cause are grouped into the same cluster. We also show how NMF can be employed for data denoising.

Figures

Figures reproduced from arXiv: 2509.03652 by the authors.

Figure 1
Figure 1. Two images from UTK dataset [see section III-A]: the original [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The average distance D¯ between basis images [see (18)] vs. the rank R for three datasets: Swimmer, Olivetti and UTK; see section IV-A. For each rank, D¯ is evaluated for 10 different seeds and the mean is taken. For D¯(UTK) we employed a random sample of 1500 images from UTK. For each dataset we show the estimated Rc for clean datasets; see Table I. B. Mean internal distance Here is an interesting structural aspect… view at source ↗
Figure 3
Figure 3. For clean and noisy datasets (Swimmer, Olivetti, UTK) we implemented 3 versions of BIC (Bayesian Information Criterion) that are defined and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: For each pair of pictures, the left image is a basis image from applying [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The same situation as in Fig. 4 (see the discussion around (26)), but now we show the histograms of the distribution of optimally matched distances [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The same situation as in Fig. 5, but now noise is absent, and we compare (for each rank) the basis images generated via two different seeds applied to [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Visualization of natural clusterization via NMF for Olivetti dataset [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 5
Figure 5. Figure 5: shows the distribution of {Ca aˆ(a)} R a=1 for various values of R. The stability is gradually lost once R changes to R = 72. For R = 108, the number of close and far away basis images is comparable to each other [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 8
Figure 8. Figure 8: Recovery of a UTK image via NMF with rank [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Denoising accuracy comparison for Swimmer and UTK datasets. Two [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 34 canonical work pages

  1. [1]

    Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values,

    P. Paatero and U. Tapper, “Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values,” Environmetrics, vol. 5, no. 2, pp. 111–126, 1994

  2. [2]

    Learning the parts of objects by non- negative matrix factorization,

    D. D. Lee and H. S. Seung, “Learning the parts of objects by non- negative matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791, 1999

  3. [3]

    Gillis, Nonnegative Matrix Factorization

    N. Gillis, Nonnegative Matrix Factorization. SIAM, 2021

  4. [4]

    On the complexity of nonnegative matrix factorization,

    S. A. Vavasis, “On the complexity of nonnegative matrix factorization,” SIAM journal on optimization , vol. 20, no. 3, pp. 1364–1377, 2010

  5. [5]

    Assessing methods for evaluating the number of components in non-negative matrix factorization,

    J. M. Maisog, A. T. DeMarco, K. Devarajan, S. Young, P. Fogel, and G. Luta, “Assessing methods for evaluating the number of components in non-negative matrix factorization,” Mathematics, vol. 9, no. 22, p. 2840, 2021

  6. [6]

    Rank selection in nonnegative matrix factorization using minimum description length,

    S. Squires, A. Pr ¨ugel-Bennett, and M. Niranjan, “Rank selection in nonnegative matrix factorization using minimum description length,” Neural computation, vol. 29, no. 8, pp. 2164–2176, 2017

  7. [7]

    Reichenbach, The direction of time

    H. Reichenbach, The direction of time . University of California Press, 1956, vol. 65

  8. [8]

    Suppes, A probabilistic theory of causality

    P. Suppes, A probabilistic theory of causality . North-Holland, Amster- dam, 1970

Show all 35 references
  1. [9]

    Peters, D

    J. Peters, D. Janzing, and B. Sch ¨olkopf, Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017

  2. [10]

    Colloquium: Bell’s theorem and locally mediated reformulations of quantum mechanics,

    K. B. Wharton and N. Argaman, “Colloquium: Bell’s theorem and locally mediated reformulations of quantum mechanics,” Reviews of Modern Physics, vol. 92, no. 2, p. 021002, 2020

  3. [11]

    The direction of time,

    O. Penrose and I. C. Percival, “The direction of time,” Proceedings of the Physical Society , vol. 79, no. 3, p. 605, 1962

  4. [12]

    A causal-model theory of conceptual representation and cat- egorization

    B. Rehder, “A causal-model theory of conceptual representation and cat- egorization.” Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 29, no. 6, p. 1141, 2003

  5. [13]

    Nonnegative matrix factorization: an analytical and interpretive tool in computational biology,

    K. Devarajan, “Nonnegative matrix factorization: an analytical and interpretive tool in computational biology,”PLoS computational biology, vol. 4, no. 7, p. e1000029, 2008

  6. [14]

    Common cause explanation,

    E. Sober, “Common cause explanation,” Philosophy of Science, vol. 51, no. 2, pp. 212–241, 1984

  7. [15]

    Resolution of simpson’s para- dox via the common cause principle,

    A. Hovhannisyan and A. Allahverdyan, “Resolution of simpson’s para- dox via the common cause principle,” arXiv preprint arXiv:2403.00957, 2024

  8. [16]

    Image denoising review: From classical to state-of-the-art approaches,

    B. Goyal, A. Dogra, S. Agrawal, B. S. Sohi, and A. Sharma, “Image denoising review: From classical to state-of-the-art approaches,” Infor- mation fusion, vol. 55, pp. 220–244, 2020

  9. [17]

    Non-negative matrix factorization with fixed row and column sums,

    N.-D. Ho and P. Van Dooren, “Non-negative matrix factorization with fixed row and column sums,” Linear Algebra and its Applications , vol. 429, no. 5-6, pp. 1020–1025, 2008

  10. [18]

    Scikit-learn: Machine learning in python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and ´E. Duch- esnay, “Scikit-learn: Machine learning in python,” Journal of Machine...

  11. [19]

    Nonnegative ranks, decompositions, and factorizations of nonnegative matrices,

    J. E. Cohen and U. G. Rothblum, “Nonnegative ranks, decompositions, and factorizations of nonnegative matrices,” Linear Algebra and its Applications, vol. 190, pp. 149–168, 1993

  12. [20]

    When does non-negative matrix factor- ization give a correct decomposition into parts?

    D. Donoho and V . Stodden, “When does non-negative matrix factor- ization give a correct decomposition into parts?” Advances in neural information processing systems , vol. 16, 2003

  13. [21]

    The most likely common cause,

    A. Hovhannisyan and A. E. Allahverdyan, “The most likely common cause,” International Journal of Approximate Reasoning , vol. 173, p. 109264, 2024

  14. [22]

    Causal feature selection,

    I. Guyon, C. Aliferis et al., “Causal feature selection,” in Computational methods of feature selection . Chapman and Hall/CRC, 2007, pp. 79– 102

  15. [23]

    A unified view of causal and non-causal feature selection,

    K. Yu, L. Liu, and J. Li, “A unified view of causal and non-causal feature selection,” ACM Transactions on Knowledge Discovery from Data (TKDD), vol. 15, no. 4, pp. 1–46, 2021

  16. [24]

    Generalised reichenbachian common cause systems,

    C. Mazzola, “Generalised reichenbachian common cause systems,” Synthese, vol. 196, no. 10, pp. 4185–4209, 2019

  17. [25]

    Horwich, Asymmetries in time: Problems in the philosophy of science

    P. Horwich, Asymmetries in time: Problems in the philosophy of science. MIT press, 1987

  18. [26]

    The principle of the common cause faces the bernstein paradox,

    J. Uffink, “The principle of the common cause faces the bernstein paradox,” Philosophy of Science, vol. 66, no. S3, pp. S512–S525, 1999

  19. [27]

    Imprecise bayesian networks as causal models,

    D. Kinney, “Imprecise bayesian networks as causal models,” Informa- tion, vol. 9, no. 9, p. 211, 2018

  20. [28]

    The database of faces,

    AT&T Laboratories Cambridge, “The database of faces,” https://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html, 1994

  21. [29]

    Age progression/regression by condi- tional adversarial autoencoder,

    Z. Zhang, Y . Song, and H. Qi, “Age progression/regression by condi- tional adversarial autoencoder,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2017

  22. [30]

    Utkface,

    Jangedoo, “Utkface,” https://www.kaggle.com/datasets/jangedoo/utkface- new

  23. [31]

    Model-order selection: a review of information criterion rules,

    P. Stoica and Y . Selen, “Model-order selection: a review of information criterion rules,” IEEE signal processing magazine , vol. 21, no. 4, pp. 36–47, 2004

  24. [32]

    On the derivation of the bayesian information criterion,

    H. S. Bhat and N. Kumar, “On the derivation of the bayesian information criterion,” School of Natural Sciences, University of California , vol. 99, no. 4, 2010

  25. [33]

    Weight of evidence: A brief survey,

    I. Good, “Weight of evidence: A brief survey,” Bayesian statistics, vol. 2, pp. 249–270, 1985

  26. [34]

    Scipy 1.0: Fundamental algorithms for scientific computing in python,

    P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright et al., “Scipy 1.0: Fundamental algorithms for scientific computing in python,” Nature Methods, vol. 17, no. 3, pp. 261–272, 2020

  27. [35]

    Causal feature learning: an overview,

    K. Chalupka, F. Eberhardt, and P. Perona, “Causal feature learning: an overview,” Behaviormetrika, vol. 44, pp. 137–164, 2017

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.