Pith. sign in

REVIEW 2 major objections 4 minor 26 references

The Forward-Forward algorithm's squared goodness is not a heuristic: it is the sufficient statistic of a likelihood-ratio test whose threshold is the optimal decision boundary.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:28 UTC pith:PPNZIQWG

load-bearing objection The likelihood-ratio derivation of Forward-Forward goodness is real and worth engaging, but the paper's empirical confirmation is shakier than the prose admits—the 'independent' GSM tail-index check is the same kurtosis statistic computed twice. the 2 major comments →

arxiv 2607.12501 v2 pith:PPNZIQWG submitted 2026-07-14 cs.LG eess.IVstat.ML

What Does Goodness Measure? A Likelihood-Ratio Account of Forward-Forward Learning

Paolo Giannitrapani This is my paper
classification cs.LG eess.IVstat.ML
keywords Forward-Forward learninggoodness functionlikelihood-ratio testsufficient statisticMahalanobis quadraticGaussian scale mixturedivisive normalizationlocal learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the two central design choices of the Forward-Forward (FF) algorithm—the squared-activation goodness and the threshold that separates positive from negative inputs—are not heuristics but derived objects of statistical decision theory. Under a generative model where pre-activations are zero-mean Gaussians differing only in scale, the log-likelihood ratio is an affine function of the squared norm, so the goodness is the sufficient statistic and the threshold is its Neyman–Pearson boundary. The account generalizes in closed form: anisotropic populations call for a Mahalanobis quadratic, and heavy-tailed (Gaussian scale mixture) populations call for a saturating statistic whose slope is a posterior precision, i.e., divisive normalization. The paper also derives the correct inter-layer normalization and identifies a scale-inflation shortcut in the pairwise objective. A sympathetic reader would care because it converts two unexplained heuristics into measurable, falsifiable quantities with principled generalizations.

Core claim

Proposition 1 states that if positive and negative pre-activations are isotropic zero-mean Gaussians with σ_+ > σ_-, the log-likelihood ratio is ℓ(z) = ½(1/σ_-² − 1/σ_+²)‖z‖² + (d/2) log(σ_-²/σ_+²); hence ‖z‖² is a sufficient statistic and the optimal rule is 'positive iff ‖z‖² > τ' with τ = −β/α. The paper's central discovery is that the FF algorithm's squared goodness is exactly this sufficient statistic, and the FF threshold is the decision boundary, turning the usual logistic reading of goodness from an assumption into a theorem. It then shows that relaxing isotropy yields the Mahalanobis goodness z^T M z with M = Σ_-^{-1} − Σ_+^{-1}, and relaxing Gaussianity yields a saturating GSM like

What carries the argument

The load-bearing object is the class-conditional likelihood ratio between positive and negative pre-activations. Under the isotropic Gaussian model it is affine in the squared norm (Proposition 1), which makes the goodness a sufficient statistic; under anisotropic Gaussians it becomes the indefinite quadratic form z^T(Σ_-^{-1}−Σ_+^{-1})z; under Gaussian scale mixtures it becomes h(q) = log F_+(q) − log F_-(q), a saturating function of the Mahalanobis energy q whose slope is the posterior mean precision of the latent scale. The Neyman–Pearson lemma supplies optimality; the monotone likelihood-ratio property supplies the monotonicity of the statistic; and the same latent scale is the condition

Load-bearing premise

The derivation assumes a layer's pre-activations are independent draws from a single class-conditional Gaussian or Gaussian scale mixture; in a convolutional layer, spatial locations share receptive fields and are strongly correlated, and if that dependence is large the derived optimality, the fitted statistics, and the aggregation predictions may be miscalibrated.

What would settle it

Measure the covariance of pre-activations across spatial locations in a trained convolutional FF layer, and compare the empirically observed aggregation gain of the saturating likelihood-ratio sum over the energy sum with the i.i.d.-derived prediction (e.g., +0.195 AUC at ν=3, K=64); under strong spatial dependence the predicted gain should shrink or vanish, falsifying the independence premise of Proposition 3(ii).

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The FF threshold is no longer a hyperparameter: for the isotropic model it is set by the scale ratio via τ = d log(σ_+²/σ_-²)/(σ_-^{-2} − σ_+^{-2}).
  • The plain squared goodness is optimal only in the isotropic pure-scale regime; when the class difference is structural, the Mahalanobis quadratic recovers discriminative signal the plain square discards.
  • Under heavy-tailed activations, each location contributes at most ν log r nats of evidence, so aggregating per-location likelihoods rather than energies is the theoretically correct readout and the advantage grows with tail heaviness and number of locations.
  • The normalization between layers should remove the length while preserving per-coordinate energy; unit-norm normalization predicts a depth collapse of goodness as width grows, which the paper observes.
  • The pairwise FF objective has a scale-inflation shortcut—loss decreases along weight-inflation rays—and a whitened, negative-metric goodness gauges it out, stabilizing training dynamics without (so far) changing representation quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the same likelihood-ratio lens should apply to any scalar local objective that contrasts two activation populations, so the paper's derivations may transfer to other local learning rules whose goals are currently empirical.
  • Our inference: if the spatial independence assumption fails in real convolutional layers, the aggregation advantage of the saturating statistic could be smaller or reversed; measuring cross-location covariance is a direct way to test this.
  • Our inference: the detection–estimation boundary suggests a practical upgrade—conditioning goodness on a single reference scalar per locality might convert the detector into a magnitude estimator, a testable extension the paper leaves to future work.
  • Our inference: the whitened goodness's gauge-fixing property may be a general mechanism for removing scale symmetries in contrastive local objectives, potentially applicable beyond FF.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a likelihood-ratio foundation for two core heuristics of Forward-Forward (FF) learning. Under an explicit generative model in which positive and negative pre-activations are zero-mean isotropic Gaussians differing only in variance, the squared norm is the sufficient statistic for the decision and the FF threshold is the Bayes decision boundary (Prop. 1). The paper extends this to anisotropic Gaussians (Mahalanobis quadratic, §4.1) and to Gaussian scale mixtures (§4.2), where the LLR is a saturating function of the Mahalanobis form with slope given by a posterior precision, yielding divisive normalization and an aggregation advantage (Props. 2–3). It then gives a theory of inter-layer normalization (Prop. 4) and identifies a scale-inflation shortcut in the pairwise FF objective, removed by a whitened negative-metric goodness (Observation 1). The empirical study measures activation statistics on a small convolutional FF network trained on clean vs distorted Imagenette images, reports that the measured regime is adverse for the plain square, that the Mahalanobis readout improves detection, that the GSM tail index matches kurtosis-based predictions, that perceptual alignment is bounded, and that whitened training stabilizes the goodness scale without a resolved downstream effect. The paper positions itself as explanatory rather than a benchmark contribution.

Significance. If the identification holds, the paper makes a valuable conceptual contribution: it converts the goodness and normalization choices of FF from heuristics into derived statistical objects, with falsifiable empirical predictions and honest null results. The theoretical core (Props. 1–4) is internally coherent, and the empirical protocol—measure-first with recorded predictions and explicit nulls—is exemplary. The synthetic calibration of the aggregation advantage is a useful addition. However, the claimed 'independent confirmation' of the GSM tail index is a circular reuse of the same fourth-moment statistic, so the empirical support for the GSM generalization is weaker than stated. This issue is local and fixable, but it currently overstates the evidence and should be corrected before publication.

major comments (2)
  1. [§6.3, §6.6, §8 and Appendix B] The 'independent confirmation' of the GSM tail index is not independent. §6.3 claims that the fitted tail indices ν̂∈[4.9,6.8] 'independently confirm' the values implied by the measured kurtosis in §6.1 (ν≈5–6.5 from κ=6/(ν−4)). Appendix B, however, defines the fitted index as ν̂=4+6/κ̄, where κ̄ is the mean excess kurtosis of whitened coordinates. This is exactly the inverse of the §6.1 relation applied to the same fourth-moment statistic; whitening is a linear transform and does not produce an independent measurement. The statements in §6.6 and §8 that the GSM description is 'quantitatively right—twice' or 'doubly validated' therefore overstate the evidence. The GSM account may still be plausible and the conditional derivation in Prop. 2 is unaffected, but the 'independent confirmation' should be withdrawn or replaced by a genuinely independent estimator (e.g., an ML fit of the Student
  2. [§4.2, §6 protocol, Appendix B] The aggregation advantage of the saturating statistic (Prop. 3(ii)) is stated and calibrated for K i.i.d. locations, but the empirical protocol pools 120 random spatial locations per image from convolutional feature maps, where overlapping receptive fields induce strong dependence. The paper does not quantify this dependence or provide an effective-sample-size / blocked-resampling analysis. Since the Gaussian/GSM covariance fits and the per-image readout comparisons in Tables 2–3 use these pooled locations, the numerical values (especially the AUC/SROCC gaps) may be miscalibrated. The central null results are probably robust because both readouts use the same locations, but the paper should either present a corrected analysis or explicitly state that the empirical readouts are descriptive and not to be interpreted under an independence model.
minor comments (4)
  1. [§4.2] The phrase 'writer=s +/s−' appears to be a typo; it should read 'where r=s+/s−'.
  2. [§5, Proposition 4] In the statement of Proposition 4, 'goodness is the mean squared activation' should be clarified as the per-coordinate mean, not the FF sum; the Θ(1/d) factor is otherwise confusing because the FF goodness is the sum of squares.
  3. [Appendix B] The tail-index estimator is described only by the formula ν̂=4+6/κ̄. Please specify the exact computation of κ̄ (mean over coordinates of excess kurtosis of whitened coordinates) and explicitly note that whitening is a linear transform that does not change the fourth-moment statistic, so that the relationship to §6.1 is transparent.
  4. [Data availability] The paper says code is available 'on reasonable request'. For a paper whose empirical program emphasizes measure-first replication, a public repository would strengthen reproducibility.

Circularity Check

1 steps flagged

GSM 'independent confirmation' is the same kurtosis-derived quantity reused, so the empirical double-validation collapses; the core LLR derivation is not circular.

specific steps
  1. fitted input called prediction [Section 6.3 and Appendix B ('Tail index')]
    "First, the fitted tail indices, ν̂∈[4.9,6.8] across layers and arms, independently confirm the values implied by the measured kurtosis: the GSM description of the activations is validated by two separate measurements. ... Tail index. ν̂=4+6/κ̄ from the mean excess kurtosis of whitened coordinates, clipped to [4.2,50]."

    Appendix B defines the fitted tail index as ν̂=4+6/κ̄, where κ̄ is the mean excess kurtosis of whitened coordinates. Section 6.1 obtains the 'implied' index from the inverse relation κ=6/(ν−4), i.e. ν=4+6/κ. These are algebraically the same function of the same measured kurtosis, so the 'independent confirmation' compares a quantity to itself in a different parametrization. Whitening is a linear transform and adds no independent information about tail weight. Thus the claim that the GSM description is 'validated by two separate measurements' is false by construction: only one fourth-moment measurement exists. This step does not make Proposition 1 circular, but it removes a stated piece of empirical support for the GSM generalization and for the 'doubly validated' conclusion.

full rationale

The central theoretical derivation is self-contained and non-circular. Proposition 1 starts from an explicit generative model (Eq. 2: two zero-mean isotropic Gaussians differing in scale), computes the log-likelihood ratio directly (Eq. 3), and derives that the LLR is increasing in ∥z∥²; the sufficient-statistic claim follows algebraically from the stated assumptions. Proposition 2 and Proposition 3 similarly derive the GSM statistic, its slope as posterior precision, and the aggregation result from explicit model assumptions, not from the conclusions being asserted. Proposition 4 and Observation 1 are conditional statements with stated hypotheses and are not simply renamed inputs. The Mahalanobis readout is fitted on a fit half and evaluated on a held-out half, which is a genuine predictive check. The one real circular step is the tail-index 'independent confirmation' in Section 6.3: the fitted ν̂ is defined in Appendix B as 4+6/κ̄, the same inverse of the kurtosis relation used in Section 6.1 to produce the 'predicted' ν≈5–6.5. The agreement is therefore a single measurement expressed in two equivalent forms, not two independent measurements. This partial circularity affects the empirical validation of the GSM generalization, but it does not invalidate the likelihood-ratio derivation itself. Use of the authors' prior work [3] for negative calibration and side probes is not load-bearing for the core theorem. Score 6 reflects one constructed 'prediction' while the principal derivation retains independent content.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central derivation uses classical decision theory plus two measured-but-postulated generative models for pre-activations. The empirical program adds several hand-set regularizers and a tail-index estimate that is presented twice as if independent. No new physical entities are introduced.

free parameters (4)
  • Class covariance matrices Σ_+, Σ_- (hence M=Σ_-^{-1}-Σ_+^{-1}) = estimated per layer on pooled pre-activations, ridge 10^{-3}; no explicit values in text
    Power the Mahalanobis readout in §6.2 and Table 2; fitted from the same trained networks, not independently measured.
  • Tail index ν_± = ν = 4 + 6/κ, clipped to [4.2,50]; fitted values 4.9–6.8
    Used in t-LLR readout; computed from mean excess kurtosis, so the claimed agreement with the kurtosis-implied 5–6.5 is a self-match, not independent confirmation.
  • Ridge regularizers = 10^{-3} (readouts/metric) and 10^{-4} (generalized eigenvalues)
    Hand-set constants that shape M, whitened metric, and the gen-eig spread; robustness not explored.
  • EMA momentum for whitened covariance = 0.05
    Controls the metric in Observation 1; whitened training plateau depends on this timescale; no sensitivity analysis.
axioms (5)
  • standard math Neyman-Pearson lemma and monotone likelihood-ratio theory are valid and applicable
    Invoked in §3–4 to justify optimality of ∥z∥² and sum_k h(q_k); paper does not prove these classical results.
  • domain assumption Pre-activations of positives/negatives are zero-mean isotropic Gaussians differing in scale (Eq. 2)
    The premise of Prop. 1; Table 1 later shows it is only weakly satisfied in trained networks, which the paper treats as an explanation rather than a derivation.
  • domain assumption Pre-activations follow a Gaussian scale mixture with common Σ and MLR-ordered mixer priors (Eq. 5)
    Basis of Props. 2–3 and the t-LLR readout; validation relies on kurtosis-derived tail index, whose independence is asserted but not established.
  • domain assumption Pooled spatial locations are i.i.d. samples from these class distributions
    Used for covariance estimation (120 locations/image) and for the aggregation advantage in Prop. 3(ii); convolutional activations are spatially correlated, and this is untested.
  • domain assumption ReLU half-rectified goodness is a monotone surrogate of the pre-activation sufficient statistic (Remark 1)
    Bridges the LLR derivation on z to deployed G(ReLU(z)); per-sample sign patterns make the relation stochastic, and all measurements are on pre-activations.

pith-pipeline@v1.3.0-alltime-deepseek · 13097 in / 20010 out tokens · 204873 ms · 2026-08-02T06:28:54.460422+00:00 · methodology

0 comments
read the original abstract

The Forward-Forward (FF) algorithm trains each layer locally, so that a scalar goodness - the sum of squared activations - is high on real inputs and low on contrastive ones, with activations normalized between layers. Both choices are usually treated as heuristics. Under an explicit generative model they are not: the squared goodness is the sufficient statistic of a likelihood-ratio test between two zero-mean populations differing in scale, and the FF threshold is its boundary. It generalizes: anisotropic populations yield a Mahalanobis goodness, the plain square being its isotropic case; heavy-tailed populations yield a saturating statistic whose slope is a posterior precision - divisive normalization - with bounded evidence and an advantage only under aggregation. The same lens characterizes the inter-layer normalization: it must remove the length while preserving per-coordinate energy, explaining a depth collapse we observe under unit-norm normalization; and the pairwise objective admits a scale-inflation shortcut that a whitened goodness removes.

Figures

Figures reproduced from arXiv: 2607.12501 by Paolo Giannitrapani.

Figure 1
Figure 1. Figure 1: The Student-t GSM likelihood ratio h(q) (d=16, r=1.10): saturating growth with finite plateau ν log r (dot￾ted), against the unbounded affine Gaussian LLR (dashed). = 3 = 5 = 8 = 50 0.6 0.7 0.8 0.9 1.0 held-out AUC k qk (energy), K = 64 k h(qk) (LLR), K = 64 energy, K = 16 LLR, K = 16 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Aggregation over K i.i.d. locations (synthetic, N=3000 per class): the Neyman–Pearson sum P k h(qk) vs. the energy sum P k qk. The gap grows with tail heaviness and K, and vanishes in the Gaussian limit. +0.000 at ν=50. These synthetic magnitudes calibrate what to expect on real activations once ν and the scale ratio are measured (§6). What a partial reference adds Proposition 2 contains, in passing, an ac… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 8 linked inside Pith

  1. [1]

    Greedy layerwise learning can scale to ImageNet, in: International Conference on Machine Learning

    Belilovsky, E., Eickenberg, M., Oyallon, E., 2019. Greedy layerwise learning can scale to ImageNet, in: International Conference on Machine Learning

    2019

  2. [2]

    Normalization as a canonical neural computation

    Carandini, M., Heeger, D.J., 2012. Normalization as a canonical neural computation. Nature Reviews Neuroscience 13, 51–62

    2012

  3. [3]

    Giannitrapani, P., Di Claudio, E.D., Jacovitti, G.,

  4. [4]

    Normalization of cell responses in cat striate cortex

    Heeger, D.J., 1992. Normalization of cell responses in cat striate cortex. Visual Neuroscience 9, 181–197

    1992

  5. [5]

    The forward-forward algorithm: Some preliminary investigations

    Hinton, G., 2022. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345

    Pith/arXiv arXiv 2022

  6. [6]

    Imagenette

    Howard, J., 2019. Imagenette. https://github.com/fastai/imagenette

    2019

  7. [7]

    Covariance-aware goodness for scalable forward- forward learning

    Jiang, X., Al-Hashimi, B.M., Xu, K., 2026. Covariance-aware goodness for scalable forward- forward learning. arXiv preprint arXiv:2605.04346

    Pith/arXiv arXiv 2026

  8. [8]

    Adam: A method for stochastic optimization, in: International Conference on Learning Representations

    Kingma, D.P., Ba, J., 2015. Adam: A method for stochastic optimization, in: International Conference on Learning Representations

    2015

  9. [9]

    Most apparent distortion: Full-reference image quality assessment and the role of strategy

    Larson, E.C., Chandler, D.M., 2010. Most apparent distortion: Full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging 19, 011006

    2010

  10. [10]

    Testing Statisti- cal Hypotheses

    Lehmann, E.L., Romano, J.P., 2005. Testing Statisti- cal Hypotheses. 3rd ed., Springer

    2005

  11. [11]

    Random synaptic feedback weights sup- port error backpropagation for deep learning

    Lillicrap, T.P., Cownden, D., Tweed, D.B., Akerman, C.J., 2016. Random synaptic feedback weights sup- port error backpropagation for deep learning. Nature Communications 7, 13276

    2016

  12. [12]

    Backpropagation and the brain

    Lillicrap, T.P., Santoro, A., Marris, L., Akerman, C.J., Hinton, G., 2020. Backpropagation and the brain. Nature Reviews Neuroscience 21, 335–346

    2020

  13. [13]

    Layer collaboration in the forward-forward algorithm

    Lorberbom, G., Gat, I., Adi, Y., Schwing, A., Hazan, T., 2023. Layer collaboration in the forward-forward algorithm. arXiv preprint arXiv:2305.12393

    Pith/arXiv arXiv 2023

  14. [14]

    Putting an end to end-to-end: Gradient-isolated learning of representations, in: Advances in Neural Information Processing Systems 32, pp

    L¨owe, S., O’Connor, P., Veeling, B.S., 2019. Putting an end to end-to-end: Gradient-isolated learning of representations, in: Advances in Neural Information Processing Systems 32, pp. 3039–3051

    2019

  15. [15]

    Direct feedback alignment pro- vides learning in deep neural networks, in: Advances in Neural Information Processing Systems 29

    Nøkland, A., 2016. Direct feedback alignment pro- vides learning in deep neural networks, in: Advances in Neural Information Processing Systems 29

    2016

  16. [16]

    Training neural networks with local error signals, in: International Conference on Machine Learning

    Nøkland, A., Eidnes, L.H., 2019. Training neural networks with local error signals, in: International Conference on Machine Learning

    2019

  17. [17]

    Repre- sentation learning with contrastive predictive coding

    van den Oord, A., Li, Y., Vinyals, O., 2018. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

    Pith/arXiv arXiv 2018

  18. [18]

    The predictive forward- forward algorithm

    Ororbia, A., Mali, A., 2023. The predictive forward- forward algorithm. arXiv preprint arXiv:2301.01452 . 9

    Pith/arXiv arXiv 2023

  19. [19]

    Image denoising using scale mixtures of Gaussians in the wavelet domain

    Portilla, J., Strela, V., Wainwright, M.J., Simoncelli, E.P., 2003. Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Transactions on Image Processing 12, 1338–1351

    2003

  20. [20]

    Learning representations by back-propagating errors

    Rumelhart, D.E., Hinton, G.E., Williams, R.J., 1986. Learning representations by back-propagating errors. Nature 323, 533–536

    1986

  21. [21]

    In search of goodness: Large scale benchmarking of goodness functions for the forward-forward algorithm

    Shah, A., Tripathi, V., 2025. In search of goodness: Large scale benchmarking of goodness functions for the forward-forward algorithm. arXiv preprint arXiv:2511.18567

    arXiv 2025

  22. [22]

    Image information and visual quality

    Sheikh, H.R., Bovik, A.C., 2006. Image information and visual quality. IEEE Transactions on Image Processing 15, 430–444

    2006

  23. [23]

    Scale mixtures of Gaussians and the statistics of natural im- ages, in: Advances in Neural Information Processing Systems 12

    Wainwright, M.J., Simoncelli, E.P., 2000. Scale mixtures of Gaussians and the statistics of natural im- ages, in: Advances in Neural Information Processing Systems 12

    2000

  24. [24]

    The cascaded forward algorithm for neural network training

    Zhao, G., Wang, T., Li, Y., Jin, Y., Lang, C., Ling, H., 2023. The cascaded forward algorithm for neural network training. arXiv preprint arXiv:2303.09728

    Pith/arXiv arXiv 2023

  25. [25]

    Activation learning by local compe- titions

    Zhou, H., 2022. Activation learning by local compe- titions. arXiv preprint arXiv:2209.13400 . 10

    Pith/arXiv arXiv 2022

  26. [2026]

    arXiv preprint arXiv:2607.08563

    Partial-reference IQA based on Hermite-Gauss structural prediction and texture deviation. arXiv preprint arXiv:2607.08563

    Pith/arXiv arXiv