Scalable Forward-Forward Algorithm

Andrii Krutsylo

REVIEW 6 major objections 4 minor 12 references

Scalable Forward-Forward Algorithm

T0 review · 6 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The Scalable Forward-Forward algorithm trains deep convolutional networks without global backpropagation, matching standard backpropagation in accuracy and, in its block-wise hybrid form, sometimes exceeding it while using less memory.

desk verdict SFF is a promising FF variant, but its headline accuracy, speed, and memory claims are undermined by an unfair evaluation protocol and an abstract that contradicts its own table. read the letter →

arxiv 2501.03176 v1 pith:JEN5QKW7 submitted 2025-01-06 cs.LG cs.NE

Andrii Krutsylo This is my paper · ORCID

classification cs.LGcs.NE

keywords forward-forwardalgorithmbackpropagation-freetraininglayerwiselearningconvolutionalneuralnetworkslocalgoodnessblock-wisetransfersmall-data

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes the Scalable Forward-Forward (SFF) algorithm, a way to train deep convolutional networks without global backpropagation: each layer or block is updated from a local loss computed in a single forward pass, so error gradients never flow across layer boundaries. The author's central claim is that SFF reaches accuracy comparable to standard backpropagation on CIFAR-10, CIFAR-100, and Imagenette with modern architectures, and that a block-wise hybrid—backpropagation inside residual blocks but not between them—tends to outperform the backpropagation baseline while using less memory. The reason this matters is that layer-local training is more modular, more memory-efficient, and compatible with non-differentiable components, which would make SFF a practical drop-in alternative to backpropagation rather than a research curiosity. The paper further reports that SFF benefits from ImageNet pretraining done with backpropagation and outperforms backpropagation when only 1,000 training samples are available.

What carries the argument

The load-bearing mechanism is a per-block goodness readout. After each trainable layer or block, a small auxiliary convolutional layer (kernel size $1\times 1$ or slightly larger) maps the activations to a tensor of shape [batch size, number of classes, height, width]; pooling over space gives a per-class goodness vector. The local loss is $L_{\mathrm{SFF}} = -\mathbb{E}\big[g_{\mathrm{pos}} - \log \sum_{j=1}^{J} \exp(g_j)\big]$, a log-sum-exp margin that compares the ground-truth goodness to all class goodnesses in a single pass and removes the need for explicit negative samples. For deeper networks, backpropagation is allowed inside each block—such as a residual block—but never between blocks, while layer normalization is applied after each trainable layer to stabilize the local objectives. This combination carries the argument: the auxiliary convolutions replace CwC's channel partitioning, the smoothed margin makes the one-pass loss trainable, and the block-wise restriction preserves FF's memory and modularity benefits.

What would settle it

Train the backpropagation baseline on CIFAR-100 CNNB and Imagenette ResNet18 using SFF's architecture (including its added layer-normalization layers), evaluate it with the same layer-ensemble goodness averaging used for SFF, and compare against SFF; if backpropagation then matches or exceeds SFF, the claimed superiority is an artifact of the evaluation protocol.

Watch

Extended reading notes

Core claim

The central claim is that the Forward-Forward recipe—separately maximizing a layer's 'goodness' on correct inputs and minimizing it on incorrect ones—can be scaled to large convolutional models without the channel-partitioning apparatus of earlier variants. SFF replaces hard class-to-channel assignments with small auxiliary convolutional layers that read each layer's activations and produce a per-class goodness tensor in one forward pass; the local objective is a log-sum-exp margin that pushes the ground-truth class's goodness above the smoothed aggregate of all classes. On Imagenette with ResNet18, the paper reports 83.24% layer-ensemble accuracy for SFF against 81.84% classifier-head accuracy for backpropagation, and on CIFAR-100's CNNB model 55.34% against 50.49%; on MobileNetV3/Imagenette, SFF trails at 77.06% versus 80.44%, so the overall claim is a mix of small wins, small losses, and one large win. Because SFF has no classifier head of its own, its reported accuracy averages the goodness vectors from all layers, while backpropagation is evaluated through its final head; the paper presents this as the natural head-free evaluation for SFF.

Load-bearing premise

The load-bearing premise is that layer-ensemble goodness accuracy for SFF and classifier-head accuracy for backpropagation are fair equivalents; if backpropagation were evaluated with the same layer-ensemble readout (or with the same added layer normalization), the reported SFF advantages could disappear.

Editorial extensions

If this is right

SFF gives a concrete recipe for training modern convolutional networks without global backpropagation, with measured peak-memory reductions of roughly 280 MB on ResNet18/Imagenette and 530 MB on MobileNetV3/Imagenette.
Because gradients stay inside blocks, SFF can accommodate non-differentiable or black-box components and allows blocks to be trained independently, supporting modular or parallel training.
Weights pretrained with backpropagation can be fine-tuned with SFF, so existing ImageNet-pretrained models remain usable under forward-only training.
The small-data experiments (1,000 training samples) suggest SFF generalizes better than backpropagation in data-scarce settings, with a 9-percentage-point gain on Imagenette/MobileNetV3.
The hybrid block-wise variant is the configuration that matches or beats backpropagation, while pure layerwise FF remains competitive only on the simpler CNN; the paper attributes the efficiency advantage to having many small blocks, as in MobileNetV3.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The headline comparisons are not apples-to-apples: SFF is scored by layer-ensemble goodness while backpropagation is scored by its classifier head. A natural test the paper does not run is to give the backpropagation baseline the same layer-ensemble readout; if BP also improves, the reported advantages on CNNB and ResNet18 would need to be reinterpreted.
The log-sum-exp margin makes each layer behave like a small softmax classifier over goodness values, so SFF can be viewed as a form of local learning with per-layer losses; this suggests connections to other local-learning methods and raises the question of whether the auxiliary layers' overhead could be reduced by sharing filters or pruning class-specific channels.
The transfer-learning results open the reverse experiment: pretrain with SFF and fine-tune with backpropagation. If the feature hierarchies are truly compatible, this should also work, and it would test whether SFF's benefit is a property of the objective or just of the starting point.
If the low-data advantage persists on medical or remote-sensing benchmarks, layerwise goodness training could become a practical default for small datasets, since it appears to regularize without explicit data augmentation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: the paper is worth reading, and it deserves a careful referee, but the core empirical claim—comparable to backprop—is not yet supported. The evaluation protocol and the abstract's speed/memory claims both need fixing.

What's actually new: SFF replaces CwC's hard channel-to-class assignment with small 1x1 convolutions that produce a per-class goodness tensor, and it uses block-wise local backprop to scale to ResNet18 and MobileNetV3. That's a clean, practical idea, and the public code should make reproduction easy. The transfer-learning result (ImageNet-pretrained weights fine-tuned with FF) is a good addition.

What I like: the method runs on modern architectures without global backprop, and the raw numbers on CNNB (81.38 on CIFAR-10, 55.34 on CIFAR-100) are in the right ballpark. The authors also state in Section 5 that SFF is slower than BP in every setting, which makes the abstract's 'similar training speed' puzzling.

The soft spots, in order of importance:

1. The 'new loss' is not new. The CwC loss in Section 2.3.2 is -log(softmax(g+)) = -g+ + logsumexp(G). The equation in Section 4.2 is -(g+ - logsumexp(G)), which is the same function. The real novelty is the auxiliary convolution, not the loss. That claim should be corrected.

2. The accuracy comparison is confounded, as you noted. SFF is evaluated by averaging goodness vectors from all layers; BP is scored by its final head. That is a committee of classifiers versus a single head. In addition, BP is run without the layer-normalization layers that SFF inserts. Both changes can easily account for a few points. The fix is straightforward: add a BP baseline with the same layer ensemble and the same normalization, and compare like with like.

3. The memory claim is not universal. On CIFAR-100 CNN, SFF peaks at 884 MB versus BP's 562 MB. The 'significantly lower memory' claim only holds on the ImageNet models, and even then it is modest on ResNet18. The paper's own data contradict the abstract.

4. The small-data experiment is a single dataset with three seeds. The 9-point gap on Imagenette is suggestive, but I would want more runs and a properly tuned BP baseline before taking it seriously.

Bottom line: the method is plausible and the direction is right, but the paper overstates its case. Send it to referees, and ask for a fair comparison protocol and a corrected abstract. If the accuracy advantage survives a layer-ensemble BP baseline, this will be a solid contribution.

Referee Report

6 major / 4 minor

Summary. The paper proposes Scalable Forward-Forward (SFF), a layerwise training method for convolutional networks. SFF replaces the class-channel partitioning of the Channel-wise Competitive Learning (CwC) method with small auxiliary convolutional layers that produce per-class goodness tensors, and trains each layer or block with a margin objective based on the log-sum-exp of class goodness values. For deeper models, gradients are allowed to flow within blocks but not between blocks. The authors evaluate SFF against backpropagation (BP) and a modified CwC baseline on CIFAR-10, CIFAR-100, and Imagenette using CNN, CNNB, ResNet18, and MobileNetV3 architectures, reporting accuracy, peak VRAM, runtime, and parameter counts, plus small-data (1000-sample) and ImageNet-pretraining transfer experiments.

Significance. If the empirical comparison were controlled, SFF would be a useful contribution: the auxiliary-convolution idea is simple and removes the hard channel-group constraints of CwC, the within-block hybrid is a practical compromise for modern architectures, and the paper ships a public implementation and reports memory, time, and parameter counts. The layerwise objective and one-pass goodness computation are interesting. However, the current evidence for the central claims is weakened by comparison-protocol and architectural confounds, and several abstract-level claims are contradicted by the paper's own tables. The contribution is promising but needs revision.

major comments (6)

[§4.4 and Table 2] The headline accuracy comparison is not controlled: SFF is evaluated by averaging goodness vectors from all layers (Section 4.4), whereas BP is evaluated only by its final classifier head. A layer ensemble is a multi-classifier aggregation and can outperform a single head even when the local training rule is weaker; the gaps in Table 2 (e.g., +4.85 on CIFAR-100 CNNB, +1.40 on Imagenette ResNet18) could therefore reflect the evaluation protocol rather than SFF. Please add a BP baseline evaluated with an equivalent layer ensemble (e.g., averaged softmax/logits of all layers) and an SFF evaluation using only its final goodness layer, and report both.
[Section 3] The SFF models include layer normalization after every trainable layer or block, while the BP baseline omits the additional normalization layers. This changes both the architecture and the optimization landscape, so any SFF accuracy improvement cannot be attributed solely to the update rule. Please train BP with the same normalization layers (and the same hyperparameter search) to isolate the effect of the training algorithm.
[Abstract and Section 5] The speed claim is internally inconsistent: the abstract says the hybrid design maintains 'a similar training speed,' but Table 2 reports SFF time ratios from 1.2x to 7.4x slower than BP in every setting, and Section 5 explicitly states 'In all experimental settings, SFF is slower than BP.' Please correct the abstract and conclusions to state the actual slowdown, or provide a controlled speed comparison under matched conditions.
[Introduction and Table 2] The claim of 'significantly lower memory allocation' is contradicted by Table 2 on CIFAR-100: SFF uses 883.78 MB versus 561.85 MB for BP on the CNN and 702.08 MB versus 561.12 MB on CNNB, i.e., 1.25--1.57x more memory. The memory advantage only appears on some architectures (e.g., Imagenette). Please qualify the memory claim and discuss the dependence on block size and auxiliary-layer overhead.
[Abstract and Section 4.3] The abstract states that the method 'eliminates the need for backpropagation,' but the competitive results on ResNet18 and MobileNetV3 are obtained with a hybrid design that allows backpropagation within each block (Section 4.3, Table 1). This is a legitimate design, but it should be stated prominently in the abstract and introduction; otherwise the central 'no backpropagation' claim is overstated.
[Section 5 and Table 2] The CwC-mod baseline is not a faithful representation of CwC, and its complete failure on CIFAR-100 (1.00 ± 0.00 for both layer and head accuracy) does not establish that CwC cannot handle large numbers of classes. The paper notes that the original CwC required superclasses for CIFAR-100, but the modified version removes CFSE blocks and ILT. Please validate the modified baseline against the original CwC configuration or report the original numbers, and temper the comparative claim accordingly.

minor comments (4)

[Section 4] The sentence 'The only tunable parameter in this approach is the kernel size of the auxiliary convolutional layer' contradicts Section 3, which describes hyperparameter searches over two learning rates and weight decay. Please clarify that this refers to architectural hyperparameters, not optimization hyperparameters.
[Table 2] The parameter counts are not directly comparable because SFF includes auxiliary layers but does not include a standard classifier head, whereas BP includes the head. Please state the head parameter count separately or use a matched architecture for the parameter comparison.
[Section 4.3] Step 3 says to compute 'positive and negative goodness factors,' but the note says the negative factor is not used in the loss for the presented results. This is confusing; please clarify whether the negative factor is used in any variant or remove it from the description.
[Throughout] There are typographical artifacts in the text, such as 'CIF AR-10' and 'CIF AR-100' in several places and a Unicode ligature in the GitHub URL. These should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SFF method and results are empirical, not derived from fitted inputs or self-cited forced choices.

full rationale

The paper's central claims are empirical: SFF is a new layerwise training objective (Section 4.2) evaluated on CIFAR-10, CIFAR-100, and Imagenette. The loss LSFF uses the ground-truth class goodness and a log-sum-exp comparison over all class goodness values; it is not fitted to the reported accuracies, and no parameter is chosen so that a later 'prediction' matches training data by construction. The method builds on prior work (CwC, reference [3]) but does not import a uniqueness theorem or an ansatz whose validity is load-bearing; the SFF goodness computation and auxiliary convolutional layers are stated directly in Sections 4.1-4.3. The comparison in Table 2 raises a legitimate fairness concern: SFF is scored with layer-ensemble accuracy (Section 4.4) while BP is scored with its classifier head, and the BP baseline omits the layer normalization added to SFF. However, this is an evaluation-protocol confound, not a circular reduction: the SFF accuracies are measured outcomes of a specified training procedure, and the paper does not define SFF's success in terms of the BP baseline or derive the claimed advantage from the benchmark itself. There are also no self-citations used to justify the central premise, and no known result is merely renamed. Thus the derivation chain is self-contained, and any weakness lies in experimental comparability rather than circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central empirical claims rest on a small number of hand-chosen hyperparameters (kernel size, learning rates, weight decay) and on assumptions inherited from the FF literature about goodness and layer normalization. The hybrid block-wise design introduces an unablated assumption that internal backpropagation preserves the benefits of local training. No formal guarantees are provided.

free parameters (3)

kernel size of auxiliary convolutional layer = not reported (stated as the only tunable parameter)
Section 4 says the only tunable parameter is the kernel size of the auxiliary convolutional layer, but no per-experiment values are given, so its effect and exact setting are unknown.
learning rates for feature layers and classifier = selected by search from a set, final values not reported
Section 3 describes a hyperparameter search over 16 learning rates and multiple weight decay values, but the chosen values for each dataset and method are not listed, making the reported results hard to reproduce without guessing.
weight decay = selected from {0, 1e-8, 1e-7, 1e-6, 1e-5}, value not reported
The weight decay is tuned per experiment; undisclosed choices could affect the comparisons between SFF and BP.

assumptions (3)

domain assumption Goodness as the mean of squared activations is a useful training signal.
Inherited from the original FF algorithm (Section 2.1) and used throughout SFF without independent justification for convolutional feature maps.
domain assumption Layer normalization stabilizes Forward-Forward training.
Section 3 states layer normalization is applied after each trainable layer or block 'following the original FF guidelines.' This is a practical heuristic, not proven for the new SFF setting.
ad hoc to paper Backpropagation within blocks does not erase the benefits of Forward-Forward.
Section 4.3 permits backpropagation inside blocks but restricts it across blocks. The paper provides no ablation isolating whether the accuracy gains come from the FF local objective or from the internal backpropagation within blocks.

invented entities (1)

Auxiliary convolutional layer
purpose: Computes a class-specific goodness tensor for each block, enabling a local loss without channel partitioning.
This is the main new component of SFF. It is not independently validated outside the reported experiments, and no theoretical or empirical analysis isolates its contribution.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scalable Forward-Forward Algorithm." pith.science (2026). https://pith.science/paper/JEN5QKW7

@misc{pith2026250103176,
  author       = {Pith},
  title        = {Pith review of: Scalable Forward-Forward Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JEN5QKW7}},
  note         = {Machine review of arXiv:2501.03176}
}

read the original abstract

We propose a scalable Forward-Forward (FF) algorithm that eliminates the need for backpropagation by training each layer separately. Unlike backpropagation, FF avoids backward gradients and can be more modular and memory efficient, making it appealing for large networks. We extend FF to modern convolutional architectures, such as MobileNetV3 and ResNet18, by introducing a new way to compute losses for convolutional layers. Experiments show that our method achieves performance comparable to standard backpropagation. Furthermore, when we divide the network into blocks, such as the residual blocks in ResNet, and apply backpropagation only within each block, but not across blocks, our hybrid design tends to outperform backpropagation baselines while maintaining a similar training speed. Finally, we present experiments on small datasets and transfer learning that confirm the adaptability of our method.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

De ep residual learning for image recognition, 2015

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. De ep residual learning for image recognition, 2015

work page 2015
[2]

Searching for mobilenetv3

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Che n, Bo Chen, Mingxing Tan, W eijun W ang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V Le, and Hartwig Adam. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV) , pages 1314–1324, 2019

work page 2019
[3]

Convolutional channel-wise competitive le arning for the forward-forward algorithm

Andreas Papachristodoulou, Christos Kyrkou, Stelios T imotheou, and Theocharis Theocharides. Convolutional channel-wise competitive le arning for the forward-forward algorithm. In Proceedings of the AAAI Conference on Artiﬁcial Intelligen ce, volume 38, pages 14536–14544, 2024

work page 2024
[4]

Geoﬀrey E. Hinton. The forward-forward algorithm: Some preliminary investigations. ArXiv, abs/2212.13345, 2022

arXiv 2022
[5]

Train- ing convolutional neural networks with the forward-forwar d algorithm

Riccardo Scodellaro, Ajinkya Kulkarni, Frauke Alves, a nd Matthias Schr¨ oter. Train- ing convolutional neural networks with the forward-forwar d algorithm. ArXiv, abs/2312.14924, 2023

arXiv 2023
[6]

Gradient-based learning applied to document recognition

Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haﬀ ner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324, 1998

work page 1998
[7]

Fashion-mn ist: a novel image dataset for benchmarking machine learning algorithms, 2017

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mn ist: a novel image dataset for benchmarking machine learning algorithms, 2017

work page 2017
[8]

Resource-eﬃcient medical image analysis with self-adapting forward-forward networks

Johanna P M¨ uller and Bernhard Kainz. Resource-eﬃcient medical image analysis with self-adapting forward-forward networks. In International Workshop on Machine Learning in Medical Imaging , pages 180–190. Springer, 2024

work page 2024

Show all 12 references

[9]

Self-contrastive forward- forward algorithm

Xing Chen, Dongshu Liu, Jeremie Laydevant, and Julie Gro llier. Self-contrastive forward- forward algorithm. arXiv preprint arXiv:2409.11593 , 2024

2024 arXiv
[10]

Improved forward-forward contrastive lea rning

R Gananath. Improved forward-forward contrastive lea rning. arXiv preprint arXiv:2405.03432, 2024

2024 arXiv
[11]

Learning multi ple layers of features from tiny images

Alex Krizhevsky, Geoﬀrey Hinton, et al. Learning multi ple layers of features from tiny images. 2009

2009
[12]

Imagenette: A smaller subset of 10 easil y classiﬁed classes from imagenet, March 2019

Jeremy Howard. Imagenette: A smaller subset of 10 easil y classiﬁed classes from imagenet, March 2019

2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

De ep residual learning for image recognition, 2015

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. De ep residual learning for image recognition, 2015

work page 2015

[2] [2]

Searching for mobilenetv3

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Che n, Bo Chen, Mingxing Tan, W eijun W ang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V Le, and Hartwig Adam. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV) , pages 1314–1324, 2019

work page 2019

[3] [3]

Convolutional channel-wise competitive le arning for the forward-forward algorithm

Andreas Papachristodoulou, Christos Kyrkou, Stelios T imotheou, and Theocharis Theocharides. Convolutional channel-wise competitive le arning for the forward-forward algorithm. In Proceedings of the AAAI Conference on Artiﬁcial Intelligen ce, volume 38, pages 14536–14544, 2024

work page 2024

[4] [4]

Geoﬀrey E. Hinton. The forward-forward algorithm: Some preliminary investigations. ArXiv, abs/2212.13345, 2022

arXiv 2022

[5] [5]

Train- ing convolutional neural networks with the forward-forwar d algorithm

Riccardo Scodellaro, Ajinkya Kulkarni, Frauke Alves, a nd Matthias Schr¨ oter. Train- ing convolutional neural networks with the forward-forwar d algorithm. ArXiv, abs/2312.14924, 2023

arXiv 2023

[6] [6]

Gradient-based learning applied to document recognition

Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haﬀ ner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324, 1998

work page 1998

[7] [7]

Fashion-mn ist: a novel image dataset for benchmarking machine learning algorithms, 2017

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mn ist: a novel image dataset for benchmarking machine learning algorithms, 2017

work page 2017

[8] [8]

Resource-eﬃcient medical image analysis with self-adapting forward-forward networks

Johanna P M¨ uller and Bernhard Kainz. Resource-eﬃcient medical image analysis with self-adapting forward-forward networks. In International Workshop on Machine Learning in Medical Imaging , pages 180–190. Springer, 2024

work page 2024