On Advancements of the Forward-Forward Algorithm

Arne P. Raulf; Markus Lange; Mauricio Ortiz Torres

REVIEW 3 major objections 5 minor 13 references

On Advancements of the Forward-Forward Algorithm

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An improved no-backprop training rule reaches 18.8% CIFAR-10 test error, a reported 20% improvement over the original algorithm.

desk verdict A useful engineering report on lightweight Forward-Forward models, but the headline 20% improvement is not established because the comparison baseline is confounded by architecture, scale, and training budget. read the letter →

arxiv 2504.21662 v2 pith:HECQ2P45 submitted 2025-04-30 cs.LG

Mauricio Ortiz Torres , Markus Lange , Arne P. Raulf This is my paper · ORCID

classification cs.LG MSC 68T07

keywords Forward-Forwardalgorithmbackpropagation-freelearningconvolutionalchannelgroupingchannel-wiselosschunkedlocalupdatesCIFAR-10lightweightneuralnetworkslow-powerhardware

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the Forward-Forward algorithm, a training rule that updates layers from local goodness signals instead of backpropagated error, can be advanced to handle CIFAR-10 without losing its low-memory, flexible training. By combining convolutional channel grouping, a channel-wise loss, chunked local updates, and inference from the last two layers, the authors report roughly a 20 percent decrease in test error percentage over the original algorithm, with the best model at 18.8 percent one-pass test error. The same setup also produces a family of lightweight models with 164,706 to 754,386 parameters and test errors between 19.3 and 24.4 percent, which would make the approach relevant for low-capacity hardware. A sympathetic reader would take the central message to be that backpropagation-free local learning can be made competitive on a harder image benchmark while retaining its structural advantages.

What carries the argument

The load-bearing object is the convolutional group-channel construction. A convolutional layer's output $Y^l$ is subdivided into $J$ groups $\hat{Y}^l_j$, one per class, and each group's mean squared activation $G^l_{n,j}$ serves as that class's goodness. The channel-wise loss $C^l = -\frac{1}{N}\sum_n \log\left(\frac{\exp((g^l_{pos})_n)}{\sum_j \exp(G^l_{n,j})}\right)$ then turns goodness into a class posterior and drives competitive learning across the channel dimension. Chunked local updates carry this signal through blocks of layers, and last-two-layer inference uses the resulting hierarchical features. This mechanism is what lets each layer act as an independent classifier while preserving a forward flow of learned features.

What would settle it

Retrain the original Forward-Forward CNN on CIFAR-10 under exactly the same conditions as the improved model, including the same epoch count, learning-rate milestones, batch normalization, data preprocessing, and random seeds, then compare one-pass test error. If the original reaches 18.8 percent rather than 44.3 percent, the claimed improvement evaporates; if the improved model cannot reproduce 18.8 percent under those conditions, the reported result fails its own internal comparison.

Watch

Extended reading notes

Core claim

The central claim is that four modifications to the Forward-Forward algorithm work together to make it competitive on CIFAR-10. First, convolutional output channels are split into groups, one group per class, so each layer can produce a class-specific goodness score rather than a single scalar. Second, a channel-wise loss (equation 3) computes a softmax over those class goodnesses and drives the correct group to dominate, making each layer an independent classifier. Third, training proceeds by chunked local updates: blocks of consecutive layers are updated from local losses, and the blocks communicate learned features forward instead of being trained in isolation. Fourth, inference uses the concatenated activity vectors of only the last two layers. The paper reports a one-pass CIFAR-10 test error of 18.8 percent for the resulting FF deep model and a multi-pass error of 18.2 percent, compared with original CNN baselines of 44.3 and 48.3 percent, and describes the overall improvement as an approximate 20 percent decrease in test error percentage. Scaling down, the same techniques yield models with 164,706 to 754,386 parameters and test errors of 19.3 to 24.4 percent.

Load-bearing premise

The load-bearing premise is that the Table I original-algorithm runs are a fairly tuned, apples-to-apples control; if those baselines lacked the training budget, normalization, learning-rate schedule, or data preprocessing that the improved runs received, the reported 20 percent decrease would not be attributable to the algorithm changes.

Editorial extensions

If this is right

If the reported results hold, Forward-Forward training can reach single-pass CIFAR-10 accuracy around 18.8 percent without ever computing a backpropagation error signal.
The channel-group construction means each class is associated with a dedicated set of filters per layer, which suggests a route to class-specific feature visualisation and to adding new classes by adding new channel groups.
Chunked local updates allow 50 to 70 epochs to converge and keep training memory bounded by the largest block, which is the property that makes low-capacity hardware implementation plausible.
The lightweight model family gives an explicit accuracy-parameter trade-off, from 24.4 percent error at 164,706 parameters to 18.8 percent at 4.1 million parameters, and a 2-bit quantized variant reports 22.3 percent error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The channel-group design points to a class-incremental extension: new classes could be added as new channel groups while freezing old groups, a test the paper does not run.
Because no activations are stored for a backward pass, chunked local updates imply training memory scales with the largest block rather than depth, so the same approach could train much deeper networks on fixed hardware; the paper does not state this corollary.
A direct lesion experiment, removing or ablating the channel group associated with one class and measuring that class's accuracy, would test whether the reported gains come from the group-channel mechanism specifically or from the other combined techniques.
The parameter-error frontier reported for the light models suggests a practical design rule: around 160 filters in the final layer and four learning-rate milestones keep test error under 20 percent, a recipe that could transfer to other datasets.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is an incremental FF engineering report, not a new algorithm. The four techniques it bundles — convolutional channel grouping, channel-wise loss, chunked local updates, and last-two-layer inference — are all cited from prior work. What is genuinely new is the channel-width sweep that yields light CIFAR-10 models (164k–754k parameters, ~19–24% test error). Those tables are the useful part; the rest is a competent survey plus a flawed comparison.

What the paper does well: the survey of FF variants in Section II is organized and accurate enough to orient a newcomer. The stack of improvements is clearly described, and the light-model family fills a practical gap for low-power hardware. The parameter/error numbers are concrete and the citation pattern is honest — the authors point to the sources of each component, which I appreciate.

The soft spots are real and load-bearing. The headline “approximate 20% decrease in test error percentage” compares Table I’s CIFAR-10 CNN (two conv layers, 32 and 64 filters, plus a linear layer) against Table II’s “FF deep” (six conv layers from 128 to 512 filters, batch norm, maxpool, 300 epochs, multi-step LR schedule). That is not an apples-to-apples control. The gap conflates depth, parameter scale, normalization, and training budget with the FF-specific changes. A same-scale baseline using the original goodness loss, standard label overlay, per-layer local updates, and the same schedule and budget would be needed to support the attribution.

Also, the claim that convolutional channel grouping alone gives 27% test error appears in Section III with no table, no run details, and no error bar. It should either be removed or fully documented. There is no code, no seeds, no repeated runs, and the “tested minimum number of 160 filters” selection is test-set-based model selection, so the reported errors may be optimistic. The 300-epoch deep model and the 50–70 epoch light models are not compared on a common budget; that is fine for a hardware-focused study, but it weakens any algorithmic comparison.

Who this is for: FF practitioners, especially people who need low-parameter CIFAR-10 models or want a compact summary of current FF tricks. It deserves a serious referee because the engineering value is real and the light-model data could be useful, but the central claim needs a controlled baseline, code, and ablations before it can be taken as established. I would not cite the 20% improvement as an established result; I might use the light-model tables as a starting point for parameter-efficient FF experiments.

Referee Report

3 major / 5 minor

Summary. This paper reviews variants of the Forward-Forward algorithm and proposes an improved version that combines convolutional channel grouping, a channel-wise loss (Eq. 3), chunked local updates, and inference based on the last two layers. It reports MNIST and CIFAR10 results for the original and improved algorithms in Tables I and II, a family of lightweight CIFAR10 models in Tables III and IV, and a brief quantization experiment. The headline claim is an approximate 20% decrease in test error percentage relative to the original algorithm, with a one-pass CIFAR10 test error of 18.8% for the improved 'FF deep' model.

Significance. If the comparison were controlled, the result would be a useful demonstration that layer-wise and local-update training can scale to CIFAR10 with compact models, which is relevant for low-power hardware. The paper's useful contributions include integrating known FF techniques into one configuration, reporting parameter counts for lightweight FF models, and showing that quantization-aware training with 2-bit weights reaches 22.3% test error. However, the central numerical claim is currently not supported by a controlled experiment, and the lack of ablations, seeds, and code limits the value of the reported numbers until corrected.

major comments (3)

[Section III, Tables I and II] The central claim of an approximate 20% improvement is not supported by the evidence because the 'original' and 'improved' models differ in architecture, depth, normalization, and training budget simultaneously. The original CIFAR10 CNN is described as two convolutional layers with 32 and 64 filters plus a linear layer, while FF deep uses six convolutional layers with 128 to 512 filters, batch norm, ReLU, maxpool, and is trained for 300 epochs with a multi-step learning-rate schedule. A same-architecture control trained with the original goodness loss in Eq. (1), standard label overlay, per-layer local updates, and the same epoch and schedule budget is required before the improvement can be attributed to the proposed FF mechanisms; without it, the 25.5-percentage-point gap in test error between Table I and Table II conflates algorithm changes with scale and training budget.
[Section III, paragraph after Table II] The statement that convolutional channel grouping 'alone is responsible for allowing the model to obtain 27% in test errors' is not backed by any table, experimental setup, or run details. Since channel grouping is one of the four claimed improvements, a dedicated ablation, or at least an explicitly documented run with only channel grouping enabled, is needed to validate both this attribution and the combined claim.
[Section III, FF optimal paragraph] Model selection appears to be performed on the test set: the authors state that they used a 'tested minimum number of 160 filters at the final convolutional layer to generate test errors under 20% for all runs.' No train/validation/test split is described, no number of seeds is given, and no error bars are reported for Tables I, II, or IV. This makes it impossible to assess variance or overfitting; the authors should state the validation protocol, report mean and standard deviation over multiple seeds, and restrict any hyperparameter choice to a validation set rather than the test set.

minor comments (5)

[Section I, author affiliations] The affiliation text 'Institut for AI-Safety and Security' should read 'Institute for AI-Safety and Security.'
[Section II.A] The description of the one-hot label overlay on the first pixels would benefit from specifying exactly how many pixels are replaced for MNIST and how the overlay is adapted for CIFAR10, since the label vector length differs between the datasets.
[Section II.B.c] The text states that chunked local updates 'outperform the original individual layer-wise training' but does not provide a quantitative comparison or a direct reference for that specific claim; adding a citation or a small supporting experiment would improve precision.
[Section III, Table III] The notation underlining numbers in the kernel-size row to indicate maxpool locations is easy to miss; a separate column or an explicit marker such as 'MP' would improve readability.
[Section III, quantization paragraph] The quantized model is reported to reach 22.3% test error at 2-bit widths, but no architecture, training setup, or comparison with the full-precision model is given; adding these details would strengthen this exploratory result.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity; only a mild test-set-based selection of the FF-optimal architecture, making its below-20% test error selected rather than independently predicted.

fitted input called prediction [Section III, 'FF optimal model' design paragraph after Table IV]
"First, using a tested minimum number of 160 filters at the final convolutional layer to generate test errors under 20% for all runs."

The paper reports FF optimal with one-pass test error 19.6% and multi-pass 19.3% in Table IV, and then explains that 160 filters were chosen because they were tested to generate test errors under 20% for all runs. The below-20% performance is therefore a selection criterion used to pick the architecture, not an independent prediction that the method discovered. This is a mild test-set-based selection loop rather than a formal derivation: the main 20%-improvement claim over the original FF algorithm is an architecture comparison, not a quantity derived from its own inputs.

full rationale

The paper is largely an empirical survey and configuration of externally published techniques (Trifecta [4], channel-wise competitive learning [7], chunked local updates [8,9]), and none of these are the authors' own prior results, so no self-citation chain is load-bearing. The four listed improvements in Section III are not derived from the reported errors, and no equation is defined in terms of the conclusion it is supposed to support. The central '20% decrease in test error percentage' claim is a comparison between Table I's original CNN and Table II's FF deep, which differ in depth, width, normalization, and training budget; that is a confounding or attribution problem, not circularity. The only circularity-adjacent step is the FF-optimal design rule quoted above, which selects the final-layer filter count to satisfy a test-error threshold and then reports that the model meets it; that specific result is selected rather than predicted. This is minor and does not infect the rest of the paper, so the score is 2.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the correctness of cited techniques and on a fair baseline comparison rather than on new theory. The only invented 'entities' are the light model architecture names, which are configurations of known layers, not new physical or mathematical objects. The reported results depend on several hand-tuned hyperparameters.

free parameters (6)

MultiStep learning rate milestones = 3-5 milestones
Section III states '3-5 milestones in between the epochs' and says this schedule 'pushes the models to the optimal 18% test error'; the values are hand-selected per model.
Learning rate decay factor gamma = 0.1 or 0.2
Section III: 'gamma in {0.1, 0.2}' chosen by hand to stabilize training.
Number of training epochs = 50-70
Section III: 'models are trained using 50-70 epochs', selected per model to reach convergence.
FF optimal final-layer filter count = 160
Section III: 'tested minimum number of 160 filters at the final convolutional layer to generate test errors under 20% for all runs'; this is a test-error-based selection.
Light model channel widths = [3,50,50,50,50]; [3,50,50,70,70]; [3,50,50,100,150]; [3,50,50,100,160,160]
Architectures in Table III are varied to trade parameter count for test error, with FF optimal chosen as the best trade-off.
Unspecified training settings
Optimizer, batch size, initial learning rate, and preprocessing or augmentation are not reported; these choices can change CIFAR10 error and are required for replication.

assumptions (3)

domain assumption The original Forward-Forward algorithm as introduced by Hinton is a valid baseline for comparison.
Section II-A defines the goodness loss and overlay procedure; Section III compares against this as 'original' without questioning its suitability as a control.
domain assumption The cited component techniques, especially convolutional channel grouping and the channel-wise loss from [7] and chunked local updates from [4], work as described.
Section III composes these methods directly and attributes performance to them, but does not independently re-derive or ablate them.
domain assumption Training and test error percentages in the tables come from comparable standard splits and preprocessing across all models.
No split, preprocessing, or augmentation details are given, so the comparison assumes a controlled setup.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On Advancements of the Forward-Forward Algorithm." pith.science (2026). https://pith.science/paper/HECQ2P45

@misc{pith2026250421662,
  author       = {Pith},
  title        = {Pith review of: On Advancements of the Forward-Forward Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HECQ2P45}},
  note         = {Machine review of arXiv:2504.21662}
}

abstract

The Forward-Forward algorithm has evolved in machine learning research, tackling more complex tasks that mimic real-life applications. In the last years, it has been improved by several techniques to perform better than its original version, handling a challenging dataset like CIFAR10 without losing its flexibility and low memory usage. We have shown in our results that improvements are achieved through a combination of convolutional channel grouping, learning rate schedules, and independent block structures during training that lead to a 20\% decrease in test error percentage. Additionally, to approach further implementations on low-capacity hardware projects, we have presented a series of lighter models that achieve low test error percentages within (21$\pm$3)\% and number of trainable parameters between 164,706 and 754,386. This serves as a basis for our future study on complete verification and validation of these kinds of neural networks.

Figures

Figures reproduced from arXiv: 2504.21662 by the authors.

**Figure 2.** Flow diagram for the multi-pass inference step. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Flow diagram for the inference step in our improved algorithm. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

[1]

The Forward-Forward Algorithm: Some Preliminary Investigations

Geoffrey Hinton. The Forward-Forward Algorithm: Some Preliminary Investigations. arXiv e-prints, page arXiv:2212.13345, December 2022

arXiv 2022
[2]

Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning

Yujie Wu, Siyuan Xu, Jibin Wu, Lei Deng, Mingkun Xu, Qing- hao Wen, and Guoqi Li. Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning. arXiv e-prints, page arXiv:2408.14925, August 2024

arXiv 2024
[3]

Gradient-based learning applied to document recognition

Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

1998
[4]

The Trifecta: Three simple techniques for training deeper Forward-Forward networks

Thomas Dooms, Ing Jyh Tsang, and Jose Oramas. The Trifecta: Three simple techniques for training deeper Forward-Forward networks. arXiv e-prints, page arXiv:2311.18130, November 2023

arXiv 2023
[5]

Training Convolutional Neural Networks with the Forward- Forward algorithm

Riccardo Scodellaro, Ajinkya Kulkarni, Frauke Alves, and Matthias Schr¨oter. Training Convolutional Neural Networks with the Forward- Forward algorithm. arXiv e-prints, page arXiv:2312.14924, December 2023

arXiv 2023
[6]

Self-Contrastive Forward-Forward Algorithm

Xing Chen, Dongshu Liu, Jeremie Laydevant, and Julie Grollier. Self-Contrastive Forward-Forward Algorithm. arXiv e-prints, page arXiv:2409.11593, September 2024

arXiv 2024
[7]

Convolutional channel-wise competitive learning for the forward-forward algorithm

Andreas Papachristodoulou, Christos Kyrkou, Stelios Timotheou, and Theocharis Theocharides. Convolutional channel-wise competitive learning for the forward-forward algorithm. Proceedings of the AAAI Conference on Artificial Intelligence, 38(13):14536–14544, Mar. 2024

work page 2024
[8]

Parallel Training of Deep Networks with Local Updates

Michael Laskin, Luke Metz, Seth Nabarro, Mark Saroufim, Badred- dine Noune, Carlo Luschi, Jascha Sohl-Dickstein, and Pieter Abbeel. Parallel Training of Deep Networks with Local Updates. arXiv e-prints, page arXiv:2012.03837, December 2020

arXiv 2012

Show all 13 references

[9]

Loco: Local contrastive representation learning

Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. Advances in neural information processing systems, 33:11142–11153, 2020

2020
[10]

LightFF: Lightweight Inference for Forward-Forward Algorithm

Amin Aminifar, Baichuan Huang, Azra Abtahi, and Amir Amini- far. LightFF: Lightweight Inference for Forward-Forward Algorithm. arXiv e-prints, page arXiv:2404.05241, April 2024

2024 arXiv
[11]

Employing Layerwised Unsupervised Learning to Lessen Data and Loss Re- quirements in Forward-Forward Algorithms

Taewook Hwang, Hyein Seo, and Sangkeun Jung. Employing Layerwised Unsupervised Learning to Lessen Data and Loss Re- quirements in Forward-Forward Algorithms. arXiv e-prints, page arXiv:2404.14664, April 2024

2024 arXiv
[12]

Variance-capturing forward-forward autoencoder (vffae): A forward learning neural net- work for fault detection and isolation of process data

Deepak Kumar, Umang Goswami, Hariprasad Kodamana, Mano- jkumar Ramteke, and Prakash Kumar Tamboli. Variance-capturing forward-forward autoencoder (vffae): A forward learning neural net- work for fault detection and isolation of process data. Process Safety and Environmental Pr...

2023
[13]

Learning multiple layers of features from tiny images

Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012

2012

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

The Forward-Forward Algorithm: Some Preliminary Investigations

Geoffrey Hinton. The Forward-Forward Algorithm: Some Preliminary Investigations. arXiv e-prints, page arXiv:2212.13345, December 2022

arXiv 2022

[2] [2]

Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning

Yujie Wu, Siyuan Xu, Jibin Wu, Lei Deng, Mingkun Xu, Qing- hao Wen, and Guoqi Li. Distance-Forward Learning: Enhancing the Forward-Forward Algorithm Towards High-Performance On-Chip Learning. arXiv e-prints, page arXiv:2408.14925, August 2024

arXiv 2024

[3] [3]

Gradient-based learning applied to document recognition

Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

1998

[4] [4]

The Trifecta: Three simple techniques for training deeper Forward-Forward networks

Thomas Dooms, Ing Jyh Tsang, and Jose Oramas. The Trifecta: Three simple techniques for training deeper Forward-Forward networks. arXiv e-prints, page arXiv:2311.18130, November 2023

arXiv 2023

[5] [5]

Training Convolutional Neural Networks with the Forward- Forward algorithm

Riccardo Scodellaro, Ajinkya Kulkarni, Frauke Alves, and Matthias Schr¨oter. Training Convolutional Neural Networks with the Forward- Forward algorithm. arXiv e-prints, page arXiv:2312.14924, December 2023

arXiv 2023

[6] [6]

Self-Contrastive Forward-Forward Algorithm

Xing Chen, Dongshu Liu, Jeremie Laydevant, and Julie Grollier. Self-Contrastive Forward-Forward Algorithm. arXiv e-prints, page arXiv:2409.11593, September 2024

arXiv 2024

[7] [7]

Convolutional channel-wise competitive learning for the forward-forward algorithm

Andreas Papachristodoulou, Christos Kyrkou, Stelios Timotheou, and Theocharis Theocharides. Convolutional channel-wise competitive learning for the forward-forward algorithm. Proceedings of the AAAI Conference on Artificial Intelligence, 38(13):14536–14544, Mar. 2024

work page 2024

[8] [8]

Parallel Training of Deep Networks with Local Updates

Michael Laskin, Luke Metz, Seth Nabarro, Mark Saroufim, Badred- dine Noune, Carlo Luschi, Jascha Sohl-Dickstein, and Pieter Abbeel. Parallel Training of Deep Networks with Local Updates. arXiv e-prints, page arXiv:2012.03837, December 2020

arXiv 2012