Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Reshaping the Forward-Forward Algorithm with a Similarity-Based Objective

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A similarity-based rewrite of the Forward-Forward algorithm reaches 56.22% on CIFAR-10 with a simple MLP, within 1.4 points of backpropagation, while needing only one forward pass at test time.

desk verdict A serious but incremental FF variant whose headline number hinges on an unspecified choice of one representative image per class; worth refereeing, but the authors must show robustness to that choice. read the letter →

arxiv 2509.08697 v1 pith:YFO472DF submitted 2025-08-29 cs.LG cs.AI

James Gong , Raymond Luo , Emma Wang , Leon Ge , Bruce Li , Felix Marattukalam , Waleed Abdulla This is my paper · ORCID
classification cs.LGcs.AI
keywords Forward-Forwardalgorithmsimilaritylearningtupletlossbiologicallyplausiblelocalsingle-passinferenceCIFAR-10FAUST
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the Forward-Forward (FF) learning rule, a backpropagation-free alternative whose goodness-score objective trails backprop in accuracy, can be reshaped as a similarity-learning problem. The proposed algorithm, FAUST, replaces the FF goodness score with (N+1)-tuplet losses over anchor-positive-negative embeddings, so each layer learns to pull same-class embeddings together and push different classes apart. Its representative variant caches one image per class as a fixed reference for training and as a nearest-neighbor prototype for inference, making test-time classification a single forward pass. On MNIST, Fashion-MNIST, and CIFAR-10, FAUST beats prior FF variants and on CIFAR-10 scores 56.22% versus backprop's 57.63%. If the claim holds, bio-plausible local learning can operate at near-backprop accuracy without the memory and locking costs of backpropagation.

What carries the argument

The load-bearing mechanism is the (N+1)-tuplet loss applied per layer with a cached set of class representatives R: one image per class, fixed across training. Each anchor forms a tuplet with its class representative as positive and all other representatives as negatives; embeddings of representatives are cached for the batch, cutting forward passes to B + C per batch. A trainable linear embedding layer maps each hidden layer's activations to a lower-dimensional space where distances are measured, while raw activations continue to the next layer. Inference sums Euclidean distances from the test embedding to each cached representative over all layers and picks the closest class.

What would settle it

Train FAUST-representative tuplet on CIFAR-10 several times, each time choosing a different image as the representative for each class, including deliberately atypical images. If test accuracy varies by more than a couple of points across choices, the representative selection, not the similarity objective, determines the reported result.

Watch

Extended reading notes

Core claim

The central discovery is that the bottleneck in FF is not the local, layer-wise learning scheme itself but the goodness-score objective and its inference procedure. By swapping in a similarity-based tuplet objective and training each layer greedily on embeddings, FAUST learns discriminative representations layer by layer; t-SNE visualizations show class clusters sharpening from a Fisher score of 2.97 at layer 1 to 9.81 at layer 4. The representative-tuplet variant constrains the tuplet optimization to fixed per-class anchors, which simplifies the problem and yields the best accuracy: 98.43% on MNIST, 89.67% on Fashion-MNIST, and 56.22% on CIFAR-10. The paper reads this as evidence that the e

Load-bearing premise

The method relies on one image per class standing in for the whole class during both training and inference; if that image is atypical, the learned embeddings and final accuracy suffer, and the paper does not measure how sensitive the result is to that choice.

Editorial extensions

If this is right

  • FAUST achieves single-pass inference for all classes; the optimal form of FF requires C forward passes per test input.
  • The similarity objective closes most of the FF-to-backprop accuracy gap: within 0.93% on MNIST/Fashion-MNIST and 1.4 points on CIFAR-10.
  • Because layers are trained locally with no backward pass, FAUST inherits FF's freedom from backward locking and from storing intermediate activations for backprop.
  • Fixed representatives simplify the tuplet optimization and give the best accuracy among the three FAUST variants, suggesting the constrained subproblem helps rather than hurts.
  • Deep layers keep improving: the Fisher discriminant score rises from 2.97 to 9.81 across four layers on Fashion-MNIST, showing layer-wise training builds increasingly separable representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If representative choice is the real bottleneck, selecting each class's representative by clustering (e.g., the image closest to the class centroid) could push accuracy further; the paper does not test this.
  • The cached-representative trick converts tuplet loss into a fixed-prototype metric-learning problem, so the method should transfer to few-shot and open-set settings where prototypes are naturally used.
  • A direct ablation varying the representative image across random seeds or using deliberately atypical images would show how much of the reported 56.22% rests on representative choice rather than the similarity objective.
  • Because inference is single-pass and layer-local, FAUST could be combined with online or continual learning updates that add new class representatives without retraining earlier layers; the paper leaves this untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FAUST, a modification of the Forward-Forward (FF) algorithm that replaces the FF goodness score with similarity-based objectives (triplet margin loss and (N+1)-tuplet loss). Learning is performed greedily layer-wise with local gradients, and inference uses cached class representatives or centroids in the embedding space, requiring a single forward pass per test input. Three variants are presented: FAUST-vanilla triplet, FAUST-vanilla tuplet, and FAUST-representative tuplet. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 with MLP architectures report that FAUST-representative tuplet achieves 98.39% on MNIST, 89.47% on Fashion-MNIST, and 56.22% on CIFAR-10 (4-layer/800-neuron setting), compared with backpropagation values of 98.64%, 90.40%, and 57.63%, respectively. The authors claim this narrows the gap between FF and backpropagation while improving inference efficiency.

Significance. If the reported results are reproducible, the paper makes a useful contribution to the biologically plausible / backpropagation-free learning literature: it provides a concrete algorithm that replaces FF's goodness objective with a metric-learning objective, offers a layer-wise local training scheme, and achieves near-backpropagation accuracy on CIFAR-10 with a simple MLP and single-pass inference. The explicit pseudocode, t-SNE visualization, and convergence analysis are strengths, as is the attempt to reduce the inference cost of FF. However, the significance is moderated by experimental and algorithmic gaps: the representative-selection rule is unspecified, the gradient flow through representatives is ambiguous, one algorithm step contradicts the prose, and the quantitative comparison includes baselines taken from other papers without re-running under identical conditions. These issues must be resolved before the central claim can be accepted.

major comments (4)
  1. [Section IV-C / Algorithm 1] The representative set R is load-bearing: one image per class is used both as the positive anchor in every training tuplet and as the cached prototype at inference. The manuscript never states how R is selected (random seed, first sample, or chosen to maximize accuracy), nor does it report sensitivity to this choice. Since the same images appear on both sides of the objective, the reported accuracy is conditioned on a particular R; a different choice could materially change the result. Please specify the selection rule, report results over multiple random representative sets and training seeds, and provide a sensitivity analysis.
  2. [Algorithm 1, lines 3–9] It is unclear whether gradients flow through the representative embeddings f^r_i during training. The text says the embeddings are 'cached as fixed reference points for the entire batch' (Section IV-C), which suggests they are detached, but the pseudocode updates W1 and W2 using a loss computed from f_i and f^r_i without specifying a stop-gradient on f^r_i. If representatives are fixed, the positives/negatives never receive gradient and are not adapted to be class-consistent; if they are not fixed, the optimization is different. This ambiguity affects the interpretation of the method and its convergence behavior.
  3. [Algorithm 1 line 10 vs. Section IV-D] Section IV-D states that 'the activations prior to the embedding layer are passed onto subsequent layers,' defining g_{i+1} = phi(W1 g_i) as the input to the next layer. But Algorithm 1 line 10 says 'Pass x = f_i, r = f^r_i to the next layer,' i.e., it passes the embedding, not the activation g_i. This is an internal inconsistency in the central training algorithm. Please correct the pseudocode or the prose, and specify which input representation is actually used by subsequent layers.
  4. [Table 1 / Section VI-A] The experimental comparison is not fully controlled. FFCM and Collab FF results are taken from their original papers (as indicated by the dagger), while the other results are presumably re-run or newly run; architecture, hyperparameter, and data-preprocessing differences can easily explain gaps on the order of 1–2%. In addition, only a single accuracy per configuration is reported, with no error bars or seed variation. On CIFAR-10 the gap between FAUST-representative (56.22%) and BP (57.63%) is 1.41 percentage points, which is within typical seed noise for this architecture. Please report mean and standard deviation over multiple seeds and, for external baselines, either re-run them under identical conditions or clearly state the protocol differences.
minor comments (5)
  1. [Algorithm 1] Typographical issues: lines 4 and 6 use '<-=' instead of '<-'. Also, 'FAUST-vanilla tupletand' appears in Section VI-A without a space.
  2. [Figure 2] The Fisher discriminant score F is used in the caption and text but never defined. Please give the exact formula used to compute F.
  3. [Figure 3] The y-axis label 'Error' with '×10^2' is confusing; the plotted values look like percentages (e.g., 80–20), so the label/scale should be clarified.
  4. [Section V] Hyperparameters are described only as 'fine-tuned' (optimizer, batch size, learning rate). For reproducibility, please report the final hyperparameter values for each dataset and variant, including learning-rate schedule, batch size, embedding size, and margin alpha if used.
  5. [References] Reference [8] is incomplete: it lists authors and year but no title or venue. In addition, reference [22] (Chen et al., SimCLR) should include the publication venue.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FAUST is an empirical algorithm paper; its reported accuracies are measurements, and the only self-referential element (training and inference share the same distance/prototype rule) is standard objective alignment, not circularity.

full rationale

The paper does not derive its reported accuracies from its assumptions by construction. It defines a similarity-based loss (Eqs. 3–4), a training algorithm (Algorithm 1), and a nearest-representative classifier (Eq. 5), then reports held-out test-set measurements in Table 1. No fitted parameter is renamed as a prediction; the representative set R is fixed before training and used symmetrically as positive anchors and inference prototypes, which is a design choice that makes training and inference objectives consistent rather than a circular dependency. All cited methods (Forward-Forward, triplet loss, N-pair loss, embedding projection) are external prior work; the paper contains no self-citations, so there is no load-bearing self-citation chain or imported uniqueness theorem. The conclusion explicitly acknowledges the representative-selection limitation ('While selecting one representative image from each class produces compelling results on the evaluated datasets, other methods of formulating the similarity objective may also be worth exploring'), showing that the issue is disclosed and is a robustness/selection concern, not a circularity concern. The central claims are thus self-contained empirical findings supported by comparison against external baselines.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends mainly on hyperparameters and on the domain assumption that greedy local similarity training creates useful embeddings. No new physical or conceptual entities are introduced beyond the algorithm itself.

free parameters (5)
  • Triplet margin alpha = not reported
    Controls the margin in the triplet loss (Eq. 3); chosen by hand, exact value not given.
  • Embedding size = 256
    Used for FAUST experiments (mentioned in Section V and Fig. 2/3 captions); not tuned across values.
  • Network depth and width = 3 layers of 500 or 4 layers of 800
    Architecture choices in Table 1; fixed across compared algorithms, presumably for fair comparison.
  • Optimizer settings = Adam, lr=0.001, batch=256
    Reported for convergence plots (Fig. 3); exact settings for Table 1 not fully specified.
  • Representative selection rule = unspecified
    The paper says one image per class is extracted, but not how it is chosen; this choice affects both training and inference.
assumptions (4)
  • domain assumption Greedy layer-wise training with local losses yields progressively more discriminative representations
    Core to FF/FAUST; supported by Fig. 2 but not proven.
  • domain assumption Euclidean distance in the embedding space is a valid similarity measure for classification
    Used in Eqs. 3 and 5; standard metric learning assumption.
  • domain assumption A single fixed image per class is representative of that class
    Assumed in FAUST-representative tuplet; no sensitivity analysis.
  • standard math Layer normalization preserves the relative information needed by subsequent layers
    Applied in Section IV; standard technique, not examined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reshaping the Forward-Forward Algorithm with a Similarity-Based Objective." pith.science (2026). https://pith.science/paper/YFO472DF

@misc{pith2026250908697,
  author       = {Pith},
  title        = {Pith review of: Reshaping the Forward-Forward Algorithm with a Similarity-Based Objective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YFO472DF}},
  note         = {Machine review of arXiv:2509.08697}
}
read the original abstract

Backpropagation is the pivotal algorithm underpinning the success of artificial neural networks, yet it has critical limitations such as biologically implausible backward locking and global error propagation. To circumvent these constraints, the Forward-Forward algorithm was proposed as a more biologically plausible method that replaces the backward pass with an additional forward pass. Despite this advantage, the Forward-Forward algorithm significantly trails backpropagation in accuracy, and its optimal form exhibits low inference efficiency due to multiple forward passes required. In this work, the Forward-Forward algorithm is reshaped through its integration with similarity learning frameworks, eliminating the need for multiple forward passes during inference. This proposed algorithm is named Forward-Forward Algorithm Unified with Similarity-based Tuplet loss (FAUST). Empirical evaluations on MNIST, Fashion-MNIST, and CIFAR-10 datasets indicate that FAUST substantially improves accuracy, narrowing the gap with backpropagation. On CIFAR-10, FAUST achieves 56.22\% accuracy with a simple multi-layer perceptron architecture, approaching the backpropagation benchmark of 57.63\% accuracy.

Figures

Figures reproduced from arXiv: 2509.08697 by the authors.

Figure 1
Figure 1. Overall algorithm architecture of FAUST. B. FAUST: Vanilla Approach Our initial endeavor involves the following framework. Training. We begin by utilizing the traditional triplet loss function defined in Section IV-A—this approach is given the code name FAUST-vanilla triplet. Since triplet loss is limited to one negative class at a time, we further adopt the (N + 1)- tuplet formulation described in Section IV-A—this… view at source ↗
Figure 2
Figure 2. t-SNE visualizations of the representations learned by FAUST-representative tuplet on Fashion-MNIST, using an MLP with four layers of 500 neurons and an embedding size of 256. The value in the upper-right corner, F, is the Fisher discriminant score. Color-to-class mapping is (T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle-boot). 20 40 60 80 100 Epochs 52 56 60 64 68 72 76 80 84 Error… view at source ↗
Figure 3
Figure 3. Convergence curves of FF, FAUST, and BP on CIFAR-10, using an MLP with three layers of 500 neurons and an embedding size of 256. All models were trained with the Adam optimizer (learning rate = 0.001, batch size = 256). In the legend, {1} denotes predictions from layer 1 alone, and {1,2,3} represents the combined predictions from all three layers. Note that BP does not provide layer-wise predictions. We compare FAUS… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 14 canonical work pages

  1. [1]

    Learning repre- sentations by back-propagating errors,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning repre- sentations by back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533–536, 1986

    1986

  2. [2]

    Can the brain do backpropagation? -exact implementation of backpropagation in predictive coding networks

    Y . Song, T. Lukasiewicz, Z. Xu, and R. Bogacz, “Can the brain do backpropagation? -exact implementation of backpropagation in predictive coding networks.” Advances in Neural Information Processing Systems, vol. 33, p. 22566–22579, Jan. 2020. [Online]. Available: https://papers.nips.cc/paper/2020/file/fec87a37cdeec1c6ecf8181c0aa2d3bf- Paper.pdf

    work page 2020

  3. [3]

    The forward-forward algorithm: Some preliminary investi- gations,

    G. Hinton, “The forward-forward algorithm: Some preliminary investi- gations,” 2022. [Online]. Available: https://arxiv.org/abs/2212.13345

    arXiv 2022

  4. [4]

    D. O. Hebb, The organization of behavior: A Neuropsychological The- ory. John Wiley & Sons, Jan. 1949

    work page 1949

  5. [5]

    Spike timing–dependent plasticity: A hebbian learning rule,

    N. Caporale and Y . Dan, “Spike timing–dependent plasticity: A hebbian learning rule,” Annual review of neuroscience , vol. 31, pp. 25–46, 02 2008

    work page 2008

  6. [6]

    Facenet: A unified embed- ding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embed- ding for face recognition and clustering,” in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , 2015, pp. 815–823

    work page 2015

  7. [7]

    Symba: Symmetric backpropagation-free contrastive learning with forward-forward algorithm for optimizing con- vergence,

    H.-C. Lee and J. Song, “Symba: Symmetric backpropagation-free contrastive learning with forward-forward algorithm for optimizing con- vergence,” 2023. [Online]. Available: https://arxiv.org/abs/2303.08418

    arXiv 2023

  8. [8]

    E. B. Terres-Escudero, J. D. Ser, and P. G. Bringas, 2025. [Online]. Available: https://arxiv.org/abs/2409.07387

    work page Pith review arXiv 2025

Show all 23 references
  1. [9]

    The trifecta: Three simple techniques for training deeper forward-forward networks,

    T. Dooms, I. J. Tsang, and J. Oramas, “The trifecta: Three simple techniques for training deeper forward-forward networks,” 2023. [Online]. Available: https://arxiv.org/abs/2311.18130

    2023 arXiv

  2. [10]

    Convolutional channel-wise competitive learning for the forward- forward algorithm,

    A. Papachristodoulou, C. Kyrkou, S. Timotheou, and T. Theocharides, “Convolutional channel-wise competitive learning for the forward- forward algorithm,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 13, p. 14536–14544, Mar. 2024. [Online]. Availa...

    2024 doi

  3. [11]

    Training convolutional neural networks with the forward-forward algorithm,

    R. Scodellaro, A. Kulkarni, F. Alves, and M. Schr ¨oter, “Training convolutional neural networks with the forward-forward algorithm,”

  4. [12]

    Layer collaboration in the forward-forward algorithm,

    G. Lorberbom, I. Gat, Y . Adi, A. Schwing, and T. Hazan, “Layer collaboration in the forward-forward algorithm,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 13, pp. 14 141–14 148, Mar. 2024. [Online]. Available: https://ojs.aaai.org/index.php/A...

    2024

  5. [13]

    Marginal contrastive loss: A step forward for forward-forward,

    H. Aghagolzadeh and M. Ezoji, “Marginal contrastive loss: A step forward for forward-forward,” in 2024 13th Iranian/3rd International Machine Vision and Image Processing Conference (MVIP) , 2024, pp. 1–6

    2024

  6. [14]

    Contrastive forward-forward: A training algorithm of vision transformer,

    ——, “Contrastive forward-forward: A training algorithm of vision transformer,” 2025. [Online]. Available: https://arxiv.org/abs/2502.00571

    2025

  7. [15]

    Learning a similarity metric discriminatively, with application to face verification,

    S. Chopra, R. Hadsell, and Y . LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition (CVPR’05), vol. 1, 2005, pp. 539–546 vol. 1

    2005

  8. [16]

    In defense of the triplet loss for person re-identification,

    A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” 2017. [Online]. Available: https://arxiv.org/abs/1703.07737

    2017 arXiv

  9. [17]

    Improved deep metric learning with multi-class n-pair loss objective,

    K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” in Advances in Neural Information Processing Systems , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29. Curran Associates, Inc., 2016

    2016

  10. [18]

    Prototypical networks for few-shot learning,

    J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” 2017. [Online]. Available: https://arxiv.org/abs/1703.05175

    2017 arXiv

  11. [19]

    Deepncm: Deep nearest class mean classifiers,

    S. Guerriero, B. Caputo, and T. Mensink, “Deepncm: Deep nearest class mean classifiers,” in ICLR 2018 Workshop , 2018. [Online]. Available: https://openreview.net/forum?id=rkPLZ4JPM

    2018

  12. [20]

    A theoretically sound upper bound on the triplet loss for improving the efficiency of deep distance metric learning,

    T.-T. Do, T. Tran, I. Reid, V . Kumar, T. Hoang, and G. Carneiro, “A theoretically sound upper bound on the triplet loss for improving the efficiency of deep distance metric learning,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 10...

    2019

  13. [21]

    Infinite mixture prototypes for few-shot learning,

    K. R. Allen, E. Shelhamer, H. Shin, and J. B. Tenenbaum, “Infinite mixture prototypes for few-shot learning,” 2019. [Online]. Available: https://arxiv.org/abs/1902.04552

    2019 arXiv

  14. [22]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” 2020. 6

    2020

  15. [2024]

    Available: https://arxiv.org/abs/2312.14924

    [Online]. Available: https://arxiv.org/abs/2312.14924

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.