Pith. sign in

REVIEW 2 major objections 7 minor 64 references

Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm

T0 review · 2 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Binary stochastic forward-forward trains networks to near-real accuracy using roughly a tenth of the energy of standard training algorithms.

desk verdict A credible binary-stochastic forward-forward algorithm with an honest variational derivation, whose headline 10x energy savings is a projection that needs hardware-grounded cost validation. read the letter →

arxiv 2507.06461 v1 pith:F5ELXJQQ submitted 2025-07-09 cs.LG cs.NE

Risi Jaiswal , Supriyo Datta , Joseph G. Makin This is my paper
classification cs.LGcs.NE
keywords forward-forwardalgorithmbinarystochasticneuronsprobabilisticbits(p-bits)energy-efficienttrainingvariationalupperboundtiledlogisticunitshardwareacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that supervised training need not be a floating-point backpropagation process: a forward-forward algorithm with binary, stochastic hidden units can match the accuracy of its real-valued counterpart on MNIST, Fashion-MNIST, and CIFAR-10 while cutting estimated training energy by about one order of magnitude (one to two orders when compute-bound). This matters because backpropagation's serial dependencies and activation storage are hard to accelerate in custom low-power hardware, whereas binary activations turn matrix multiplications into indexing operations that p-bit devices could in principle perform cheaply. The binary learning rule is derived as the gradient of a variational upper bound, so it is a principled training objective rather than a heuristic hack. If the hardware assumptions hold, this points toward on-device learning at a fraction of today's energy cost.

What carries the argument

The load-bearing object is the binary stochastic forward-forward learning rule (13): a gradient step for a single row of weights that multiplies the class-probability error by an integer 'surprise' factor — one when the unit's Bernoulli sample disagrees with whether its firing probability is above or below 1/2 — and by the binary input. The rule is derived by minimizing a variational upper bound on the true conditional cross-entropy, using a one-sample Monte Carlo estimate of the expected class probabilities. Its hardware appeal is that binary activations reduce matrix multiplication to indexing-and-adding, and the real-valued factors that survive are folded into per-channel constants or into the diagonal Jacobian, leaving almost no full-width multiplications outside the first convolutional layer. Tiled logistic units, which sum several p-bit samples with shifted biases, approximate ReLU-like activations and recover accuracy as the number of tied units grows.

What would settle it

Build or simulate a small p-bit-based accelerator implementing BGBSFF and directly measure end-to-end training energy per sample on MNIST; if the measured energy per sample divided by the energy of a comparable backprop-trained model is not near 0.1 or below after counting p-bit and interface energy, the paper's central quantitative claim is overturned for that implementation. A purely algorithmic falsifier would be a dataset or architecture where binarized stochastic forward-forward loses far more than a few points of accuracy relative to CwC-FF despite enough tied units.

Watch

Extended reading notes

Core claim

The central discovery is that stochastic binarization need not cost much accuracy in forward-forward learning. The authors start from the channelwise-competitive forward-forward (CwC-FF) algorithm, where each layer solves the classification problem locally through a fixed readout matrix, and replace real-valued hidden activations with Bernoulli samples from logistic units. They show that the correct learning signal is obtained by descending the gradient of a variational upper bound on the marginal cross-entropy, approximated with a single Monte Carlo sample (BSFF), and that the real-valued derivative hill can itself be replaced by an integer-valued surprise function (BGBSFF) with little accuracy loss. With several tied binary units per activation ('tiled logistic units'), accuracy approaches or matches the real-valued CwC-FF baseline on all three datasets. The paper calculates that the resulting algorithm replaces nearly all multiplications with indexing and binary memory traffic, yielding an estimated one-order-of-magnitude energy saving in the memory-bound regime and one to two orders in the compute-bound regime.

Load-bearing premise

The headline energy saving assumes that drawing a Bernoulli sample from a p-bit and performing binary indexing costs almost no energy compared with real-valued multiplication and memory access, so the estimate leaves out the energy of the p-bit devices themselves and of the analog-to-digital interfaces that would connect them to the rest of the chip; if those costs are substantial, the advertised saving shrinks.

Editorial extensions

If this is right

  • On MNIST, Fashion-MNIST, and CIFAR-10, BGBSFF with seven tied units reaches test accuracies close to the real-valued CwC-FF baseline (the paper reports 99.3%, 89.5%, and 72.4%, respectively).
  • Because binary activations convert convolutions into indexing operations, the dominant multiplications are confined to the first layer plus a small per-channel BatchNorm adjustment, which is what produces the estimated 10x to 100x energy reduction.
  • The binarization recipe transfers across the forward-forward family, so as future forward-only algorithms improve, they should be binarizable with similarly small accuracy loss.
  • Dropping BatchNorm and computing the loss at max-pooling makes BGBSFF fully binary, trading accuracy on harder datasets for an additional order-of-magnitude energy saving.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A working p-bit implementation would turn the energy estimate into a measurement; the most direct test is a small chip or FPGA-emulated p-bit array training a single convolutional layer end to end.
  • Because the derivation only relies on local objectives and Bernoulli sampling, the same binarization recipe could be applied to other forward-only training schemes, potentially lowering their energy costs by a similar factor.
  • The accuracy drop without BatchNorm suggests that a cheap, analog-friendly replacement for batch normalization could be the key to closing the remaining gap on more difficult datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper derives and evaluates two binary, stochastic forward-forward algorithms, BSFF and BGBSFF, for supervised learning. Starting from a real-valued logistic forward-forward model (Eqs. 4-7), the authors replace deterministic activations with Bernoulli hidden units, minimize a variational upper bound J (Eq. 9) on the marginal relative entropy, and obtain a tractable gradient (Eq. 10) whose class-probability expectation is approximated by a single sample (Eq. 11). An integer-valued surrogate for the logistic Jacobian (Eq. 12) yields the fully-binary-gradient variant BGBSFF (Eq. 13). To recover accuracy lost by binarization, 'tiled logistic units' (Eq. 14) sum M tied Bernoulli samples with linearly spaced biases to approximate a ReLU. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 with the CwC-FF architecture (Fig. 3, Tables II and IV) show that the M=7 variants reach 99.3%, 89.5%, and 72.4%, respectively, close to the real-valued CwC-FF baseline (91.4% and 76.3% on FMNIST and CIFAR-10). The energy analysis (Section V.A, Appendix B) counts 32-bit multiplications and word-level memory accesses under a Von-Neumann model, converts binary convolutions into indexing operations, and projects savings of roughly 10x (memory-dominated) to 10-100x (compute-dominated) versus backpropagation and real-valued CwC-FF, assuming p-bit sampling and indexing costs are 'comparatively negligible' (Section V.A.b).

Significance. The paper's main methodological asset is that the binary learning rule is derived rather than postulated: Eq. (13) descends the gradient of the variational upper bound in Eq. (9), with all approximations (one-sample Monte Carlo, the large-D directional derivative in Appendix A, and the integer Jacobian surrogate (12)) stated explicitly; the absence of fitted parameters in the energy accounting (fixed architecture counts rather than calibrated coefficients) is a further strength. The empirical protocol is transparent: five independent runs per configuration, full hyperparameter tables (Table III), explicit disclosure of the BSN:1 learning-rate tuning, and candid reporting of the no-BatchNorm experiment (Table II) and of preliminary CIFAR-100 results. The central accuracy claim is credible, with BGBSFF:7 within roughly one point of CwC-FF on FMNIST and a few points on CIFAR-10, and the proposal that binary activations reduce convolutions to indexing operations is a testable, falsifiable hardware prediction.

major comments (2)
  1. [Section V.A.b, Section V.A.c, Appendix B] The claim that p-bit sampling and binary indexing have 'comparatively negligible' energy costs is the load-bearing premise for the headline savings, but it is asserted rather than modeled or measured. The cost model counts 32-bit multiplications and word-level memory accesses only; it excludes the bias-current energy of the MTJ p-bits themselves (refs. [30], [43]), the sense amplifiers needed to convert each p-bit's analog output into a digital bit, and the M-fold multiplication of these per-activation costs in tiled units (Section IV.B; M=7 is used for the best CIFAR-10 accuracy, Tables II and IV). The quantitative risk is most acute in the memory-dominated regime: the model charges BSFF 1/32 of a word access per activation, but every activation also requires one (or M) p-bit sample-and-sense events; if a single sense event costs even a small fraction of a 32-bit word access, the factor-32 memory savings erodes, and for the M=7 case the claimed margin is already only about 10.7x (Section V.A.c). I request a sensitivity analysis: add per-sample sampling-plus-sensing energy and per-activation readout energy into the Appendix B counts, calibrate these to the cited p-bit literature where possible, and report the crossover at which the savings fall below 10x; alternatively, qualify the abstract's energy claim accordingly.
  2. [Section V.B, Table IV, Fig. 3] The statement that 'performance is roughly equal to performance with the standard (real-valued, deterministic) CwC-FF on all three datasets' overstates the CIFAR-10 result. On CIFAR-10, BGBSFF:7 reaches 72.4 ± 0.18 versus 76.32 ± 0.30 for CwC-FF (Table IV; the same gap is visible in Fig. 3), a roughly 3.9-point difference with non-overlapping standard errors, and BSFF:7 consistently outperforms BGBSFF:7 on this dataset, so the fully-binary gradient costs additional accuracy. The paper's overall claim remains defensible, since relative to the 10-20-point gap between forward-forward and backpropagation on CIFAR-10 (Section VI.A) a few points is indeed close, but the main text should state the residual gap quantitatively and anchor it to the M=7 configuration rather than saying 'roughly equal.'
minor comments (7)
  1. [Appendix C.A] The code availability statement reads 'can be accessed at this repository' without supplying a URL; the actual repository link must be provided for the experiments to be independently verifiable.
  2. [Table IV] The column header duplicates 'CwCFF' and the data columns appear misaligned with the header: the third data column (77.47 on FMNIST, 53.5 on CIFAR-10) numerically matches BGBSFF:1 in Table II, so the header omits the BGBSFF:1 through BGBSFF:7 labels and the CIFAR-10 row is missing its BGBSFF:7 entry.
  3. [Section VI.A] The claim that binary-stochasticizing Hinton's forward-forward algorithm 'only negligibly reduces' its accuracy is unsupported, since the parenthetical states that those results were not included; either report the results or remove the claim.
  4. [Section VI.C] The assumption of a digital architecture is stated only in the Discussion; Section V.A and Appendix B should state this assumption where the 32-bit first-layer input and the exclusion of analog-to-digital conversion costs are introduced.
  5. [Appendix B] The opening 'Notation Disclaimer' says the appendix notation 'may differ from or be independent of the notation used in the main text'; Table I and Appendix B should use one shared notation for N, C, H, W, K, and L so that the reader can verify that the dominant terms of Table I match the appendix counts.
  6. [Throughout] Typos and small textual errors: 'foward-pass-only' (Section I), the duplicated 'which is which is' (Section V.B), 'Per Layes' (Appendix B.A), and an unbalanced parenthesis in 'In large networks > 10 layers)' (Section VI.B).
  7. [Section III.B] The tightness argument for the variational bound (V provides more bits of information about the hidden units than Z, which provides only log K) is heuristic; stating explicitly that the gap equals I(\hat U; Z | V) \le \log K would make the scope of the claim precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the learning rule is derived from an explicit variational upper bound and validated against external benchmarks; energy figures are operation counts under a stated cost model.

full rationale

The derivation chain is self-contained. The BSFF update (Eq. 11) is obtained as the gradient of an explicit variational upper bound (Eq. 9, Appendix A), and the BGBSFF update (Eq. 13) is a stated integer-valued approximation to Eq. 11 whose expectation is plotted against al(1−al) in Fig. 2a. Replacing the conditional expectation with a single sample is a standard Monte Carlo estimator, not a redefinition of the target. Accuracy is measured on MNIST, FMNIST, and CIFAR-10 without fitting a parameter to those accuracies, and the energy projections are operation counts from Appendix B under an explicit cost model. The p-bit citations ([30], [39], [43]) involve overlapping authors, but they are experimental/device results cited as external evidence for device speed and behavior, not a theorem whose conclusion is the paper's own claim. The main weakness—omitting p-bit sampling and ADC/readout energy from the cost model—is an estimation-risk concern, not a circular reduction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claim rests on architectural choices (M), an asymptotic approximation, single-sample estimation, and the key hardware assumptions of negligible p-bit and indexing costs. These are reasonable but not proven. M=7 is a hand-chosen parameter tuned to accuracy, and the hardware assumptions are the most uncertain part of the energy projection.

free parameters (2)
  • M, number of tied Bernoulli samples per tiled logistic unit = 7 (best accuracy; 1, 2, 3 also evaluated)
    Chosen by hand as the size of the stochastic approximation to a ReLU; controls the accuracy-energy trade-off.
  • Learning rate for the BSN:1 configuration on MNIST and FMNIST = 5e-4/5e-3 (MNIST), 1e-4/1e-3 (FMNIST)
    Admitted in Section IV-D: for the single-Bernoulli-unit case, performance was 'quite variable' without tuning, so the learning rate was tuned for these two datasets.
assumptions (5)
  • domain assumption The hidden dimension D is large enough for the asymptotic approximation in Eq (19) to hold.
    The derivation of Eq (10) replaces a difference of log-partition functions with a directional derivative, which is exact only as D grows large (Appendix A, Eq 19).
  • domain assumption A single sample from each hidden layer is a sufficient estimate of the conditional expectation in Eq (10).
    Eq (11) uses one sample u-hat per input to approximate the full expectation; the paper relies on training over many batches to average out the sampling noise.
  • domain assumption p-bits provide Bernoulli samples with negligible energy and area cost.
    Section II-C describes p-bits from prior device literature; the energy calculation in Section V.A assumes binary sampling is effectively free.
  • domain assumption Matrix multiplication with binary activations reduces to indexing operations with negligible cost.
    This is the basis for the compute savings in the energy comparison, stated in Section I and Section V.A.
  • ad hoc to paper The sum of M logistic functions with linearly spaced biases approximates a ReLU or softplus well enough.
    The tiled logistic units (Eq 14) are introduced to approximate ReLU; the approximation is validated only empirically through downstream accuracy.
invented entities (1)
  • Tiled logistic unit (a stochastic unit composed of M tied Bernoulli samples with offset thresholds)
    purpose: Approximates a ReLU activation while producing binary outputs, so matrix multiplications remain index/add operations.
    New construction introduced in Section IV-B; its usefulness is supported only by the accuracy results in this paper, not by external evidence or theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm." pith.science (2026). https://pith.science/paper/F5ELXJQQ

@misc{pith2026250706461,
  author       = {Pith},
  title        = {Pith review of: Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5ELXJQQ}},
  note         = {Machine review of arXiv:2507.06461}
}
read the original abstract

Reducing energy consumption has become a pressing need for modern machine learning, which has achieved many of its most impressive results by scaling to larger and more energy-consumptive neural networks. Unfortunately, the main algorithm for training such networks, backpropagation, poses significant challenges for custom hardware accelerators, due to both its serial dependencies and the memory footprint needed to store forward activations for the backward pass. Alternatives to backprop, although less effective, do exist; here the main computational bottleneck becomes matrix multiplication. In this study, we derive forward-forward algorithms for binary, stochastic units. Binarization of the activations transforms matrix multiplications into indexing operations, which can be executed efficiently in hardware. Stochasticity, combined with tied weights across units with different biases, bypasses the information bottleneck imposed by binary units. Furthermore, although slow and expensive in traditional hardware, binary sampling that is very fast can be implemented cheaply with p-bits (probabilistic bits), novel devices made up of unstable magnets. We evaluate our proposed algorithms on the MNIST, Fashion-MNIST, and CIFAR-10 datasets, showing that its performance is close to real-valued forward-forward, but with an estimated energy savings of about one order of magnitude.

Figures

Figures reproduced from arXiv: 2507.06461 by the authors.

Figure 1
Figure 1. Discriminative models for (a) deterministic and (b) binary-stochastic forward-forward. In Hinton’s FF, V corresponds to labeled images and Z to indicates whether the label is good or bad; in CwC-FF, V is the image and Z the one-hot label. Each of the K rows of Wout corresponds to a category. Thus it “assigns” subsets of the activities to different categories and then simply averages activities within these subsets. … view at source ↗
Figure 2
Figure 2. Approximation functions. (a) Exact (blue) and expected approximate (green) derivatives of the logistic function. (b) The response functions for an exact rectified linear unit (blue) and “tiled logistic units,” with one (orange), two (green), three (red), and seven (purple) tied units. Each tiled-logistic response function corresponds to a single realization. Substituting (12) for al(1 − al) in (11) yields dJ dwl in … view at source ↗
Figure 3
Figure 3. Comparison of accuracies for the MNIST, FMNIST, and CIFAR-10 datasets. The leftmost boxplot represents results with ReLU activation, i.e. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Simple computation model used in Appendix [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: Prediction error convergence plots for MNIST, FMNIST, and CIFAR-10 (from left to right). [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 47 canonical work pages

  1. [30]

    Subnanosecond fluctuations in low-barrier nanomagnets,

    J. Kaiser, A. Rustagi, K. Y . Camsari, J. Z. Sun, S. Datta, and P. Upad- hyaya, “Subnanosecond fluctuations in low-barrier nanomagnets,” Phys- ical Review Applied , vol. 12, 11 2019

    work page 2019

  2. [43]

    Integer factorization using stochastic magnetic tunnel junctions,

    W. A. Borders, A. Z. Pervaiz, S. Fukami, K. Y . Camsari, H. Ohno, and S. Datta, “Integer factorization using stochastic magnetic tunnel junctions,” Nature, vol. 573, pp. 390–393, 9 2019

    work page 2019

  3. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 10

    work page

  4. [2]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural informa- tion processing systems , vol. 25, 2012

    2012

  5. [3]

    Deep speech: Scaling up end-to-end speech recognition,

    A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y . Ng, “Deep speech: Scaling up end-to-end speech recognition,” 12 2014. [Online]. Available: http://arxiv.org/abs/1412.5567

    arXiv 2014

  6. [4]

    Highly accurate protein structure prediction with alphafold,

    J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. ˇZ´ıdek, A. Potapenko, A. Bridgland, C. Meyer, S. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Boden- stein,...

    work page 2021

  7. [5]

    The cost of training machines is becoming a problem,

    “The cost of training machines is becoming a problem,” The Economist, 2020, accessed: 2024-08-12

    work page 2020

  8. [6]

    Ai’s growing demand for data centers is threatening tech’s climate goals,

    D. Milmo, A. Hern, and J. Ambrose, “Ai’s growing demand for data centers is threatening tech’s climate goals,” , Jul. 2024, accessed: 2024- 08-12

    work page 2024

Show all 64 references
  1. [7]

    Ai is an energy hog. this is what it means for climate change,

    C. Crownhart, “Ai is an energy hog. this is what it means for climate change,” , May 2024, accessed: 2024-08-12

    2024

  2. [8]

    Mobilenets: Efficient convolutional neural networks for mobile vision applications,

    A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” 4 2017. [Online]. Available: http://arxiv.org/abs/1704.04861

    2017 arXiv

  3. [9]

    Fully quantized network for object detection,

    R. Li, Y . Wang, F. Liang, H. Qin, J. Yan, and R. Fan, “Fully quantized network for object detection,” in Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition , 2019, pp. 2810–2819

    2019

  4. [10]

    Puma: A programmable ultra-efficient memristor-based accelerator for machine learning inference,

    A. Ankit, I. E. Hajj, S. R. Chalamalasetti, G. Ndu, M. Foltin, R. S. Williams, P. Faraboschi, W. M. Hwu, J. P. Strachan, K. Roy, and D. S. Milojicic, “Puma: A programmable ultra-efficient memristor-based accelerator for machine learning inference,” in International Conference ...

    2019

  5. [11]

    A survey of quantization methods for efficient neural network infer- ence,

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network infer- ence,” in Low-Power Computer Vision. Chapman and Hall/CRC, 2022, pp. 291–326

    2022

  6. [12]

    Quantized neural networks: Training neural networks with low preci- sion weights and activations,

    I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y . Bengio, “Quantized neural networks: Training neural networks with low preci- sion weights and activations,” pp. 1–30, 2018

    2018

  7. [13]

    Learning repre- sentations by back-propagating errors,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning repre- sentations by back-propagating errors,” nature, vol. 323, no. 6088, pp. 533–536, 1986

    1986

  8. [14]

    Hardware implementation of memristor-based artificial neural networks,

    F. Aguirre, A. Sebastian, M. L. Gallo, W. Song, T. Wang, J. J. Yang, W. Lu, M. F. Chang, D. Ielmini, Y . Yang, A. Mehonic, A. Kenyon, M. A. Villena, J. B. Rold ´an, Y . Wu, H. H. Hsu, N. Raghavan, J. Su ˜n´e, E. Miranda, A. Eltawil, G. Setti, K. Smagulova, K. N. Salama, O. Kre...

    2024

  9. [15]

    Re- cent progress in analog memory-based accelerators for deep learning,

    H. Tsai, S. Ambrogio, P. Narayanan, R. M. Shelby, and G. W. Burr, “Re- cent progress in analog memory-based accelerators for deep learning,” 6 2018

    2018

  10. [16]

    Dot-product engine for neuro- morphic computing: Programming 1t1m crossbar to accelerate matrix- vector multiplication,

    M. Hu, J. P. Strachan, Z. Li, E. M. Grafals, N. Davila, C. Graves, S. Lam, N. Ge, J. J. Yang, and R. S. Williams, “Dot-product engine for neuro- morphic computing: Programming 1t1m crossbar to accelerate matrix- vector multiplication,” in Proceedings - Design Automation Confer...

    2016

  11. [17]

    Isaac: A con- volutional neural network accelerator with in-situ analog arithmetic in crossbars,

    A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V . Srikumar, “Isaac: A con- volutional neural network accelerator with in-situ analog arithmetic in crossbars,” in Proceedings - 2016 43rd International Symposium on Computer ...

    2016

  12. [18]

    Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks,

    C. Frenkel, M. Lefebvre, and D. Bol, “Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks,” Frontiers in Neuroscience, vol. 15, 2 2021

    2021

  13. [19]

    The forward-forward algorithm: Some preliminary investigations,

    G. Hinton, “The forward-forward algorithm: Some preliminary investigations,” 12 2022. [Online]. Available: http://arxiv.org/abs/2212. 13345

    2022

  14. [20]

    Signal propagation: A framework for learning and inference in a forward pass,

    A. Kohan, E. A. Rietman, and H. T. Siegelmann, “Signal propagation: A framework for learning and inference in a forward pass,” 4 2022. [Online]. Available: http://arxiv.org/abs/2204.01723

    2022 arXiv

  15. [21]

    Error-driven input modulation: Solving the credit assignment problem without a backward pass,

    G. Dellaferrera and G. Kreiman, “Error-driven input modulation: Solving the credit assignment problem without a backward pass,” 1

  16. [22]

    One forward is enough for neural network training via likelihood ratio method,

    J. Jiang, Z. Zhang, C. Xu, Z. Yu, and Y . Peng, “One forward is enough for neural network training via likelihood ratio method,” arXiv preprint arXiv:2305.08960, 2023

    2023 arXiv

  17. [23]

    Exploiting inherent error resiliency of deep neural networks to achieve extreme energy efficiency through mixed-signal neurons,

    B. Chatterjee, P. Panda, S. Maity, A. Biswas, K. Roy, and S. Sen, “Exploiting inherent error resiliency of deep neural networks to achieve extreme energy efficiency through mixed-signal neurons,” IEEE Trans- actions on Very Large Scale Integration (VLSI) Systems , vol. 27, no....

    2019

  18. [24]

    Training deep neural networks with low precision multiplications,

    M. Courbariaux, Y . Bengio, and J.-P. David, “Training deep neural networks with low precision multiplications,” 12 2014. [Online]. Available: http://arxiv.org/abs/1412.7024

    2014 arXiv

  19. [25]

    Binary neural networks: A survey,

    H. Qin, R. Gong, X. Liu, X. Bai, J. Song, and N. Sebe, “Binary neural networks: A survey,” 3 2020. [Online]. Available: http://arxiv.org/abs/ 2004.03333http://dx.doi.org/10.1016/j.patcog.2020.107281

    2020 arXiv

  20. [26]

    Probabilistic binary neural networks,

    J. W. T. Peters and M. Welling, “Probabilistic binary neural networks,” 9 2018. [Online]. Available: http://arxiv.org/abs/1809.03368

    2018 arXiv

  21. [27]

    Accelerating binarized convolutional neural networks with software-programmable fpgas,

    R. Zhao, W. Song, W. Zhang, T. Xing, J.-H. Lin, M. Srivastava, R. Gupta, and Z. Zhang, “Accelerating binarized convolutional neural networks with software-programmable fpgas,” in Proceedings of the 2017 ACM/SIGDA international symposium on field-programmable gate arrays, 2017,...

    2017

  22. [28]

    Accelerating binarized neural networks: Comparison of fpga, cpu, gpu, and asic,

    E. Nurvitadhi, D. Sheffield, J. Sim, A. Mishra, G. Venkatesh, and D. Marr, “Accelerating binarized neural networks: Comparison of fpga, cpu, gpu, and asic,” in 2016 International Conference on Field- Programmable Technology (FPT). IEEE, 2016, pp. 77–84

    2016

  23. [29]

    Deep learning binary neural network on an fpga,

    Y . Zhou, S. Redkar, and X. Huang, “Deep learning binary neural network on an fpga,” in 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS) . IEEE, 2017, pp. 281–284

    2017

  24. [31]

    Convolutional channel-wise competitive learning for the forward- forward algorithm,

    A. Papachristodoulou, C. Kyrkou, S. Timotheou, and T. Theocharides, “Convolutional channel-wise competitive learning for the forward- forward algorithm,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 13, 2024, pp. 14 536–14 544

    2024

  25. [32]

    Ran- dom synaptic feedback weights support error backpropagation for deep learning,

    T. P. Lillicrap, D. Cownden, D. B. Tweed, and C. J. Akerman, “Ran- dom synaptic feedback weights support error backpropagation for deep learning,” Nature Communications, vol. 7, 11 2016

    2016

  26. [33]

    Direct feedback alignment provides learning in deep neural networks,

    A. Nøkland, “Direct feedback alignment provides learning in deep neural networks,” 9 2016. [Online]. Available: http://arxiv.org/abs/1609.01596

    2016 arXiv

  27. [34]

    The cas- caded forward algorithm for neural network training,

    G. Zhao, T. Wang, Y . Li, Y . Jin, C. Lang, and H. Ling, “The cas- caded forward algorithm for neural network training,” arXiv preprint arXiv:2303.09728, 2023

    2023 arXiv

  28. [35]

    Equilibrium propagation: Bridging the gap between energy-based models and backpropagation,

    B. Scellier and Y . Bengio, “Equilibrium propagation: Bridging the gap between energy-based models and backpropagation,” Frontiers in Computational Neuroscience, vol. 11, 5 2017

    2017

  29. [36]

    Agnostic physics-driven deep learning,

    B. Scellier, S. Mishra, Y . Bengio, and Y . Ollivier, “Agnostic physics-driven deep learning,” 5 2022. [Online]. Available: http: //arxiv.org/abs/2205.15021 10

    2022 arXiv

  30. [37]

    Estimating or propagating gradients through stochastic neurons for conditional computation,

    Y . Bengio, N. L ´eonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” 8

  31. [38]

    Reintroducing straight-through estimators as principled methods for stochastic binary networks,

    A. Shekhovtsov and V . Yanush, “Reintroducing straight-through estimators as principled methods for stochastic binary networks,” 6

  32. [39]

    Autonomous probabilistic coprocessing with petaflips per second,

    B. Sutton, R. Faria, L. A. Ghantasala, R. Jaiswal, K. Y . Camsari, and S. Datta, “Autonomous probabilistic coprocessing with petaflips per second,” IEEE Access, vol. 8, pp. 157 238–157 252, 2020

    2020

  33. [40]

    Probabilistic computing with p-bits,

    J. Kaiser and S. Datta, “Probabilistic computing with p-bits,” Applied Physics Letters, vol. 119, no. 15, 2021

    2021

  34. [41]

    Massively parallel probabilistic computing with sparse ising machines,

    N. A. Aadit, A. Grimaldi, M. Carpentieri, L. Theogarajan, J. M. Martinis, G. Finocchio, and K. Y . Camsari, “Massively parallel probabilistic computing with sparse ising machines,” 10 2021. [Online]. Available: http://arxiv.org/abs/2110.02481http://dx.doi.org/10. 1038/s41928-0...

    2021 arXiv

  35. [42]

    Stochastic p-bits for invertible logic,

    K. Y . Camsari, R. Faria, B. M. Sutton, and S. Datta, “Stochastic p-bits for invertible logic,” Physical Review X , vol. 7, 7 2017

    2017

  36. [44]

    A view of the EM algorithm that justifies incremental, sparse, and other variants,

    R. M. Neal and G. E. Hinton, “A view of the EM algorithm that justifies incremental, sparse, and other variants,” Learning in Graphical Models , 1998

    1998

  37. [45]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift,

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” 2015. [Online]. Available: https://arxiv.org/abs/1502.03167

    2015 arXiv

  38. [46]

    The mnist database of handwritten digit images for machine learning research,

    L. Deng, “The mnist database of handwritten digit images for machine learning research,” IEEE Signal Processing Magazine , vol. 29, no. 6, pp. 141–142, 2012

    2012

  39. [47]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,

    H. Xiao, K. Rasul, and R. V ollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” 2017. [Online]. Available: https://github.com/zalandoresearch/fashion-mnist

    2017

  40. [48]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” 2009

    2009

  41. [49]

    Ai and memory wall,

    A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer, “Ai and memory wall,” IEEE Micro, 2024

    2024

  42. [50]

    Evolution of convolutional neural network (cnn): Compute vs memory bandwidth for edge ai,

    D. Chenna, “Evolution of convolutional neural network (cnn): Compute vs memory bandwidth for edge ai,” arXiv preprint arXiv:2311.12816 , 2023

    2023 arXiv

  43. [51]

    1.1 computing’s energy problem (and what we can do about it),

    M. Horowitz, “1.1 computing’s energy problem (and what we can do about it),” in 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC) . IEEE, 2014, pp. 10–14

    2014

  44. [52]

    A method to estimate the energy consumption of deep neural networks,

    T.-J. Yang, Y .-H. Chen, J. Emer, and V . Sze, “A method to estimate the energy consumption of deep neural networks,” in 2017 51st asilomar conference on signals, systems, and computers . IEEE, 2017, pp. 1916– 1920. 11 APPENDIX A LOSS FUNCTIONS FOR BINARY STOCHASTIC FORWARD -F...

    2017

  45. [57]

    Forward Pass(layer ℓ): z(ℓ) = w ∗ u(ℓ−1) (21) a(ℓ) = ReLU(z(ℓ)) (22) ˜u(ℓ) = max pool a(ℓ) , (23) u(ℓ) = BN ˜u(ℓ) . (24) 14 Algorithm 1 ForwardPassOneLayer Require: Layer index ℓ, batch size N Ensure: Batch-normalized outputs u for layer ℓ {First Pass: Accumulate Mean & Varian...

  46. [58]

    The gradient of the Maxpool operation involves only an indexing operation through the maximum element in each patch

    Backward Pass: To compute the weight gradient ∂L ∂W (ℓ) for any layer ℓ, the following steps are followed. The gradient of the Maxpool operation involves only an indexing operation through the maximum element in each patch. Therefore, for 15 simplicity, we combine the BatchNor...

  47. [59]

    Initialize for the last layer: ∆(L)[c, h, w] = ∂L ∂u(L)[c, h, w] for each channel c and spatial location (h, w)

  48. [60]

    goodness

    Backward pass for layers ℓ = L down to 1: a) Compute the gradient w.r.t. pre-activation z(ℓ): ∂L ∂z (ℓ)[c, h, w] = ∆ (ℓ)[c, h, w] · h ∂u(ℓ)[c, h, w] ∂ ˜u(ℓ)[c, h, w]| {z } BN derivative · ∂a(ℓ)[c, h, w] ∂z (ℓ)[c, h, w]| {z } ReLU or BSN derivative i Here, ∆(ℓ)[ c, h, w] is the...

  49. [61]

    Forward Pass (layer ℓ): z(ℓ)[c] = X c′ W (ℓ)[ c, c′ ] ∗ u(ℓ−1)[ c′ ] (convolution) (25) a(ℓ)[c] = ReLU z(ℓ)[c] (activation) (26) ˜u(ℓ)[c] = max pool a(ℓ)[c] (max pooling) (27) u(ℓ)[c] = BN ˜u(ℓ)[c] (batch normalization) (28) • Total Number of Multiplications (per layer) = N C(...

  50. [62]

    Weight Update at layer ℓ: ∂L(ℓ) ∂w (ℓ): Here the loss is defined by grouping channels c into subsets Sj. ∂L ∂z (ℓ) [n, c, h, w] = NclassX j=1 ∂L ∂Gj ∂Gj ∂u(ℓ)[c, n, h, w] · ∂u(ℓ)[n, c, h, w] ∂a(ℓ)[n, c, h, w] · ∂a(ℓ)[n, c, h, w] ∂z (ℓ)[n, c, h, w] , (29) ∂L ∂w (ℓ)[ c, c′, k1, ...

  51. [63]

    Rewriting the pre-activation z(ℓ)[c] to highlight these terms: z(ℓ)[c] = C(ℓ−1) X c(ℓ−1)=1 w(ℓ)[c, c(ℓ−1)] α(ℓ−1)[c(ℓ−1)] ∗ u(ℓ−1)[c(ℓ−1)] + w(ℓ)[c, c(ℓ−1)] δ(ℓ−1)[c(ℓ−1)]

    Forward Pass (layer ℓ): z(ℓ)[c] = w(ℓ)[c, ·] ∗ u(ℓ−1)[·] (31) p(ℓ)[c, i] = σ z(ℓ)[c, i] (32) a(ℓ)[c, i] ∼ Bernoulli p(ℓ)[c, i] ∀ c, i (33) ˜u(ℓ)[c] = max pool a(ℓ)[c] (34) u(ℓ)[c] = BN ˜u(ℓ)[c] (35) Because the BSN activation outputs are in the set {0, 1} u(ℓ)[c] ∈ n − µ[c]√ σ...

  52. [64]

    Weight Update at layer ℓ: ∂L(ℓ) ∂w (ℓ): • Total Number of Multiplications MulCount = N C(ℓ)H (ℓ)W (ℓ) | {z } Algo 6,Line 13 + N C(ℓ)H (ℓ)W (ℓ) C (ℓ−1)K 2 | {z } Algo 6,Real multiplications at Line 17 + X ℓ=2 2N C(ℓ)C (ℓ−1)K 2 | {z } Algo 6 ,Conv binary multiplications at Line ...

  53. [2013]

    Available: http://arxiv.org/abs/1308.3432

    [Online]. Available: http://arxiv.org/abs/1308.3432

    arXiv

  54. [2018]

    Available: http://arxiv.org/abs/1810.04805

    [Online]. Available: http://arxiv.org/abs/1810.04805

    arXiv

  55. [2020]

    Available: http://arxiv.org/abs/2006.06880

    [Online]. Available: http://arxiv.org/abs/2006.06880

    2006 arXiv

  56. [2022]

    Available: http://arxiv.org/abs/2201.11665

    [Online]. Available: http://arxiv.org/abs/2201.11665

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.