Pith. sign in

REVIEW 4 major objections 4 minor 42 references

Breaking the Conventional Forward-Backward Tie in Neural Networks: Activation Functions

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that during backpropagation the activation function's derivative affects only gradient magnitude, not direction, so forward and backward passes can be decoupled without hurting learning.

desk verdict A well-run empirical study of surrogate gradients that overreaches: the theoretical propositions are mathematically wrong, and the paper's own experiments contradict its universal claims. read the letter →

arxiv 2509.07236 v1 pith:FIFYMSRR submitted 2025-09-08 cs.NE cs.AIcs.LG

Luigi Troiano , Francesco Gissi , Vincenzo Benedetto , Genny Tortora This is my paper
classification cs.NEcs.AIcs.LG MSC 68T0768T05
keywords backpropagationactivationfunctionsgradientdirectionforward-backwarddecouplingbinaryneuralnetworksHeavisidestepfunctionstochasticmodulationnetworktraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper challenges the assumption that backpropagation must use the exact derivative of the activation function. It claims the activation derivative only scales gradient magnitude, while the sign or direction of each weight update is fixed by the linear connections, so replacing the derivative with any strictly positive function — even constants, rectangles, or random noise — should not stop learning. The authors prove this decomposition on a single-unit classifier and extend it to layered networks, then support it with MLP, LeNet5, and binary-neural-network experiments. If the claim holds, activation functions with flat or non-differentiable regions such as the Heaviside step become trainable, and derivative computation in the backward pass can be simplified or dropped.

What carries the argument

The load-bearing identity is the Jacobian decomposition in backpropagation: ∇_{Z_{l-1}} G_l = W_l^T ⊙ φ'_l(W_l Z_{l-1}), with the untied variant D_{l-1}^{untied} = (J_l)^T D_l ⊙ g(Z_{l-1}) for strictly positive g. This separates the linear operator that carries directional information from the activation derivative that only scales each component, and it is what lets the paper claim direction dominance and then replace the derivative with arbitrary positive proxies.

What would settle it

Train a multi-layer network (for instance a small Transformer) with Heaviside forward activations and, in every layer, a backward replacement that is strictly positive but random in magnitude, then compare test performance against standard backprop under the same schedule. If the decoupled run fails to approach the backprop baseline on a nontrivial dataset, the claim that only gradient direction matters is contradicted. A cheaper check: on a trained two-layer MLP, measure the angle between the standard gradient update and the untied-gradient update over many steps; if the angle often exceeds 9

Watch

Extended reading notes

Core claim

The paper argues that the conventional tie between forward activations and backward gradients in neural network training is unnecessary. Its central proposition is that the direction of weight updates is independent of the activation function's derivative: the derivative enters the backpropagation equations only as an element-wise scalar multiplier, so it changes the magnitude of each update but not which direction each component points. That direction is set by the linear weight matrices and the activations from the preceding layer. From this, the paper derives that any strictly positive function of the pre-activations can replace the activation derivative in the backward pass without chang

Load-bearing premise

The proof assumes that replacing the activation derivative with any strictly positive function that preserves each component's sign is enough for stochastic gradient descent to keep learning, without proving that the loss actually descends or that update magnitudes cannot matter.

Editorial extensions

If this is right

  • Activation functions with large flat or nondifferentiable regions, including the Heaviside step, become trainable by choosing an independent backward gradient proxy.
  • Simpler backward computations (constants, rectangles) can reduce computational and memory overhead while matching or beating tied gradients, as observed in the LeNet5 experiments.
  • Gradient jamming experiments show that randomizing gradient magnitude preserves learning for ReLU and linear activations, supporting the primacy of direction over magnitude.
  • Common optimization techniques — momentum, clipping, scaling, noise, normalization — can be re-read as magnitude modulations that leave the gradient direction intact.
  • Binary neural network training, previously justified by heuristics, receives a theoretical rationale: non-differentiable binary activations can be trained with straightforward positive backward proxies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If direction dominance generalizes to deep Transformers and graph networks as Proposition 2 asserts, one could design backward passes that store only signs or random projections instead of full activation derivatives, potentially cutting training memory; this is testable but untested in the paper.
  • The paper's own data show logistic activations collapse under full random jamming while ReLU and linear do not, suggesting that bounded saturating forward functions rely on magnitude information more than the theory's sign-only argument would predict.
  • The propositions give a formal footing for straight-through estimators; a natural next experiment is comparing constant positive proxies with stochastic ones whose mean equals the true derivative, to see whether distribution shape (not just positivity) matters for convergence speed.
  • A cheap falsifying check follows from Eq. (24): if per-neuron positive scaling g(Z) can rotate the aggregate update more than 90 degrees from the true gradient on realistic batches, the claim 'sign preservation suffices' would not hold for a full update vector.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript argues that the direction of weight updates in neural networks is independent of activation-function derivatives and is predominantly determined by linear connections. It formalizes this as three propositions, then supports it with experiments on a Single Unit Classifier, an MLP, and LeNet-5 in which the backward gradient is replaced by constant, rectangular, triangular, or stochastic functions, and with BNN experiments using Heaviside activations. The paper concludes that the conventional forward-backward symmetry can be relaxed without impairing learning.

Significance. If valid, the claimed result would provide a formal justification for straight-through estimators and would license training non-differentiable activation functions, which is practically valuable. The paper has strengths: a clear research question, a reproducible experimental protocol with 20 runs per setting, several datasets and architectures, and honest limitations regarding more complex architectures. However, the theoretical core is not sound: the proof of Proposition 1 mischaracterizes element-wise multiplication as scalar scaling, and Proposition 3 does not establish a descent or convergence property. The experiments are not sufficient to rescue the general claim, since several configurations show clear performance collapses. I therefore cannot recommend publication as a theoretical contribution; an empirical-only claim would require substantial reframing.

major comments (4)
  1. [Section 3, Eq. (22), Proposition 1] Equation (22) writes ∇_{Z_{l-1}} G_l = W_l^T ⊙ φ'_l(W_l Z_{l-1}) and then treats φ'_l as an 'element-wise scalar multiplier'. If φ'_l has distinct values across neurons, each row/column of W_l^T is scaled by a different factor, so the vector direction is not preserved. For example, with W_l^T = [[1,0],[0,1]] and D_l = (1,1), φ' = (1,0) gives a different direction than φ' = (0,1). Thus Proposition 1 is invalid as stated, and the BNN justification in Section 5 relies on this statement.
  2. [Section 3, Proposition 3, Eq. (24)] Strict positivity of g only preserves the sign of each component of (J_l)^T D_l at one layer. It does not preserve the sign of the full gradient with respect to earlier weights after composing layers, and it does not imply that the update is a descent direction. Concrete two-layer linear counterexample: W2^T = [[1,-1],[0.5,1]], x = (1,1), dL/da2 = (1,1), φ2' = (1,0.001), surrogate g2 = (0.001,1), φ1' = g1 = (1,1). Then δ1 = (0.999,0.501), δ̃1 = (-0.999,1.0005), and δ1·δ̃1 ≈ -0.497 < 0. The surrogate update for W1 moves opposite to the true gradient. Hence Eq. (24) does not establish the feasibility claim.
  3. [Section 4.8, Full-Jamming] Full-Jamming replaces the backward gradient by a uniform random value in [0,1]. Since all sampled values are nonnegative, negative feedback gradients have their signs flipped. The text claims that this 'ensures that only the sign (directionality) of the gradient is preserved', which is contradicted by the definition and Figure 8(a). This mischaracterization weakens the interpretation that direction, not magnitude, drives learning.
  4. [Section 4, Tables 3-5] The empirical support is mixed. In Table 3, the untied Log/1 configuration is consistently 2-5 points below tied across most settings. In Table 4, Log/1 has standard deviations as high as 0.289 and near-chance means at low learning rates. In Table 5, ReLU/1 collapses to about 0.11 accuracy for LR=0.05 and 0.1 at all batch sizes. These results do not support the unqualified conclusion that decoupling 'does not impair learning'; they indicate that some surrogate choices severely hurt training. The claim should be restricted to specific surrogates and regimes.
minor comments (4)
  1. [Section 4.3 / Figure 6] The text says '(a) constant, (b) rectangular, (c) triangular (d)', but Figure 6 has only (a)-(c) and the caption labels (b) Rectangular and (c) Triangular. Please align the numbering.
  2. [Section 2, Eq. (3)] Typo: 'proximated' should be 'approximated'.
  3. [Figure 8 caption] The caption refers to 'Experiment 6', but the section is Experiment 5 (Section 4.8).
  4. [Section 3, Eq. (22)] The notation W_l^T ⊙ φ'_l(...) is dimensionally ambiguous; use diag(φ'_l(...))W_l or state the Hadamard convention explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical propositions are independent of the experiments, and no fitted parameter or self-citation is presented as a prediction.

full rationale

The paper's derivation chain is not circular. The mathematical claims (Propositions 1–3) are derived from the backpropagation update equations (10)–(11) and geometric arguments, not from fitted constants, benchmark tuning, or the experimental results. The experiments use arbitrary untied gradient surrogates (constant, rectangular, triangular, random) and report accuracy; they do not fit a surrogate to training data and then present the resulting accuracy as a prediction. There are no self-citations in the reference list and no load-bearing reliance on the authors' prior work. The discussion of Straight-Through Estimator, BinaryConnect, and XNOR-Net is explicitly comparative and does not import an unverified uniqueness theorem or ansatz. The main weakness is mathematical invalidity rather than circularity: Proposition 3 equates 'direction' with per-component sign and assumes that positive element-wise scaling cannot impair learning, which is not sufficient in deep networks—different positive per-neuron scales change the vector direction across layers, and composed surrogates can produce updates opposing the true gradient. That is a correctness gap, not a reduction of the conclusion to the input by construction. Under the hard rules, circularity requires a specific exhibited reduction (e.g., fitted parameter renamed as prediction, or a conclusion identical to a definition). No such reduction is present here, so the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central theory introduces no fitted constants, but it leans on two unproved assumptions: element-wise positive scaling is equated with preserving gradient direction, and sign preservation is equated with trainability. The experimental section adds hand-chosen surrogate functions and a noise normalization constant fitted to observed gradient magnitudes. No new physical or model entities are postulated.

free parameters (4)
  • Constant surrogate gradient value = 1
    Hand-chosen g(Z)=1 for Log/1, ReLU/1, and Step/1 experiments; no criterion or sensitivity analysis given, and performance varies strongly across settings.
  • Rectangular window support = not fully specified; Figure 8c uses |x| <= 5
    Window width is a hand-chosen design parameter for Rect and Rectangular-Jamming; accuracy changes substantially with window shape, so the choice matters.
  • Noise normalization constant = not reported numerically
    Section 4.8 states noise functions are normalized so their mean matches the average gradient magnitude from traditional training; this constant is fitted from data and omitted from the paper.
  • Triangular surrogate shape = not specified as a formula
    Figure 6(c) shows a triangular function without an equation or parameter values; used in ReLU/Triang and Step/Triang experiments.
assumptions (5)
  • standard math Backpropagation equations (10)-(11) give the exact gradient-based training dynamics.
    Foundational assumption for all propositions; standard for MLPs but extends only informally to other architectures.
  • domain assumption For every architecture the layer Jacobian factorizes as J_l^T ⊙ phi'_l(Z_{l-1}) (Eq. 23).
    Proposition 2 asserts this for CNNs, RNNs, transformers, and GNNs with a footnote-level justification; for attention and graph aggregation the factorization is not demonstrated.
  • ad hoc to paper Element-wise multiplication by a strictly positive vector preserves gradient direction.
    Used in Proposition 3 (Eq. 24); false when the vector has unequal components, since per-coordinate scaling rotates the vector.
  • ad hoc to paper Learning depends on direction and sign, not magnitude, of weight updates.
    Central premise of the conclusions; no convergence theorem, descent condition, or loss-geometry assumption supports it.
  • domain assumption Test accuracy on balanced benchmark splits is enough to validate training effectiveness.
    Section 4.3; reasonable for the experiments but limits the general claim to accuracy-focused small-scale classification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking the Conventional Forward-Backward Tie in Neural Networks: Activation Functions." pith.science (2026). https://pith.science/paper/FIFYMSRR

@misc{pith2026250907236,
  author       = {Pith},
  title        = {Pith review of: Breaking the Conventional Forward-Backward Tie in Neural Networks: Activation Functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FIFYMSRR}},
  note         = {Machine review of arXiv:2509.07236}
}
read the original abstract

Gradient-based neural network training traditionally enforces symmetry between forward and backward propagation, requiring activation functions to be differentiable (or sub-differentiable) and strictly monotonic in certain regions to prevent flat gradient areas. This symmetry, linking forward activations closely to backward gradients, significantly restricts the selection of activation functions, particularly excluding those with substantial flat or non-differentiable regions. In this paper, we challenge this assumption through mathematical analysis, demonstrating that precise gradient magnitudes derived from activation functions are largely redundant, provided the gradient direction is preserved. Empirical experiments conducted on foundational architectures - such as Multi-Layer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), and Binary Neural Networks (BNNs) - confirm that relaxing forward-backward symmetry and substituting traditional gradients with simpler or stochastic alternatives does not impair learning and may even enhance training stability and efficiency. We explicitly demonstrate that neural networks with flat or non-differentiable activation functions, such as the Heaviside step function, can be effectively trained, thereby expanding design flexibility and computational efficiency. Further empirical validation with more complex architectures remains a valuable direction for future research.

Figures

Figures reproduced from arXiv: 2509.07236 by the authors.

Figure 1
Figure 1. Block diagram illustrating the training process of a multi-layer neural network made up of blocks. The [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Illustration of weight updates’ impact on hyperplane configurations. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. The forward and backward flows characterizing the training process of the SUC are clearly illustrated in Fig. 4, [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Block diagram showing the training of SUC, where [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Training depiction of a neural network. Here, [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Gradient contributions used in the experiments in place of the activation functions gradients. Specifically, we [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Training accuracy and loss trends for tied and untied gradient configurations in the MLP architecture with [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Noise functions used for gradient modulation in Experiment 6: (a) Full-Jamming, (b) Positive-Jamming, and [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 26 canonical work pages

  1. [1]

    Widrow, M

    B. Widrow, M. A. Lehr, 30 years of adaptive neural networks: perceptron, madaline, and backpropagation, Proceedings of the IEEE 78 (9) (1990) 1415–1442

    work page 1990

  2. [2]

    Cauchy, M´ethode g´en´erale pour la r´esolution des syst`emes d’´equations simultan´ees, Comptes Rendus 25 (1847) 536–538

    A.-L. Cauchy, M´ethode g´en´erale pour la r´esolution des syst`emes d’´equations simultan´ees, Comptes Rendus 25 (1847) 536–538

    work page

  3. [3]

    Robbins, S

    H. Robbins, S. Monro, A stochastic approximation method, The Annals of Mathematical Statistics (1951) 400–407

    work page 1951

  4. [4]

    Widrow, M

    B. Widrow, M. E. Hoff, Adaptive ”adaline” neuron using chemical ”memistors”, Stanford Electron. Labs. Tech. Rep. 1553 (2) (1960) 1–26. 28 Breaking the Conventional Forward-Backward Tie in Neural Networks: Activation FunctionsA PREPRINT

    work page 1960

  5. [5]

    D. E. Rumelhart, G. E. Hinton, R. J. Williams, Learning representations by back-propagating errors, Nature 323 (6088) (1986) 533–536

    work page 1986

  6. [6]

    Pascanu, T

    R. Pascanu, T. Mikolov, Y . Bengio, On the difficulty of training recurrent neural networks, in: Proceedings of the 30th International Conference on International Conference on Machine Learning - V olume 28, ICML’13, JMLR.org, 2013, p. III–1310–III–1318

    work page 2013

  7. [7]

    Bottou, Stochastic gradient learning in neural networks, in: Proceedings of Neuro-N ˆımes 91, EC2, Nimes, France, 1991

    L. Bottou, Stochastic gradient learning in neural networks, in: Proceedings of Neuro-N ˆımes 91, EC2, Nimes, France, 1991. URLhttp://leon.bottou.org/papers/bottou-91c

    work page 1991

  8. [8]

    Qian, On the momentum term in gradient descent learning algorithms, Neural Netw

    N. Qian, On the momentum term in gradient descent learning algorithms, Neural Netw. 12 (1) (1999) 145–151. doi:10.1016/S0893-6080(98)00116-6

    doi:10.1016/s0893-6080(98)00116-6 1999

Show all 42 references
  1. [9]

    A. Veit, M. Wilber, S. Belongie, Residual networks behave like ensembles of relatively shallow networks, in: Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, Curran Associates Inc., Red Hook, NY , USA, 2016, p. 550–558

    2016

  2. [10]

    Mahdavimanshadi, M

    M. Mahdavimanshadi, M. G. Anaraki, M. Mowlai, Z. Ahmadirad, A multistage stochastic optimization model for resilient pharmaceutical supply chain in covid-19 pandemic based on patient group priority, in: 2024 Systems and Information Engineering Design Symposium (SIEDS), 2024, p...

    2024

  3. [11]

    Santos, T

    S. Santos, T. Breaux, T. Norton, S. Haghighi, S. Ghanavati, Requirements Satisfiability with In-Context Learning , in: 2024 IEEE 32nd International Requirements Engineering Conference (RE), IEEE Computer Society, Los Alamitos, CA, USA, 2024, pp. 168–179.doi:10.1109/RE59067.2024.00025

    2024

  4. [12]

    F. S. Banitaba, S. Aygun, M. S. Moghadam, A. Jalilvand, B. Li, M. H. Najafi, Adversarial attack bypass by stochastic computing, IEEE Embedded Systems Letters (2025) 1–1doi:10.1109/LES.2025.3538552

    2025

  5. [13]

    J. Martens, Deep learning via hessian-free optimization, in: Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, Omnipress, Madison, WI, USA, 2010, p. 735–742

    2010

  6. [14]

    Martens, R

    J. Martens, R. Grosse, Optimizing neural networks with kronecker-factored approximate curvature, ICML’15, JMLR.org, 2015, p. 2408–2417

    2015

  7. [15]

    Ebadi, A

    M. Ebadi, A. Hosseini, M. Hosseini, A projection type steepest descent neural network for solving a class of nonsmooth optimization problems, Neurocomput. 235 (C) (2017) 164–181. doi:10.1016/j.neucom.2017.01. 010

    2017 doi

  8. [16]

    D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.arXiv:1412.6980

    2015 arXiv

  9. [17]

    Duchi, E

    J. Duchi, E. Hazan, Y . Singer, Adaptive subgradient methods for online learning and stochastic optimization, J. Mach. Learn. Res. 12 (2011) 2121–2159

    2011

  10. [18]

    Tieleman, G

    T. Tieleman, G. Hinton, Lecture 6.5 - rmsprop: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural Networks for Machine Learning (2012). URLhttps://www.cs.toronto.edu/ ~tijmen/csc321/slides/lecture_slides_lec6.pdf

    2012

  11. [19]

    Nesterov, A method for solving the convex programming problem with convergence rateo(1/k2), Doklady Akademii Nauk SSSR 269 (3) (1983) 543–547

    Y . Nesterov, A method for solving the convex programming problem with convergence rateo(1/k2), Doklady Akademii Nauk SSSR 269 (3) (1983) 543–547

    1983

  12. [20]

    M. D. Zeiler, Adadelta: An adaptive learning rate method, arXiv preprint arXiv:1212.5701 (2012). arXiv: 1212.5701

    2012 arXiv

  13. [21]

    X. Wang, Z. Tang, H. Tamura, M. Ishii, W. Sun, An improved backpropagation algorithm to avoid the local minima problem, Neurocomputing 56 (2004) 455–460.doi:10.1016/j.neucom.2003.08.006

    2004 doi

  14. [22]

    M. Gori, A. Tesi, On the problem of local minima in backpropagation, IEEE Transactions on Pattern Analysis and Machine Intelligence 14 (1) (1992) 76–86.doi:10.1109/34.107014

    1992 doi

  15. [23]

    H. Yong, J. Huang, X. Hua, L. Zhang, Gradient centralization: A new optimization technique for deep neural networks, arXiv preprint arXiv:2004.01461 (2020).arXiv:2004.01461

    2004 arXiv

  16. [24]

    Salimans, D

    T. Salimans, D. P. Kingma, Weight normalization: a simple reparameterization to accelerate training of deep neural networks, NIPS’16, Curran Associates Inc., Red Hook, NY , USA, 2016, p. 901–909

    2016

  17. [25]

    Ioffe, C

    S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015, pp. 448–456. arXiv:1502.03167. 29 Breaking the Conventional Forward-Backwa...

    2015 arXiv

  18. [26]

    Santurkar, D

    S. Santurkar, D. Tsipras, A. Ilyas, A. Madry, How does batch normalization help optimization?, in: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, Curran Associates Inc., Red Hook, NY , USA, 2018, p. 2488–2498

    2018

  19. [27]

    Ulyanov, A

    D. Ulyanov, A. Vedaldi, V . Lempitsky, Improved texture networks: Maximizing quality and diversity in feed- forward stylization and texture synthesis, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 4105–4113.doi:10.1109/CVPR.2017.437

    2017 doi

  20. [28]

    Huang, S

    X. Huang, S. Belongie, Arbitrary style transfer in real-time with adaptive instance normalization (2017). arXiv: 1703.06868

    2017 arXiv

  21. [29]

    J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization (2016).arXiv:1607.06450

    2016 arXiv

  22. [30]

    Y . Wu, K. He, Group normalization (2018).arXiv:1803.08494

    2018 arXiv

  23. [31]

    S. Qiao, H. Wang, C. Liu, W. Shen, A. Yuille, Micro-batch training with batch-channel normalization and weight standardization (2020).arXiv:1903.10520

    2020 arXiv

  24. [32]

    Huang, X

    L. Huang, X. Liu, Y . Liu, B. Lang, D. Tao, Centered weight normalization in accelerating training of deep neural networks, in: 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2822–2830. doi:10.1109/ICCV.2017.305

    2017 doi

  25. [33]

    Sutskever, J

    I. Sutskever, J. Martens, G. Dahl, G. Hinton, On the importance of initialization and momentum in deep learning, ICML’13, JMLR.org, 2013, p. III–1139–III–1147

    2013

  26. [34]

    Pascanu, T

    R. Pascanu, T. Mikolov, Y . Bengio, On the difficulty of training recurrent neural networks, in: International conference on machine learning, 2013, pp. 1310–1318

    2013

  27. [35]

    NVIDIA, Mixed precision training, Online: https://docs.nvidia.com/deeplearning/sdk/ mixed-precision-training/index.html(2018)

    2018

  28. [36]

    Neelakantan, L

    A. Neelakantan, L. Vilnis, Q. V . Le, I. Sutskever, L. Kaiser, K. Kurach, J. Martens, Adding gradient noise improves learning for very deep networks (2015).arXiv:1511.06807

    2015 arXiv

  29. [37]

    Alpaydin, C

    E. Alpaydin, C. Kaynak, Optical Recognition of Handwritten Digits, UCI Machine Learning Repository (1998). doi:10.24432/C50P49. URLhttps://doi.org/10.24432/C50P49

    1998 doi

  30. [38]

    LeCun, C

    Y . LeCun, C. Cortes, C. J. Burges, The mnist database of handwritten digits (1998). URLhttp://yann.lecun.com/exdb/mnist/

    1998

  31. [39]

    H. Xiao, K. Rasul, R. V ollgraf, Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms (2017).arXiv:1708.07747

    2017 arXiv

  32. [40]

    Bengio, N

    Y . Bengio, N. L´eonard, A. Courville, Estimating or propagating gradients through stochastic neurons for condi- tional computation (2013).arXiv:1308.3432

    2013 arXiv

  33. [41]

    Courbariaux, Y

    M. Courbariaux, Y . Bengio, J.-P. David, Binaryconnect: Training deep neural networks with binary weights during propagations, in: Advances in neural information processing systems, 2015

    2015

  34. [42]

    Rastegari, V

    M. Rastegari, V . Ordonez, J. Redmon, A. Farhadi, Xnor-net: Imagenet classification using binary convolutional neural networks, in: European Conference on Computer Vision, 2016. 30

    2016

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.