Can Local Learning Match Self-Supervised Backpropagation?

Ariane Delrocq; Guillaume Bellec; Wulfram Gerstner; Wu S. Zihan

REVIEW 4 major objections 3 minor 19 references

Can Local Learning Match Self-Supervised Backpropagation?

T0 review · 4 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In deep linear networks with orthonormal weights and optimally learned per-layer projections, local self-supervised learning produces exactly the same weight updates as global backpropagation; the theory then yields local convnet rules that

desk verdict Real theory, good empirical gains, but the exact-equivalence claim is pointwise under idealizations; still worth refereeing with requested revisions. read the letter →

arxiv 2601.21683 v2 pith:6KEVTSVT submitted 2026-01-29 cs.LG

Wu S. Zihan , Ariane Delrocq , Wulfram Gerstner , Guillaume Bellec This is my paper · ORCID

classification cs.LG

keywords locallearningself-supervisedbackpropagationdeeplinearnetworkscontrastiveCLAPPdirectfeedbackspatialdependence

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that local self-supervised learning rules are not inherently weaker than global backpropagation-based self-supervised learning (BP-SSL). It proves that in deep linear networks with orthonormal weight matrices, a local layer-wise contrastive loss with an optimally learned per-layer projection B^l produces exactly the same gradient for every feedforward weight as global BP-SSL would. Starting from that theorem, the paper modifies CLAPP-style local rules in two theory-guided ways—letting B^l depend on 2D spatial location in convolutional networks, and routing the reference signal from the top layer instead of each local layer—and shows that the resulting variants match the downstream classification accuracy of a comparable global BP-SSL baseline on CIFAR-10, STL-10, and Tiny ImageNet. A sympathetic reader would care because the result reframes the local/global gap as an optimization and architecture problem rather than an algorithmic impossibility, and it gives biological learning rules a route to global objectives without a backward pass.

What carries the argument

The load-bearing object is the per-layer projection matrix B^l (or the identity, in Forward-forward-style rules) that maps a reference vector c^l onto the layer's activity space inside the local loss. The theorem's key algebraic fact is B^l_* = (W^L ... W^{l+1})^T B^L_* (W^L ... W^{l+1}), an identity that orthonormal weight matrices make available and that converts the local per-layer gradient into the exact backpropagated gradient. Two structural variants carry the practical argument: direct feedback, which replaces the reference c^l with the top-layer activity z^L and improves gradient similarity when layers shrink, and 2D spatial dependence, which lets B^l vary by spatial location in conv

What would settle it

Train a deep linear network with orthonormal weights and a CLAPP-style loss, updating B^l by gradient descent on the same timescale as W^l, and measure the cosine similarity between the local gradient and the global BP-SSL gradient across layers; if similarity drops below 1 in early layers, the instantaneous optimality of B^l—not orthonormality alone—is doing the load-bearing work in Theorem 3.1.

Watch

Extended reading notes

Core claim

The central claim is Theorem 3.1: in a deep linear network with orthonormal weight matrices and a layer-wise contrastive loss of the form f(z_pos^T B^l c_pos) + f(-z_neg^T B^l c_neg), the gradient of the locally minimized loss with respect to W^l equals the gradient that global backpropagation computes for the last-layer loss, whenever B^l is the identity or the unconstrained minimizer of the local loss. The proof works by identifying the optimal per-layer projection as B^l_* = (W^L ... W^{l+1})^T B^L_* (W^L ... W^{l+1}), which orthonormality turns into an exact match between local and backpropagated gradient directions. The authors then prove a companion result for shrinking-width linear ne

Load-bearing premise

The load-bearing premise is that the per-layer projection B^l is always at its exact optimal value when the feedforward weights are updated, because B^l is assumed to evolve on a much faster timescale; the actually trained convnets update B^l by gradient steps, so the exact equality in Theorem 3.1 is not guaranteed to hold there.

Editorial extensions

If this is right

In the idealized regime of Theorem 3.1, a local-SSL update is not merely a heuristic approximation of BP-SSL; it is the same gradient, so any local/global gap must come from optimization or architecture rather than from locality itself.
When the number of neurons shrinks across layers, feeding the top-layer activity back as the reference signal is provably closer to the global BP-SSL gradient (in the linear case) than using each layer's own activity, and this transfer extends numerically to ReLU networks.
In convolutional networks, giving the per-layer projection B^l a 2D spatial structure increases the cosine similarity of local and BP-SSL gradients and improves downstream accuracy; direct feedback adds a further small improvement.
The CLAPP++ variants match the accuracy of a same-architecture global BP-SSL baseline on CIFAR-10, STL-10, and Tiny ImageNet, and improve on the previous state of the art for local-SSL on those benchmarks.
CLAPP++DFB stores a single top-layer context vector instead of a context vector per layer, which lowers the memory footprint of training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Restating the paper's own proof as a design principle: a local rule whose per-layer projection is trained to mimic the backpropagated gradient should inherit BP-SSL's behavior. This predicts that richer, data-dependent families of B^l—nonlinear projections, or projections trained to directly minimize gradient mismatch—would shrink the residual gap in deep nonlinear networks further.
The patch sizes that control spatial sharing of B^l are set by hand per dataset and layer, not derived from the theory. A testable extension would learn or infer those sizes from the data; if performance survives, the mechanism is spatial structure itself, not the extra tuned parameters.
If the same spatial-dependence and direct-feedback ideas transfer to non-contrastive local losses (variance/decorrelation losses) and to larger modern architectures, the practical upshot would be that self-supervised representation learning can run on neuromorphic or energy-constrained hardware without a global backward pass.
The biological analogy the authors draw—a top-layer signal gating plasticity like apical dendritic input—could be turned into a concrete prediction: a cortex-like circuit with only a delayed global novelty/prediction signal should learn hierarchical representations as well as one with layer-specific error propagation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The paper is worth reading. It proves a clean equivalence result and backs it with solid benchmark gains. But the headline claim runs a bit ahead of the evidence: the exact gradient equality is a pointwise identity under idealizations that do not survive the training trajectories used in the experiments.

What is actually new: Theorem 3.1 shows that in a deep linear network with orthonormal equal-width weights, if each layer's projection B^l is at its optimal value, then the local-SSL gradient equals the global BP-SSL gradient. The proof is elementary linear algebra, but it is correct, and the unifying notation for CLAPP/FF/SCFF is handy. The direct-feedback (DFB) variant and the 2D spatial dependence for convnets are genuinely new algorithmic ideas, and they are motivated by the theory. The empirical results are real: CLAPP++ and its DFB/both variants outperform previous local-SSL methods on CIFAR-10, STL-10, and Tiny ImageNet, and come close to (or beat) the BP baselines on CIFAR and Tiny. That is a meaningful step for the subfield.

The soft spots are proportionate. The main one: Theorem 3.1 is a fixed-point statement. It assumes B^l is at its optimum and W^l is orthonormal at the moment of the update. Gradient descent destroys both: after a step, W^l is no longer orthonormal, and in the convnet experiments B^l is trained by Adam, not held at an optimum. Figure 2 verifies gradients at a fixed initialization without updating W, so it confirms a pointwise identity, not an equivalence of learning algorithms. The abstract's "implements exactly the same weight update" is therefore too strong if read along a learning trajectory. The paper does disclaim that the theorem doesn't carry to nonlinear networks, but the trajectory issue is left unqualified in the abstract.

Second, the "matches ... InfoNCE" claim is not supported on STL-10: CLAPP++both gets 79.62±0.28 vs BP-InfoNCE 81.97±1.23; the intervals don't overlap. It does match on CIFAR-10 and Tiny ImageNet.

Third, practical reproducibility: no code is shipped, and the spatial patch sizes in Table A2 are hand-picked per dataset/layer with no sensitivity analysis. Minor, but a referee should ask for it.

Overall: the theory is sound under its stated assumptions, the empirical contribution is real, and the authors are honest about limitations. The overclaims are fixable with wording and a trajectory-level caveat. This should go to peer review rather than being desk-rejected; I would expect a solid, not transformative, paper after revision.

Referee Report

4 major / 3 minor

Summary. The paper asks whether local, backpropagation-free self-supervised learning can match end-to-end backprop-based self-supervised learning. It first proves that for deep linear networks with orthonormal weight matrices and an optimally trained projection matrix B^l, the per-layer local SSL gradient equals the global BP-SSL gradient (Theorem 3.1). A second result (Theorem 3.3) shows that when layer widths shrink, using the top-layer representation as the context vector makes local updates closer to BP in a deep linear network with linear f. Guided by these observations, the paper introduces CLAPP++, CLAPP++DFB, and CLAPP++both, which add 2D spatial dependence and top-down feedback to the CLAPP objective, and reports results on CIFAR-10, STL-10, and Tiny ImageNet, including a new state of the art among local-SSL rules.

Significance. Theorems 3.1 and 3.3 are valuable because they give precise, if narrow, conditions under which local Hebbian-like rules reproduce BP-SSL updates exactly or approximately. The proof of Theorem 3.1 is clean and the paper is transparent about its assumptions (equal-width orthonormal layers, instantaneous optimality of B^l, normalization-free losses). The empirical study is broad, with two ablations (spatial dependence, direct feedback), multiple baselines, and detailed training details that would support replication. The main weakness is that the link between the idealized theory and the convolutional-network experiments is heuristic: the 2D spatial sharing pattern is chosen by hand (Table A2) and is not derived from the theorem. The paper would be strengthened by releasing code and by a more guarded wording of the 'matches BP-SSL' claim, which fails on STL-10.

major comments (4)

[Abstract and Table 2] The claim that the best local-SSL rule 'matches the performance of a comparable global BP-SSL with InfoNCE or CPC-like loss functions' is not supported on STL-10. In Table 2, CLAPP++both achieves 79.62±0.28 while BP-InfoNCE achieves 81.97±1.23, a 2.35-point gap; BP-CLAPP++ achieves 80.36±0.26, whose 95% interval does not overlap with that of CLAPP++both. The statement is defensible for CIFAR-10 and Tiny ImageNet but not for STL-10. Please revise the abstract and conclusions to report the per-dataset comparison and to use 'approaches' or 'is comparable on some datasets' instead of 'matches'.
[§3.1, Theorem 3.1 and Eq. (2)] Eq. (2) is proven as a pointwise identity at an orthonormal weight configuration where B^l is assumed to be at its instantaneous optimum. It is not a statement about the learning trajectory: the local update ΔW^l is a generic full-rank gradient step, so applying it to an orthonormal W^l leaves W^l non-orthonormal, and the theorem's premise fails at the next update. Figure 2 computes similarities at a fixed initialization and never trains W^l, so it does not demonstrate trajectory equivalence. If the abstract or contribution list is intended to claim that local-SSL 'implements exactly the same weight update' as global BP-SSL during training, the authors need to prove that orthonormality (and B-optimality) is preserved or restored along the dynamics, or explicitly state that the equality is instantaneous. This caveat should also appear in the abstract.
[§3.3, §4 and Table A2] The convolutional variants are described as 'theory-guided', but the theory in §3.1–3.2 does not derive the 2D spatial sharing pattern of B^l; the patch sizes n×n in Table A2 are hand-picked per dataset and layer. The only support for spatial dependence is an intuition about BP error propagation and a simulation on a 4-layer linear convnet with kernel/stride 2 (Fig. 4B). This is a heuristic, not a theorem. The paper should explicitly label the patch sizes as a hyperparameter choice and provide a sensitivity analysis over n, since the main empirical improvement comes from this choice. Otherwise the statement 'theory indicates that feedback weights should have a structured 2D spatial dependence' (contribution bullet) goes beyond what is proven.
[§3.1, Theorem 3.1 statement] The theorem is stated with 'B^l_* = I (e.g., forward-forward)', but the paragraph immediately after the theorem says the result has not been extended to networks with layer-activity normalization, which Forward-Forward requires according to Table 1. Because normalization is a nonlinear operation, a Forward-Forward network is not a deep linear network in the sense of Theorem 3.1. The parenthetical example is therefore misleading and should be removed or changed to a normalization-free case; the theorem's proof only covers the linear, normalization-free setting.

minor comments (3)

[Appendix A.2] The displayed equation for L* in the proof of Theorem 3.3 appears to have a typo: it reads 'min_{D1} (∆z^l)^T V∅ D1 c^L', but the preceding derivation of L^{l,*,B1}=L* uses V_l D1 V_l^T c^l. If the printed equation is literal, it is not equal to L*. The intended proof is recoverable, but the typo should be fixed.
[§3.1, Eq. (1)] The theorem assumes f is convex with f' ≤ 0, but the CLAPP loss uses f(x)=max(0,1−x), which is non-differentiable at the kink. The proof uses f'. Please state whether the result holds for subgradients or restrict the theorem to differentiable convex f (the simulations use softplus, which is smooth).
[Fig. 4C / Appendix B] The figure caption and the 'Simulation for Figure 4 C' paragraph say that spatial dependence is removed in the last layer for CLAPP++ and CLAPP++DFB, but Table A2 lists patch sizes for all six layers (including L6). Please clarify how the last-layer sharing is set in this control and why it does not change classification accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Theorem 3.1 is a conditional identity proven from explicit assumptions; the algorithmic variants are empirical designs benchmarked against external methods, not fitted parameters repackaged as predictions.

full rationale

The paper's central theoretical claim (Theorem 3.1) is a conditional identity: under stated assumptions (deep linear network, orthonormal W^l, B^l at its unique convex optimum or B=I, no normalization), it proves that the local-SSL gradient equals the global BP-SSL gradient. The proof explicitly derives the mapping B^l_* = (W^L...W^{l+1})^T B^L_* (W^L...W^{l+1}) and uses orthonormality plus strict convexity to show this is the minimizer of the local loss. This does not assume the conclusion; it is a derivation from the assumptions. The numerical verification in Figure 2 checks the identity at a fixed orthonormal initialization, as stated, and the authors explicitly note the theorem does not carry over to nonlinear networks. The convnet variants are not claimed to satisfy the theorem; spatial dependence and direct feedback are design heuristics, tested empirically against external SOTA (SCFF, LPL) and against the paper's own BP baselines. Definition 3.4 is explicitly labeled as a non-local upper bound requiring BP, so it is a control, not a local prediction. Self-citations to CLAPP (Illing et al. 2021; Delrocq et al. 2024) supply the algorithm family being analyzed, but the proof and benchmarks do not reduce to those citations. Patch sizes in Table A2 are manually chosen hyperparameters, not outputs of the theory, so no fitted parameter is renamed as a prediction. No circular step can be identified.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central theorem rests on idealizations (orthonormal weights, instant optimality of B^l, no normalization, constant context) that are openly stated. The practical CLAPP++ variants depend on additional hand-chosen architectural details (patch sizes) and a heuristic spatial-dependence assumption. No new physical or ontological entities are introduced; the new components are algorithmic parameterizations.

free parameters (2)

L2 regularization λ on B^l = 0.01 (deep linear simulations), 0.02 (linear CNN)
Added in Eq. (1) to ensure strict convexity and a unique B^l_*; the value is chosen by hand and not theoretically pinned, though the equivalence proof should be insensitive to it.
Spatial patch sizes n×n sharing B^l per layer/dataset = CIFAR10: 4,4,4,2,2,1; STL10: 12,12,12,6,6,3; Tiny-ImageNet: 8,8,8,4,4,2 (Table A2)
Defines the 2D spatial dependence of CLAPP++; no selection rule or sensitivity analysis is reported, and this design choice is a major component of the reported performance gains.

assumptions (5)

domain assumption Training happens with a separation of timescales: B^l is at its global optimum B^l_* when W^l is updated.
Explicitly assumed in §3.1 ('evolves on a faster timescale than W^l'); needed for equality in Theorem 3.1 and Corollary 3.2. Real training in Appendix B updates B^l by gradient descent, so this premise is violated.
domain assumption Weight matrices W^l are orthonormal (and square) for Theorem 3.1.
Statement of Theorem 3.1; requires constant width across layers. Not true for the convnets in Table 2, so the theorem does not directly apply to the benchmark results.
domain assumption For Theorem 3.3, f is linear and weights are semi-orthonormal with shrinking widths.
Appendix A.2 proves the DFB improvement only for this restricted setting; simulation extends to softplus and ReLU but without proof.
domain assumption For DFB, the context c^l = z'^L is treated as constant with respect to W^l.
Stated before Corollary 3.2 ('Since c^l is considered as constant with respect to W^l...'). If the top-down context is allowed to backprop through W^l, the update is no longer the simple local loss derivative.
ad hoc to paper 2D spatial dependence of B^l improves BP approximation in nonlinear convnets.
Motivated by a linear-convnet simulation (§3.3 and Fig. 4B), not by a theorem for nonlinear networks; the claim is heuristic and is the basis for CLAPP++.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can Local Learning Match Self-Supervised Backpropagation?." pith.science (2026). https://pith.science/paper/6KEVTSVT

@misc{pith2026260121683,
  author       = {Pith},
  title        = {Pith review of: Can Local Learning Match Self-Supervised Backpropagation?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KEVTSVT}},
  note         = {Machine review of arXiv:2601.21683}
}

read the original abstract

While end-to-end self-supervised learning with backpropagation (global BP-SSL) has become central for training modern AI systems, theories of local self-supervised learning (local-SSL) have struggled to build functional representations in deep neural networks. To establish a link between global and local rules, we first develop a theory for deep linear networks: we identify conditions for local-SSL algorithms (like Forward-forward or CLAPP) to implement exactly the same weight update as a global BP-SSL. Starting from the theoretical insights, we then develop novel variants of local-SSL algorithms to approximate global BP-SSL in deep non-linear convolutional neural networks. Variants that improve the similarity between gradient updates of local-SSL with those of global BP-SSL also show better performance on image datasets (CIFAR-10, STL-10, and Tiny ImageNet). The best local-SSL rule with the CLAPP loss function matches the performance of a comparable global BP-SSL with InfoNCE or CPC-like loss functions, and improves upon state-of-the-art for local SSL on these benchmarks.

Figures

Figures reproduced from arXiv: 2601.21683 by the authors.

**Figure 1.** Overview of BP and local-SSL (A) To train the target synapse (blue), BP gradients need to be propagated down a one-to-one matching error network (red) that is gated (green) by the feedforward activations. (B): In Local-SSL, plasticity is modulated by predictive signals from same-layer lateral projections (red arrow) and global scalar values (green). (C) We show that local-SSL with feedback from the top layer bette… view at source ↗

**Figure 2.** Numerical verification of Theorem 3.1. Cosine similarity of the gradient update between BP-SSL training and local-SSL rules across layers (error bar indicating 95% confidence interval computed through different batches of input). Simulations are for theorem 3.1 as well as the cases when conditions are dropped. Random fixed B means that the B l are not optimized to be B l ∗. ReLU MLP means adding ReLU activation at … view at source ↗

**Figure 3.** Numerical verification that direct feedback improves backpropagation approximation. Cosine similarity of the gradient update between local-SSL and BP-SSL across layers. A: Simulation for theorem 3.3 on a 6-layer deep linear network with shrinking widths, both for linear f(x) = −x and non-linear (softplus) f(x) = log(1 + exp(−x)). B: Comparison of local-SSL and theoretical optimum of BP approximation. Simulations are… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Numerical verification that spatial dependence improves BP approximation. (A): Illustration of spatial dependence: without spatial dependence (left), the same B l is used to project c l (blue) onto neurons across the feature map, so they share the same color; with spat…

**Figure 5.** Figure 5: Comparison between CLAPP (Illing et al., 2021) and CLAPP++. Orange bars are theory guided algorithmic changes. Local-SSL SOTA As summarized in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 1 canonical work pages

[1]

Otherwise, we use the strict convexity of Ll with respect to Bl, which implies that Bl ∗ is uniquely defined

In the case Bl ∗ =I , we simply observe that products of matrices are cancelling each other because the matrices are orthonormal. Otherwise, we use the strict convexity of Ll with respect to Bl, which implies that Bl ∗ is uniquely defined. Then using the orthonormality of W l, it means that the norm of (W L . . . Wl+1)⊤BL(W L · · ·Wl+1) is the norm of BL,...

2026
[2]

Patches from the same image are used as positive samples

MNIST inputs are randomly cropped into 16x16 patches. Patches from the same image are used as positive samples. We use the same type 2 loss with softplus, with regularization λ= 0.02 . When we train the model with spatial dependence, z1, c1 features in the local 2x2 patches share the same Bl in the first layer, but no features share Bl in subsequent layer...

2021
[3]

20.629674

doi: 10.1101/2024.12. 20.629674. Ernoult, M., Grollier, J., Querlioz, D., Bengio, Y ., and Scel- lier, B. Updates of equilibrium prop match gradients of backprop through time in an rnn with static input. 9 Submission and Formatting Instructions for ICML 2026 Advances in neural information processing systems, 32,

work page doi:10.1101/2024.12 2024
[9]

Benchmarking pre- dictive coding networks – made simple

Pinchetti, L., Qi, C., Lokshyn, O., Emde, C., M’Charrak, A., Tang, M., Frieder, S., Menzat, B., Oliviers, G., Bogacz, R., Lukasiewicz, T., and Salvatori, T. Benchmarking pre- dictive coding networks – made simple. InProceedings of the 13th International Conference on Learning Rep- resentations, ICLR 2025, Singapore, 24–28 April 2025,

2025
[11]

11 Submission and Formatting Instructions for ICML 2026 Schultz, W., Dayan, P., and Montague, P

doi: 10.1101/407007. 11 Submission and Formatting Instructions for ICML 2026 Schultz, W., Dayan, P., and Montague, P. R. A neural substrate of prediction and reward.Science, 275(5306): 1593–1599,

doi:10.1101/407007 2026
[12]

Weilenmann, C., Ziogas, A

URL https://arxiv.org/abs/1807.03748. Weilenmann, C., Ziogas, A. N., Zellweger, T., Portner, K., Mladenovi´c, M., Kaniselvan, M., Moraitis, T., Luisier, M., and Emboras, A. Single neuromorphic memristor closely emulates multiple synaptic mechanisms for en- ergy efficient neural networks.Nature Communications, 15(1):6898,

arXiv
[18]

After performing a hyperparameter search and selecting the best model, we found that predictive coding fails to learn useful representations

because it does not require additional signals to control separate phases of updating activities and updating parameters. After performing a hyperparameter search and selecting the best model, we found that predictive coding fails to learn useful representations. To further confirm our findings of the inferior performance of predictive coding in Table 2, ...

2023
[19]

Detailed choices are specified in Table A4: Table A4.Hyperparameters for different losses in predictive coding

and LPL (Halvagal & Zenke, 2023). Detailed choices are specified in Table A4: Table A4.Hyperparameters for different losses in predictive coding. T is the number of updates for h and w for each input. ηh is the learning rate for updating h. These two hyperparameters are specific to predictive coding and are thereby tuned. Other hyperparameters are kept th...

2023

Show all 19 references

[1949]

The forward-forward algorithm: Some prelimi- nary investigations.arXiv preprint arXiv:2212.13345, 2 (3):5,

Hinton, G. The forward-forward algorithm: Some prelimi- nary investigations.arXiv preprint arXiv:2212.13345, 2 (3):5,

arXiv
[1966]

Putting an end to end-to-end: Gradient-isolated learning of representations

10 Submission and Formatting Instructions for ICML 2026 L¨owe, S., O’Connor, P., and Veeling, B. Putting an end to end-to-end: Gradient-isolated learning of representations. Advances in neural information processing systems, 32,

2026
[2013]

Echospike predictive plasticity: An online local learning rule for spiking neural networks.arXiv preprint arXiv:2405.13976,

Graf, L., Su, Z., and Indiveri, G. Echospike predictive plasticity: An online local learning rule for spiking neural networks.arXiv preprint arXiv:2405.13976,

arXiv
[2016]

More specifically, we use these methods as approximations of backpropagation in the End-to-End training of the CLAPP objective

and Predictive Coding (PC) (Whittington & Bogacz, 2017). More specifically, we use these methods as approximations of backpropagation in the End-to-End training of the CLAPP objective. Self-supervised Directed Feedback AlignmentIn DFA, the partial derivative of the loss with r...

2017
[2017]

optimizes the energy F by first updating the activity hl to an equilibrium and then updating the parameters θl. The feedback information is propagated down from the top layer when the activities hl are updated using the gradient descent of F: ∆hl =η h[−ϵl +ρ ′(hl)⊙(W l+1T ϵl+1...

2024
[2018]

A stable, fast, and fully automatic learning algorithm for predictive coding networks

Salvatori, T., Song, Y ., Yordanov, Y ., Millidge, B., Xu, Z., Sha, L., Emde, C., Bogacz, R., and Lukasiewicz, T. A stable, fast, and fully automatic learning algorithm for predictive coding networks. InProceedings of the 12th International Conference on Learning Representatio...

2024
[2019]

Vicreg: Variance- invariance-covariance regularization for self-supervised learning

Bardes, A., Ponce, J., and Lecun, Y . Vicreg: Variance- invariance-covariance regularization for self-supervised learning. InICLR 2022-International Conference on Learning Representations,

2022
[2021]

K., Fokam, C

Kappel, D., Nazeer, K. K., Fokam, C. T., Mayr, C., and Subramoney, A. Block-local learning with probabilistic latent representations.arXiv preprint arXiv:2305.14974,

arXiv
[2022]

Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781,

Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781,

arXiv
[2024]

Akrout, M., Wilson, C., Humphreys, P., Lillicrap, T., and Tweed, D

doi: 10.1101/2024.04.10.588837. Akrout, M., Wilson, C., Humphreys, P., Lillicrap, T., and Tweed, D. B. Deep learning without weight transport. Advances in neural information processing systems, 32,

2024 doi
[2025]

Proof A.1

12 Submission and Formatting Instructions for ICML 2026 A. Proof A.1. Proof of batched theorem 3.1 with batched loss function Consider the loss of type 2 with batched inputBand L-2 regularization: Ll = X µ∈B f −zµ,l neg ⊤ Blcµ,l +f zµ,l pos ⊤ Blcµ,l +λ||B l||2 F , We show as i...

2026

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Otherwise, we use the strict convexity of Ll with respect to Bl, which implies that Bl ∗ is uniquely defined

In the case Bl ∗ =I , we simply observe that products of matrices are cancelling each other because the matrices are orthonormal. Otherwise, we use the strict convexity of Ll with respect to Bl, which implies that Bl ∗ is uniquely defined. Then using the orthonormality of W l, it means that the norm of (W L . . . Wl+1)⊤BL(W L · · ·Wl+1) is the norm of BL,...

2026

[2] [2]

Patches from the same image are used as positive samples

MNIST inputs are randomly cropped into 16x16 patches. Patches from the same image are used as positive samples. We use the same type 2 loss with softplus, with regularization λ= 0.02 . When we train the model with spatial dependence, z1, c1 features in the local 2x2 patches share the same Bl in the first layer, but no features share Bl in subsequent layer...

2021

[3] [3]

20.629674

doi: 10.1101/2024.12. 20.629674. Ernoult, M., Grollier, J., Querlioz, D., Bengio, Y ., and Scel- lier, B. Updates of equilibrium prop match gradients of backprop through time in an rnn with static input. 9 Submission and Formatting Instructions for ICML 2026 Advances in neural information processing systems, 32,

work page doi:10.1101/2024.12 2024

[4] [9]

Benchmarking pre- dictive coding networks – made simple

Pinchetti, L., Qi, C., Lokshyn, O., Emde, C., M’Charrak, A., Tang, M., Frieder, S., Menzat, B., Oliviers, G., Bogacz, R., Lukasiewicz, T., and Salvatori, T. Benchmarking pre- dictive coding networks – made simple. InProceedings of the 13th International Conference on Learning Rep- resentations, ICLR 2025, Singapore, 24–28 April 2025,

2025

[5] [11]

11 Submission and Formatting Instructions for ICML 2026 Schultz, W., Dayan, P., and Montague, P

doi: 10.1101/407007. 11 Submission and Formatting Instructions for ICML 2026 Schultz, W., Dayan, P., and Montague, P. R. A neural substrate of prediction and reward.Science, 275(5306): 1593–1599,

doi:10.1101/407007 2026

[6] [12]

Weilenmann, C., Ziogas, A

URL https://arxiv.org/abs/1807.03748. Weilenmann, C., Ziogas, A. N., Zellweger, T., Portner, K., Mladenovi´c, M., Kaniselvan, M., Moraitis, T., Luisier, M., and Emboras, A. Single neuromorphic memristor closely emulates multiple synaptic mechanisms for en- ergy efficient neural networks.Nature Communications, 15(1):6898,

arXiv

[7] [18]

After performing a hyperparameter search and selecting the best model, we found that predictive coding fails to learn useful representations

because it does not require additional signals to control separate phases of updating activities and updating parameters. After performing a hyperparameter search and selecting the best model, we found that predictive coding fails to learn useful representations. To further confirm our findings of the inferior performance of predictive coding in Table 2, ...

2023

[8] [19]

Detailed choices are specified in Table A4: Table A4.Hyperparameters for different losses in predictive coding

and LPL (Halvagal & Zenke, 2023). Detailed choices are specified in Table A4: Table A4.Hyperparameters for different losses in predictive coding. T is the number of updates for h and w for each input. ηh is the learning rate for updating h. These two hyperparameters are specific to predictive coding and are thereby tuned. Other hyperparameters are kept th...

2023