Pith. sign in

REVIEW 4 major objections 4 minor 31 references

Introduction to Predictive Coding Networks for Machine Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A predictive coding network with 3.58 million parameters reaches 99.92 percent top-1 accuracy on CIFAR-10 after four epochs using only local, biology-inspired updates.

desk verdict A clean PCN tutorial wrapped around a CIFAR-10 claim that looks like test-label leakage; the benchmark should not be trusted. read the letter →

arxiv 2506.06332 v1 pith:2ZY7Y7GH submitted 2025-05-31 cs.NE cs.AIcs.LG

Mikko Stenlund This is my paper
classification cs.NEcs.AIcs.LG
keywords predictivecodingnetworkslocallearningrulesinferenceenergy-basedmodelsCIFAR-10supervisedbiologicallyplausible
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a tutorial that derives predictive coding networks from a single prediction-error energy and shows how the same update rules serve both unsupervised and supervised learning. Its central empirical claim is that a three-layer PCN with 3,577,100 parameters, trained for four epochs on CIFAR-10 with no data augmentation and no backpropagation, reaches 99.92 percent top-1 test accuracy, a result the paper says beats the previous 99.5 percent it cites. A sympathetic reader would care because the learning rule is local: each weight update uses only presynaptic activity, postsynaptic error, and a local activation derivative, which is the kind of rule biological circuits and low-power hardware can implement. The paper also releases the Python implementation and trained weights, making the claim directly checkable.

What carries the argument

The load-bearing object is the predictive-coding energy $E = \frac{1}{2}\sum_{l=0}^{L-1}\|\varepsilon^{(l)}\|^2$, where $\varepsilon^{(l)}=x^{(l)}-\hat{x}^{(l)}$ is the difference between a layer's activity and the prediction sent down from the layer above. All updates descend this energy: inference moves each latent $x^{(l)}$ by $-\eta_{\mathrm{infer}}\big(\varepsilon^{(l)} - W^{(l-1)\top}(f'^{(l-1)}(a^{(l-1)})\odot\varepsilon^{(l-1)})\big)$, and learning moves each weight by $+\eta_{\mathrm{learn}}\big(f'^{(l)}(a^{(l)})\odot\varepsilon^{(l)}\big)x^{(l+1)\top}$. The shared factor $h^{(l)}=f'^{(l)}(a^{(l)})\odot\varepsilon^{(l)}$, a gain-modulated prediction error, is what makes both updates local, and the supervised extension simply injects $W_{\mathrm{out}}^{\top}\varepsilon_{\mathrm{sup}}$ at the top layer.

What would settle it

Run the accompanying notebook's test phase many times with fresh random latent initializations and compare the distribution of top-1 accuracies; the central claim fails if the typical accuracy is far below 99.92 percent, or if accuracy drops sharply when test-time latents are all initialized to a fixed constant instead of random draws.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a predictive coding network can be trained end-to-end on a standard vision benchmark using only local, biologically plausible updates, and that this training generalizes well: after four epochs, top-1 accuracy is 99.92 percent and top-3 is 99.99 percent on CIFAR-10. The architecture has latent layers of width 1000, 500, and 10, with a linear readout from the top latent; weights are initialized once at the start, latents are reinitialized to small Gaussian noise for every sample, inference runs for 50 steps per sample at rate 0.05, and learning takes 500 weight steps per batch at rate 0.005. Because the energy landscape is deformed only by local, activity-based terms, the result indicates that global error backpropagation is not needed for strong supervised generalization in this setting.

Load-bearing premise

The load-bearing premise is that the test evaluation is clean: no label information leaks into the frozen-weight inference, and the single 99.92 percent run is representative rather than a lucky draw from the randomly initialized latents.

Editorial extensions

If this is right

  • A PCN with 3.58 million parameters reaches 99.92 percent top-1 accuracy on CIFAR-10 after four epochs with no augmentation, which the paper identifies as higher than the previous 99.5 percent it cites.
  • The same code path handles unsupervised and supervised learning; the only change is the top-layer error signal, so the local-update machinery is not tied to label supervision.
  • Because inference runs for a variable number of steps, the network supports anytime computation: simple inputs settle quickly, and only surprising inputs need deeper inference.
  • Weight updates are neuron-local in the paper's sense, compatible with plasticity that does not require a global gradient tape.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the reported accuracy is stable across repeated test rounds and independent reimplementations, it implies that optimizing latent states during inference can substitute for much of the capacity and augmentation that feedforward vision models normally require.
  • Editorial extension: the paper's random test-time latent initialization makes predictions noisy; averaging over several latent draws, or initializing latents with an amortized bottom-up network as in the hybrid model the paper mentions, would be a natural next experiment and could make the headline number more robust.
  • Editorial extension: the local update rules are stated as extending to convolutional, recurrent, and graph layers, but the paper does not test those variants; checking whether the same two-timescale training holds in those architectures is a direct testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This manuscript is a tutorial introduction to predictive coding networks (PCNs). It defines a hierarchical energy function, derives the inference and learning update rules as gradient descent on that energy, presents unsupervised and supervised algorithms in pseudocode, and gives a PyTorch-oriented implementation. The application section reports a CIFAR-10 experiment: a 3,577,100-parameter PCN trained for 4 epochs with batch size 500, no data augmentation, and hyperparameters T_infer=50, T_learn=500, achieving 99.92% top-1 and 99.99% top-3 accuracy, which the paper claims tops the Papers With Code leaderboard at the time of writing.

Significance. The derivation of the PCN inference and learning rules from a single energy is clear, correct in substance, and useful for onboarding practitioners; the explicit algorithms and the disclosure that the experiment was virtually one-shot are strengths. However, the paper's central empirical claim—that a 3.58M-parameter PCN trained for 4 epochs without augmentation beats state-of-the-art CIFAR-10 accuracy—is extraordinary and, as presented, not supported by reproducible evidence. The test protocol is ambiguous about whether labels are used during test-time inference; no error bars or repeated runs are reported despite the paper's own statement that accuracies vary with random latent initialization; the leaderboard comparison is unverified; and the text contains a direct contradiction about whether the result is competitive. If the result is real and reproducible, it would be a major finding for the PCN community, but the current evidence does not establish it.

major comments (4)
  1. [Section 5.3 and Algorithm 2] The test-time protocol is ambiguous in a load-bearing way. Section 5.3 says the inference loop is 'executed, exactly as in the base algorithm' for each input-label pair (x^(0), y). If 'base algorithm' refers to Algorithm 2 (the supervised algorithm), then the test loop computes epsilon_sup = y_hat - y and uses W_out^T epsilon_sup to update the top latent before reading out y_hat, which would be direct test-label leakage and would trivially inflate accuracy. If, instead, the test loop is meant to be the unsupervised algorithm with epsilon^(L)=0, the paper must state this explicitly and provide the test-time code. As written, the 99.92% claim in Table 1 is not verifiable.
  2. [Section 5.3 and Table 1] The reported accuracies come from a single test pass, yet Section 5.3 concedes that predictions contain random noise from the randomly initialized latents and that accuracies vary from one test round to the next. A single realization cannot support a leaderboard-topping claim; report the mean and standard deviation over repeated test rounds with different latent initializations, and ideally over multiple training seeds. Without this, the result could be a lucky draw.
  3. [Section 5.5] The statement that 'neither the model architecture nor the hyperparameters were tuned for performance in any way or at any point' is contradicted by the preceding disclosure, which describes changing eta_infer from 0.1 to 0.05 and eta_learn from 0.001 to 0.005 after the first run because inference was unstable and learning was slow. That is hyperparameter adjustment based on observed behavior, even if not a systematic search. The 'virtually one-shot' characterization must be revised to acknowledge that the reported result came from a second configuration after an unsuccessful first run.
  4. [Section 5.5 and Table 1] The assertion that 99.92% top-1 accuracy 'comfortably tops the leaderboard on Papers With Code' and that the previous record was 99.5% set by ViT-H/14 is not substantiated by any citation, link, or benchmark protocol. A claim of this magnitude requires the exact leaderboard snapshot, the evaluation protocol, and the code used to produce Table 1. The same paragraph also says PCNs are 'generally not yet competitive with state-of-the-art deep learning methods in terms of... raw accuracy,' which directly contradicts the preceding leaderboard claim; these statements need to be reconciled.
minor comments (4)
  1. [Section 2] The convergence statement says 'several works have established... that the alternating optimization procedure... converges to a local minimum' and cites reference [7], whose title is '(Non-)Convergence results for predictive coding networks.' The statement should be qualified to distinguish the settings in which convergence is guaranteed from those in which it is not.
  2. [Section 4.2] The text says the supervised extension 'entails simply clamping the top latent representation x^(L) to a predicted label y_hat,' but the described method adds a readout layer and does not clamp x^(L); this wording is misleading and should be corrected.
  3. [Abstract and Section 5] The dataset name appears as 'CIF AR-10' in the abstract and once in Section 5; it should be 'CIFAR-10.'
  4. [Section 5.5] The closing sentence 'Happy hunting!' is informal for a journal article and should be removed or replaced.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: PCN update rules follow from the stated energy, and the CIFAR-10 benchmark is an empirical report, not a restatement of its inputs.

full rationale

The paper's derivation chain is self-contained: the inference and learning rules in Sections 3.1-3.2 are computed by explicit gradient descent on the stated energy L = (1/2) sum ||epsilon^(l)||^2, and the supervised extension in Section 4.2 adds L_sup = (1/2)||y_hat - y||^2 and backpropagates it into x^(L) by the same gradient calculation. There is no fitted parameter relabeled as a prediction, no uniqueness theorem imported from the authors' prior work, and no known empirical pattern merely renamed. The CIFAR-10 accuracy in Table 1 is presented as an experimental observation with a disclosure that the run was one-shot and hyperparameters were not tuned, which is an empirical claim rather than a circular derivation. The only self-referential element is the pointer to the author's GitHub repository for code and trained weights, which is not load-bearing for any mathematical or empirical assertion. The test section would be clearer if it explicitly stated that labels are withheld from the test-time inference loop, since the phrase 'exactly as in the base algorithm' is ambiguous given Algorithm 2's use of y, but that ambiguity is a potential implementation/leakage concern, not a circularity in the paper's argument. No load-bearing step reduces by construction to its own input.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The theoretical derivations rely on standard matrix calculus and on cited convergence results. The empirical headline depends on several hand-chosen hyperparameters, the exact test-time protocol, and random latent initialization. No new theoretical entities are introduced.

free parameters (7)
  • eta_infer = 0.05
    Inference step size; initially 0.1, halved after the first run became unstable.
  • eta_learn = 0.005
    Learning rate; initially 0.001, quintupled after the first run was slow.
  • T_infer = 50
    Number of inference steps per sample; chosen by hand.
  • T_learn = 500
    Learning steps per batch, set equal to batch size; chosen by hand.
  • architecture = 3072-1000-500-10 plus 10x10 readout
    Layer widths chosen by hand following 'battle-tested practice'; no architecture search.
  • batch_size = 500
    Batch size chosen by hand; affects gradient averaging and number of updates.
  • latent_initialization = N(0,1)
    Latents initialized as standard Gaussians, both in training and testing; no warm starting or ensembling.
assumptions (5)
  • domain assumption Alternating inference and learning converges to a local minimum of the PCN energy.
    Relied on in Section 2, Convergence; cited to prior works, not proved for this setting.
  • standard math The ReLU derivative is a valid local gain signal for inference and learning.
    Used throughout Algorithms 1-3; standard in gradient-based training despite non-differentiability at zero.
  • domain assumption Flattened CIFAR-10 images are a suitable input representation for dense PCNs.
    Section 5.1 flattens 32x32x3 images to 3072-d vectors, discarding spatial structure.
  • ad hoc to paper Decreasing energy trajectories are a valid implementation sanity check.
    Section 5.4 asserts trajectories should strictly decrease for small rates and uses this as a correctness check; no rigorous guarantee is given for the selected rates.
  • domain assumption Randomly initialized latents at test time give representative predictions.
    Section 5.3 acknowledges prediction noise and run-to-run variation; the single reported accuracy assumes this variance is negligible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Introduction to Predictive Coding Networks for Machine Learning." pith.science (2026). https://pith.science/paper/2ZY7Y7GH

@misc{pith2026250606332,
  author       = {Pith},
  title        = {Pith review of: Introduction to Predictive Coding Networks for Machine Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZY7Y7GH}},
  note         = {Machine review of arXiv:2506.06332}
}
read the original abstract

Predictive coding networks (PCNs) constitute a biologically inspired framework for understanding hierarchical computation in the brain, and offer an alternative to traditional feedforward neural networks in ML. This note serves as a quick, onboarding introduction to PCNs for machine learning practitioners. We cover the foundational network architecture, inference and learning update rules, and algorithmic implementation. A concrete image-classification task (CIFAR-10) is provided as a benchmark-smashing application, together with an accompanying Python notebook containing the PyTorch implementation.

Figures

Figures reproduced from arXiv: 2506.06332 by the authors.

Figure 1
Figure 1. A graph representation of a PCN with three latent layers. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A supervised extension of the PCN with three latent layers in Figure [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Partial zoom-in views of the batch-averaged energy trajectories recorded during [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 13 canonical work pages

  1. [1]

    Predictive coding: A brief introduction and review for machine learning researchers, 2022

    Nick Alonso. Predictive coding: A brief introduction and review for machine learning researchers, 2022. Blog post. URL:https://neuralnetnick.com/2022/12/28/

    work page 2022

  2. [2]

    A theoretical framework for inference learning

    Nick Alonso, Beren Millidge, Jeff Krichmar, and Emre Neftci. A theoretical framework for inference learning. InProceedings of the 36th International Conference on Neural Information Processing Systems, volume 36, pages 37335–37348, 2022. URL:https:// dl.acm.org/doi/10.5555/3600270.3602976

    work page arXiv 2022

  3. [3]

    Horace B. Barlow. Possible principles underlying the transformation of sensory messages. In W. A. Rosenblith, editor,Sensory Communication, pages 217–234. MIT Press, Cam- bridge, MA, 1961

    work page 1961

  4. [4]

    Bastos, W

    Andre M. Bastos, W. Martin Usrey, Rick A. Adams, George R. Mangun, Pascal Fries, and Karl J. Friston. Canonical microcircuits for predictive coding.Neuron, 76(4):695–711, 2012.doi:10.1016/j.neuron.2012.10.038

    doi:10.1016/j.neuron.2012.10.038 2012

  5. [5]

    Caucheteux, A

    C. Caucheteux, A. Gramfort, and JR King. Evidence of a predictive coding hierarchy in the human brain listening to speech.Nature Human Behaviour, 7:430–441, 2023.doi: 10.1038/s41562-022-01516-2

    doi:10.1038/s41562-022-01516-2 2023

  6. [6]

    Loihi: A neuromorphic manycore processor with on-chip learning.IEEE Micro, 38(1):82–99, 2018

    Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, George Dimou, Harish Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning.IEEE Micro, 38(1):82–99, 2018. doi:10.1109/MM.2018.112130359

    arXiv 2018

  7. [7]

    (Non-)Convergence results for predictive coding networks

    Simon Frieder and Thomas Lukasiewicz. (Non-)Convergence results for predictive coding networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Research, pages 6793–6810, 2022. URL:https...

    work page 2022

  8. [8]

    A theory of cortical responses.Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1456):815–836, 2005.doi:10.1098/rstb.2005.1622

    Karl Friston. A theory of cortical responses.Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1456):815–836, 2005.doi:10.1098/rstb.2005.1622

    arXiv 2005

Show all 31 references
  1. [9]

    The free-energy principle: a unified brain theory?Nature Reviews Neuro- science, 11(2):127–138, 2010.doi:10.1038/nrn2787

    Karl Friston. The free-energy principle: a unified brain theory?Nature Reviews Neuro- science, 11(2):127–138, 2010.doi:10.1038/nrn2787

    2010 doi

  2. [10]

    Richard L. Gregory. Perceptions as hypotheses.Philosophical Transactions of the Royal Society of London B: Biological Sciences, 290(1038):181–197, 1980.doi:10.1098/rstb. 1980.0090

    1980

  3. [11]

    World models.CoRR, abs/1803.10122, 2018

    David Ha and J¨ urgen Schmidhuber. World models.CoRR, abs/1803.10122, 2018. URL: http://arxiv.org/abs/1803.10122

    2018 arXiv

  4. [12]

    Dover, New York, 1867

    Hermann von Helmholtz.Treatise on Physiological Optics, Volume III: Concerning the perceptions in general. Dover, New York, 1867. Translated by J. P. C. Southall, 3rd ed., 1962

    1962

  5. [13]

    Keller and Thomas D

    Georg B. Keller and Thomas D. Mrsic-Flogel. Predictive processing: A canonical cortical computation.Neuron, 100(2):424–435, 2018.doi:10.1016/j.neuron.2018.10.003. 20

    2018 doi

  6. [14]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Techni- cal report, University of Toronto, 2009. URL:https://www.cs.toronto.edu/ ~kriz/ learning-features-2009-TR.pdf

    2009

  7. [15]

    Deep predictive coding networks for video prediction and unsupervised learning

    William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. InInternational Conference on Learning Rep- resentations (ICLR), 2017. URL:https://arxiv.org/abs/1605.08104

    2017 arXiv

  8. [16]

    Tight stability, convergence, and robustness bounds for predictive coding networks

    Ankur Mali, Tommaso Salvatori, and Alexander Ororbia. Tight stability, convergence, and robustness bounds for predictive coding networks. 2024.doi:10.48550/arXiv.2410. 04708

    2024 doi

  9. [17]

    Predictive coding as backprop and natu- ral gradients, 2020

    Beren Millidge. Predictive coding as backprop and natu- ral gradients, 2020. Blog post. URL:https://www.beren.io/ 2020-09-12-Predictive-Coding-As-Backprop-And-Natural-Gradients/

    2020

  10. [18]

    Seth, and Christopher L

    Beren Millidge, Anil K. Seth, and Christopher L. Buckley. Predictive coding: a theoretical and experimental review. 2021. URL:https://arxiv.org/abs/2107.12979

    2021 arXiv

  11. [19]

    Beren Millidge, Alexander Tschantz, and Christopher L. Buckley. Predictive cod- ing approximates backprop along arbitrary computation graphs.Neural Computation, 34:1329–1368, 2022.doi:10.1162/neco_a_01497

    2022 doi

  12. [20]

    An introduction to predictive coding networks for machine learning, 2025

    Monadillo. An introduction to predictive coding networks for machine learning, 2025. GitHub repository containing the following supplements to this document: Python note- book and model weights. URL:https://github.com/Monadillo/pcn-intro

    2025

  13. [21]

    On the computational architecture of the neocortex

    David Mumford. On the computational architecture of the neocortex. ii. the role of cortico- cortical loops.Biological Cybernetics, 66(3):241–251, 1992.doi:10.1007/BF00198477

    1992 doi

  14. [22]

    N’dri, William Gebhardt, C´ eline Teuli` ere, Fleur Zeldenrust, Rajesh P

    Antony W. N’dri, William Gebhardt, C´ eline Teuli` ere, Fleur Zeldenrust, Rajesh P. N. Rao, Jochen Triesch, and Alexander Ororbia. Predictive coding with spiking neural networks: a survey. 2024. URL:https://arxiv.org/abs/2409.05386

    2024 arXiv

  15. [23]

    Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.Nature neuroscience, 2(1):79– 87, 1999.doi:10.1038/4580

    1999 doi

  16. [24]

    A stable, fast, and fully automatic learning algorithm for predictive coding networks

    Tommaso Salvatori, Yuhang Song, Yordan Yordanov, Beren Millidge, Zhenghua Xu, Lei Sha, Cornelius Emde, Rafal Bogacz, and Thomas Lukasiewicz. A stable, fast, and fully automatic learning algorithm for predictive coding networks. InInternational Conference on Learning Representa...

    2024 doi arXiv

  17. [25]

    Learning factorial codes by predictability minimization.Neural Computation, 4(6):863–879, 1992.doi:10.1162/neco.1992.4.6.863

    J¨ urgen Schmidhuber. Learning factorial codes by predictability minimization.Neural Computation, 4(6):863–879, 1992.doi:10.1162/neco.1992.4.6.863

    1992 doi

  18. [26]

    Neural elements for predictive coding.Frontiers in Psychology, 7:1792, 2016.doi:10.3389/fpsyg.2016.01792

    Stewart Shipp. Neural elements for predictive coding.Frontiers in Psychology, 7:1792, 2016.doi:10.3389/fpsyg.2016.01792

    2016

  19. [27]

    Can the brain do backpropagation? — exact implementation of backpropagation in predictive coding networks

    Yuhang Song, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Can the brain do backpropagation? — exact implementation of backpropagation in predictive coding networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Ad- vances in Neural Informatio...

    2020

  20. [28]

    Spratling

    M.W. Spratling. Reconciling predictive coding and biased competition models of cortical function.Frontiers in Computational Neuroscience, 2:4, 2008.doi:10.3389/neuro.10. 004.2008

    2008 doi

  21. [29]

    Seth, and Christopher L

    Alexander Tschantz, Beren Millidge, Anil K. Seth, and Christopher L. Buckley. Hybrid predictive coding: Inferring, fast and slow.PLOS Computational Biology, 19(8):1–31, 08 2023.doi:10.1371/journal.pcbi.1011280

    2023 doi

  22. [30]

    K. S. Walsh, D. P. McGovern, A. Clark, and R. G. O’Connell. Evaluating the neurophys- iological evidence for predictive processing as a model of perception.Ann N Y Acad Sci, 464(1):242–268, 3 2020.doi:10.1111/nyas.14321

    2020 doi

  23. [31]

    James C. R. Whittington and Rafal Bogacz. An approximation of the error backprop- agation algorithm in a predictive coding network with local hebbian synaptic plasticity. Neural Computation, 29:1229–1262, 2017.doi:10.1162/NECO_a_00949. 22

    2017 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.