Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Predictive coding is a block-coordinate descent method on the MDL two-part codelength; each local sweep provably tightens a high-probability risk bound.

desk verdict The central claim is about a redefined exact-argmin algorithm, and the codelength objective changes between sections; the actual PC update is never shown to descend the MDL objective. read the letter →

arxiv 2505.14635 v2 pith:65TL6CIT submitted 2025-05-20 cs.LG

Benjamin Prada , Shion Matsumoto , Abdul Malik Zekri , Ankur Mali This is my paper
classification cs.LG MSC 68T0768Q3290C2694A17
keywords predictivecodingminimumdescriptionlengthtwo-partcodegeneralizationboundblock-coordinatedescentmodelcomplexitylocallearningrulesdeepnetworktheory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Predictive coding (PC) is a biologically inspired learning rule in which neurons reduce local prediction errors; this paper claims that, when run layer by layer, PC is exactly a block-coordinate descent algorithm on the minimum description length (MDL) two-part code objective. The central result is that one full sweep of layerwise PC updates monotonically decreases the empirical codelength Ĉ(θ) = R̂(θ) + L(θ)/N, where R̂ is the training loss and L(θ) = −log p(θ) is the parameter codelength under a factorized prefix-code prior. Under an exact-layer-minimization assumption, repeated sweeps converge to a blockwise stationary point of that codelength, so PC finds a local MDL optimum. If these theorems are correct, PC-trained networks inherit a high-probability risk bound R(θ) ≤ R̂(θ) + L(θ)/N + ln(1/δ)/N that tightens with every sweep, giving a generalization guarantee for PC that does not depend on approximating backpropagation.

What carries the argument

The load-bearing object is the empirical two-part codelength $\hat{C}(\theta) = \hat{R}(\theta) + \frac{1}{N} L(\theta)$, where $\hat{R}$ is the average per-sample negative log-likelihood and $L(\theta) = -\log p(\theta)$ is the code-length of the parameters under a factorized prefix-code prior. The proof machinery is the identification of a PC sweep with exact cyclic block-coordinate descent on $\hat{C}$: at each layer, in the idealized update, the weights are chosen as the exact minimizer of the codelength with all other layers fixed. This identification turns the generalization bound $R(\theta) \le \hat{R}(\theta) + L(\theta)/N + \ln(1/\delta)/N$ into an objective that PC actively descends, so each sweep both compresses the model and tightens the high-probability risk guarantee.

What would settle it

Run the standard predictive-coding update $\Delta\theta_l = -\eta(\varepsilon_l \odot f'(\theta_l x_{l-1})) x_{l-1}^T$ on a network and record $\hat{C}(\theta) = \hat{R}(\theta) + \frac{1}{N} L(\theta)$ after every sweep; if $\hat{C}$ ever increases on a single sweep, the claimed monotone codelength decrease fails for the actual algorithm. At convergence, perturb one layer's weights by small noise and count how often $\hat{C}$ decreases; the paper predicts this should essentially never happen, so a large fraction of decreases would falsify the blockwise-stationarity claim.

Watch

Extended reading notes

Core claim

The paper's core discovery is that the local, energy-based dynamics of predictive coding can be read as exact layerwise minimization of a two-part description length, not merely as a biologically plausible approximation to backpropagation. Formally, defining the empirical codelength $\hat{C}(\theta) = \hat{R}(\theta) + \frac{1}{N} L(\theta)$ with $L(\theta) = -\log p(\theta)$, the paper proves that every block update in a PC sweep chooses the layer weights that minimize $\hat{C}$ holding other layers fixed; therefore one sweep cannot increase $\hat{C}$, and applying the concentration-based bound $R(\theta) \le \hat{R}(\theta) + L(\theta)/N + \ln(1/\delta)/N$ to the updated parameters gives a tighter generalization guarantee. Repeating sweeps yields a nonincreasing, bounded codelength sequence whose limit point satisfies the block-coordinate stationarity condition $\theta^\infty_l = \arg\min_{\theta_l} \hat{C}(\theta^\infty_1, \dots, \theta_l, \dots, \theta^\infty_L)$. The paper frames this as a convergence to an approximate MDL-optimal solution.

Load-bearing premise

The proof needs each layer to be updated to the exact best value for that layer while the other layers stay fixed, but the actual predictive-coding rule takes a small local gradient step; the paper does not show these are equivalent.

Editorial extensions

If this is right

  • One full layerwise PC sweep never increases the empirical two-part codelength, so starting from any parameters, the high-probability risk bound improves after a single sweep.
  • Under exact layer minimization, repeated PC sweeps converge to a blockwise stationary point of the MDL objective, meaning no single layer can be adjusted alone to lower the codelength.
  • On a fixed compute budget, a PC sweep and a backpropagation pass have comparable cost, and the paper's budget analysis argues PC yields a strictly tighter risk bound than backpropagation under its depth-attenuated gradient assumption.
  • Because the generalization bound is algorithm-independent, the PC-specific contribution is a proof that this particular local learning rule actually minimizes the right-hand side of the bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer a testable design principle: any local learning rule that can be shown to implement block-coordinate descent on a two-part codelength would inherit the same Occam-style risk bound, so other energy-based rules could be compared by measuring their codelength decrease per sweep.
  • A direct extension would replace the exact argmin update with the actual predictive-coding gradient step and measure how much codelength decreases per sweep; the theorem's exact-layer-minimization assumption is exactly what such an experiment would probe.
  • If the exact-argmin idealization fails, the monotone descent theorem need not hold, so the practical value of the framework hinges on whether the local PC update descends the empirical codelength on real losses, which the paper's simulation suggests but does not prove.
  • The bound's dependence on the prior suggests an experimental comparison: varying the Gaussian prior scale should change the generalization gap for PC-trained networks in a predictable way, since larger code-lengths worsen the right-hand side of the risk bound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a theoretical link between predictive coding (PC) and the minimum description length (MDL) principle. It derives an Occam-style generalization bound (Theorem 1), claims that each layerwise PC sweep monotonically reduces an empirical two-part codelength (Theorem 2), and claims that repeated PC updates converge to a blockwise stationary point of that codelength (Theorem 3). The appendices contain additional rate claims and a small two-layer linear simulation comparing PC and backpropagation. The central thesis is that PC should be understood as a compression-driven learning algorithm rather than merely an approximation to backpropagation.

Significance. If the central claims held, the paper would provide a useful new interpretation of predictive coding and would connect a biologically plausible learning rule to an information-theoretic generalization bound. Theorem 1 is a standard Occam/PAC-Bayes-type bound and is essentially correct under its stated assumptions. However, the paper's main results are not established for the predictive-coding algorithm defined in Section 2: Theorem 2 and Assumption A3 replace the local gradient update with an exact blockwise argmin, and the codelength objective changes between Sections 3.1 and 3.2. Because these issues affect the headline contributions, the significance of the paper as it stands is limited.

major comments (4)
  1. [Section 3.2, Theorem 2] The theorem defines 'one full sweep of layerwise PC' as θ_l^{(l)} = arg min_{θ_l} Σ_i c(θ; x_i). This is not the PC learning update of Section 2, which is Δθ_l = -η(ε_l ⊙ f'(θ_l x_{l-1})) x_{l-1}^T. No result in the paper shows that this finite-step-size gradient update descends the blockwise codelength objective or even decreases it; Appendix C.1 asserts a sufficient-decrease condition 'via error-correcting feedback' but gives no proof from the Section 2 update. Consequently, Theorems 2 and 3 (through Assumption A3) apply to an exact block-coordinate minimization algorithm, not to the PC algorithm that is defined and simulated in the paper.
  2. [Sections 3.1 and 3.2] The empirical codelength is defined inconsistently. Section 3.1 defines ĀC(θ) = ĀR(θ) + L(θ)/N, while Section 3.2 defines per-sample c(θ; x) = L(θ) + ℓ(θ; x) and ĀC(θ) = (1/N)Σ_i c(θ; x_i) = L(θ) + ĀR(θ). These differ by a factor N in the complexity term. Theorem 2's conclusion R(θ_PC) ≤ ĀC(θ^{(0)}) + ln(1/δ)/N is therefore a statement about a different, much weaker objective, and it does not connect to the MDL objective used in the abstract and Section 3.1. The proof of Theorem 2 also writes ĀC as a sum without the 1/N factor, reinforcing the inconsistency.
  3. [Section 3.3, Theorem 3] The convergence proof is incomplete even if Assumption A3 is granted. Monotone convergence of the real sequence C(θ^{(t)}) does not imply convergence of the parameter sequence; the Bolzano-Weierstrass step requires boundedness of the iterates or of a level set, which is not stated among (A1)-(A3). Moreover, continuity and exact coordinate minimization do not by themselves guarantee that every limit point is a coordinate-wise minimizer of C; classical coordinate-descent convergence results require additional hypotheses on the objective and the update rule. The theorem as stated is therefore not proven.
  4. [Appendix A, Lemma 4 and Theorem 4] The claimed norm contraction for backpropagation is not justified. From ‖θ_{t+1}‖² = ‖θ_t‖² + η²‖∇ĀR‖² - 2η⟨θ_t, ∇ĀR⟩, the proof derives a lower bound ‖θ_T‖² ≥ ‖θ_0‖² + η²G²(Σ ρ^{L-l†})². But Cauchy-Schwarz only bounds the absolute value of the inner product; without a sign or alignment assumption on ⟨θ_t, ∇ĀR⟩, the parameter norm can decrease. Thus the claimed strict Occam improvement of PC over backpropagation under equal compute is not established.
minor comments (5)
  1. [Theorem 2] The theorem uses ln(1/δ) in the final bound, while Theorem 1 and Lemma 2 have ln(2/δ); the mismatch should be corrected.
  2. [Section 4.2] Figure 1 reports mean trajectories and shaded standard deviations but no statistical comparison; the claim that PC 'consistently' reaches lower MDL values across all seeds is not directly demonstrated.
  3. [Section 4.3] The perturbation percentages (2.5% vs 18.7%) are reported without a null model or confidence intervals, so their evidentiary value for blockwise stationarity is limited.
  4. [Appendix C.1] The assertion that 'asymptotic regularity' Σ_t ‖θ^{(t+1)} - θ^{(t)}‖² < ∞ follows from a decreasing and bounded energy sequence is not valid without additional assumptions, such as a sufficient-descent condition that supplies a quantitative decrease.
  5. [Section 1] The introductory 'Summary of Contributions' describes the bound as holding 'for every parameter realization' and any learner, which is fine, but the word 'novel' overstates what is a standard Occam/PAC-Bayes union-bound argument.

Circularity Check

2 steps flagged · score 7.0 of 10

Theorems 2–3 prove descent/convergence for a PC sweep redefined as exact blockwise codelength minimization, not for the gradient-based PC update of Section 2; the codelength objective also changes between §3.1 and §3.2.

  1. self definitional [Section 3.2, Theorem 2 (layerwise PC sweep definition and proof), contrasted with Section 2 'Predictive Coding Updates']
    "for l = 1, . . . , L: θ(l) l = arg min θl P N i=1 c(θ(l−1) 1 , . . . , θl, . . . , θ(l−1) L ; x(i))."

    The theorem's PC sweep is defined to be the exact minimizer of the per-sample codelength c(θ; x). With that definition, the inequality Ĉ(θPC) ≤ Ĉ(θ(0)) is a direct consequence of blockwise coordinate minimization and needs no property of predictive coding. The algorithm actually introduced in Section 2 is the local gradient update Δθl = −η(εl ⊙ f′(θlxl−1))x⊤ l−1 after inference; the paper never shows that this finite-step update equals or descends the exact blockwise argmin of the codelength. Hence the headline 'PC minimizes empirical codelength' is true for the theorem's redefined update by construction, not for the §2 PC dynamics.

  2. self definitional [Section 3.3, Theorem 3, Assumption (A3) and proof]
    "(A3) (Exact Layer Minimization) At each PC update step, the updated layer θl exactly minimizes C with respect to its block, holding other layers fixed."

    The assumption already asserts that PC's updates are exact blockwise minimizers of the codelength C. The theorem then derives standard coordinate-descent consequences: monotone convergence of C(θ(t)) and blockwise stationarity at a limit point. This is the textbook BCD theorem, and the proof uses (A3) directly in step (iii); no argument connects (A3) to the energy-based updates Δθl = −η(εl ⊙ f′(θlxl−1))x⊤ l−1 from Section 2. The convergence result is therefore imported from the assumption rather than derived from the PC algorithm claimed to be analyzed.

full rationale

The independent component is Theorem 1 (Occam MDL bound): it is a self-contained Bernstein/union-bound argument and does not depend on PC. If the paper only claimed that bound, it would receive a low circularity score. However, the paper's central load-bearing results, Theorems 2 and 3, reduce to definitions and assumptions. Theorem 2 defines 'one full sweep of layerwise PC' as θl = arg min over the codelength, making monotone codelength decrease true by construction; it does not prove that the gradient PC update in Section 2 performs this exact minimization. Theorem 3 assumes 'Exact Layer Minimization' (A3), which is precisely the property the theorem concludes, so the convergence result is the standard block-coordinate-descent fact imported as an assumption. In addition, the codelength objective changes between sections: §3.1 defines Ĉ(θ) = R̂(θ) + L(θ)/N, while §3.2 defines c(θ;x) = L(θ) + ℓ(θ;x) and Ĉ = (1/N)Σ c = R̂ + L, so the quantity whose descent is proved is not the quantity appearing in the Occam bound as stated. There is no load-bearing self-citation; the flaw is definitional substitution, not citation laundering. A score of 7 reflects that the central PC-MDL claim is partially circular/forced by definition, while the generalization bound itself remains an independent standard result.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The paper's central claims rest on standard probabilistic inequalities and on a sequence of domain assumptions about the loss, the prior, and the update rule. The most consequential assumption is that a PC sweep is an exact blockwise minimizer of the two-part codelength; this is what turns the descent statement into a tautology and is not true of the gradient-based PC updates in the paper.

free parameters (2)
  • Gaussian prior precision α
    Used in the illustrative regression examples and in Appendix A to define the complexity penalty L(θ) = (1/(2σ_p^2)) Σ ||θ_l||^2. The value is chosen by hand and affects the bound's numerical value, but no data are fitted.
  • Noise variance σ^2 = 0.1 (simulation)
    Used in the two-layer linear simulation (Table 1). The value is chosen for the experiment and does not affect the theoretical claim.
assumptions (7)
  • standard math Bernstein's inequality
    Lemma 1, used to control deviations of bounded i.i.d. losses.
  • standard math Union bound over countable hypothesis class
    Lemma 2 and Theorem 1 rely on a countable Θ and a prefix-code prior so that Σ_θ p(θ) ≤ 1.
  • domain assumption Loss per example bounded in [0,1]
    Theorem 1 assumes 0 ≤ ℓ(θ;x) ≤ 1 for all θ and x; negative log-likelihoods are not bounded in general.
  • domain assumption Factorized layerwise likelihood and prior
    Section 3 postulates P_θ(D) = Π_i Π_l P_{θ_l}(x_l^{(i)} | x_<l^{(i)}) and p(θ) = Π_l p_l(θ_l), which is a specific generative structure not true for arbitrary deep networks.
  • ad hoc to paper Exact layer minimization (A3)
    Theorem 2 and Theorem 3 assume each PC update exactly minimizes the full codelength over a layer. Real predictive coding uses local gradient updates, not exact global minimization.
  • domain assumption Continuity and boundedness of C (A1, A2)
    Theorem 3 requires C ≥ 0 and continuous; these are mild but not automatic.
  • ad hoc to paper Depth-attenuated BP gradients (A3 in Appendix A)
    Lemma 4 and Theorem 4 in Appendix A assume ||∇_{θ_l} R̂|| ≤ G ρ^{L-l}, a condition used to show PC keeps smaller norm than BP.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning." pith.science (2026). https://pith.science/paper/65TL6CIT

@misc{pith2026250514635,
  author       = {Pith},
  title        = {Pith review of: Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/65TL6CIT}},
  note         = {Machine review of arXiv:2505.14635}
}
abstract

We present the first theoretical framework that connects predictive coding (PC), a biologically inspired local learning rule, with the minimum description length (MDL) principle in deep networks. We prove that layerwise PC performs block-coordinate descent on the MDL two-part code objective, thereby jointly minimizing empirical risk and model complexity. Using Hoeffding's inequality and a prefix-code prior, we derive a novel generalization bound of the form $R(\theta) \le \hat{R}(\theta) + \frac{L(\theta)}{N}$, capturing the tradeoff between fit and compression. We further prove that each PC sweep monotonically decreases the empirical two-part codelength, yielding tighter high-probability risk bounds than unconstrained gradient descent. Finally, we show that repeated PC updates converge to a block-coordinate stationary point, providing an approximate MDL-optimal solution. To our knowledge, this is the first result offering formal generalization and convergence guarantees for PC-trained deep models, positioning PC as a theoretically grounded and biologically plausible alternative to backpropagation.

Figures

Figures reproduced from arXiv: 2505.14635 by the authors.

Figure 1
Figure 1. MDL convergence comparison for PC and BP. Solid lines denote mean MDL cost; shaded [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Histogram of MDL cost changes after random perturbations at the PC convergence point. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization

    cs.IT 2026-03 reject novelty 4.0 of 10

    A curvature-based layer gain is converted into optimal allocation and pruning via water-filling and convex penalties, but the experiments substitute the baseline's influence scores and show only modest, mixed gains.

Reference graph

Works this paper leans on

43 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Allen-Zhu, Y

    Z. Allen-Zhu, Y . Li, and Z. Song. A convergence theory for deep learning via over- parameterization. In International conference on machine learning, pages 242–252. PMLR, 2019

    work page 2019

  2. [2]

    Alonso, B

    N. Alonso, B. Millidge, J. Krichmar, and E. O. Neftci. A theoretical framework for inference learning. Advances in Neural Information Processing Systems, 35:37335–37348, 2022

    work page 2022

  3. [3]

    Beck and L

    A. Beck and L. Tetruashvili. On the convergence of block coordinate descent type methods. SIAM journal on Optimization, 23(4):2037–2060, 2013

    work page 2013

  4. [4]

    Bernstein

    S. Bernstein. On a modification of chebyshev’s inequality and of the error formula of laplace. Ann. Sci. Inst. Sav. Ukraine, Sect. Math, 1(4):38–49, 1924

    work page 1924

  5. [5]

    Bolte, S

    J. Bolte, S. Sabach, and M. Teboulle. Proximal alternating linearized minimization for noncon- vex and nonsmooth problems. Mathematical Programming, 146(1):459–494, 2014

    work page 2014

  6. [6]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, 2019

    work page 2019

  7. [7]

    Friston and S

    K. Friston and S. Kiebel. Predictive coding under the free-energy principle. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521):1211–1221, 2009

    work page 2009

  8. [8]

    Grippof and M

    L. Grippof and M. Sciandrone. Globally convergent block-coordinate techniques for uncon- strained optimization. Optimization methods and software, 10(4):587–637, 1999

    work page 1999

Show all 43 references
  1. [9]

    C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T....

    2020

  2. [10]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770–778, 2016

    2016

  3. [11]

    Hinton, L

    G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V . Vanhoucke, P. Nguyen, T. N. Sainath, et al. Deep neural networks for acoustic modeling in speech recogni- tion. IEEE Signal Processing Magazine, 29(6):82–97, 2012

    2012

  4. [12]

    Kendall, R

    J. Kendall, R. Pantone, K. Manickavasagam, Y . Bengio, and B. Scellier. Training end-to-end analog neural networks with equilibrium propagation. arXiv:2006.01981, 2020

    2006 arXiv

  5. [13]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, volume 25, 2012

    2012

  6. [14]

    A. Mali, T. Salvatori, and A. Ororbia. Tight stability, convergence, and robustness bounds for predictive coding networks. arXiv preprint arXiv:2410.04708, 2024

    2024 arXiv

  7. [15]

    Millidge, T

    B. Millidge, T. Salvatori, Y . Song, R. Bogacz, and T. Lukasiewicz. Predictive coding: Towards a future of deep learning beyond backpropagation? In Proceedings of the 31st International Joint Conference on Artificial Intelligence and the 25th European Conference on Artificial ...

    2022

  8. [16]

    Millidge, A

    B. Millidge, A. Seth, and C. L. Buckley. Predictive coding: A theoretical and experimental review. arXiv:2107.12979, 2021

    2021 arXiv

  9. [17]

    Millidge, Y

    B. Millidge, Y . Song, T. Salvatori, T. Lukasiewicz, and R. Bogacz. A theoretical framework for inference and learning in predictive coding networks. In The Eleventh International Conference on Learning Representations, 2023

    2023

  10. [18]

    Nesterov

    Y . Nesterov. Smooth minimization of non-smooth functions. Mathematical programming, 103:127–152, 2005

    2005

  11. [19]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. https://openai.com/research/gpt-4, 2023

    2023

  12. [20]

    A. Ororbia. Spiking neural predictive coding for continually learning from data streams. Neurocomputing, 544:126292, 2023

    2023

  13. [21]

    Ororbia and D

    A. Ororbia and D. Kifer. The neural coding framework for learning generative models. Nature Communications, 13(1):1–14, 2022

    2022

  14. [22]

    Ororbia and A

    A. Ororbia and A. Mali. Convolutional neural generative coding: Scaling predictive coding to natural images. arXiv preprint arXiv:2211.12047, 2022

    2022 arXiv

  15. [23]

    Ororbia and A

    A. Ororbia and A. Mali. Active predictive coding: Brain-inspired reinforcement learning for sparse reward robotic control problems. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3015–3021. IEEE, 2023

    2023

  16. [24]

    Ororbia, A

    A. Ororbia, A. Mali, C. L. Giles, and D. Kifer. Continual learning of recurrent neural networks by locally aligning distributed representations. IEEE Transactions on Neural Networks and Learning Systems, 31(10):4267–4278, 2020

    2020

  17. [25]

    Ororbia, A

    A. Ororbia, A. Mali, C. L. Giles, and D. Kifer. Lifelong neural predictive coding: Learning cumulatively online without forgetting. Advances in Neural Information Processing Systems, 35:5867–5881, 2022

    2022

  18. [26]

    Pinchetti, C

    L. Pinchetti, C. Qi, O. Lokshyn, G. Olivers, C. Emde, M. Tang, A. M’Charrak, S. Frieder, B. Menzat, R. Bogacz, et al. Benchmarking predictive coding networks–made simple. arXiv preprint arXiv:2407.01163, 2024

    2024 arXiv

  19. [27]

    Radford, J

    A. Radford, J. W. Kim, M. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. 11

    2021

  20. [28]

    R. P. Rao, D. C. Gklezakos, and V . Sathish. Active predictive coding: A unifying neural model for active perception, compositional learning, and hierarchical planning. Neural Computation, 36(1):1–32, 2023

    2023

  21. [29]

    R. P. N. Rao and D. H. Ballard. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79–87, 1999

    1999

  22. [30]

    Rissanen

    J. Rissanen. Modeling by shortest data description. Automatica, 14:465–471, 1978

    1978

  23. [31]

    Salvatori, A

    T. Salvatori, A. Mali, C. L. Buckley, T. Lukasiewicz, R. P. Rao, K. Friston, and A. Ororbia. Brain- inspired computational intelligence via predictive coding. arXiv preprint arXiv:2308.07870, 2023

    2023

  24. [32]

    Salvatori, L

    T. Salvatori, L. Pinchetti, B. Millidge, Y . Song, T. Bao, R. Bogacz, and T. Lukasiewicz. Learning on arbitrary graph topologies via predictive coding. Advances in Neural Information Processing Systems, 2022

    2022

  25. [33]

    Salvatori, Y

    T. Salvatori, Y . Song, Y . Hong, L. Sha, S. Frieder, Z. Xu, R. Bogacz, and T. Lukasiewicz. Associative memories via predictive coding. In Advances in Neural Information Processing Systems, volume 34, 2021

    2021

  26. [34]

    Salvatori, Y

    T. Salvatori, Y . Song, Z. Xu, T. Lukasiewicz, and R. Bogacz. Reverse differentiation via predictive coding. In Proceedings of the 36th AAAI Conference on Artificial Intelligence. AAAI Press, 2022

    2022

  27. [35]

    Y . Song, T. Lukasiewicz, Z. Xu, and R. Bogacz. Can the brain do backpropagation? — Exact implementation of backpropagation in predictive coding networks. In Advances in Neural Information Processing Systems, volume 33, 2020

    2020

  28. [36]

    Y . Song, B. G. Millidge, T. Salvatori, T. Lukasiewicz, Z. Xu, and R. Bogacz. Inferring neural activity before plasticity: A foundation for learning beyond backpropagation. Nature Neuroscience, 2023

    2023

  29. [37]

    M. Tang, T. Salvatori, B. Millidge, Y . Song, T. Lukasiewicz, and R. Bogacz. Recurrent predictive coding models for associative memory employing covariance learning. bioRxiv, 2022

    2022

  30. [38]

    P. Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of optimization theory and applications, 109:475–494, 2001

    2001

  31. [39]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  32. [40]

    D. Walsh. Occam’s razor: A principle of intellectual elegance. American Philosophical Quarterly, 16(3):241–244, 1979

    1979

  33. [41]

    S. J. Wright. Coordinate descent algorithms. Mathematical programming, 151(1):3–34, 2015. 12 A Strict Occam Improvement This appendix provides a formal analysis demonstrating that predictive coding (PC) achieves a strictly better generalization bound—under the PAC-Bayes/MDL fr...

    2015

  34. [43]

    Many practical ReLU networks empirically satisfy KL with θ ≈ 1 2

    or even linear convergence (θ = 1 2). Many practical ReLU networks empirically satisfy KL with θ ≈ 1 2. 22 E.2 Computational-complexity perspective Global minimization of the two–part codelength C(θ) = ˆR(θ) + 1 2σ2p P l ∥θl∥2 is NP–hard in general (the MDL inference problem r...

  35. [5545]

    IJCAI/AAAI Press, July 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.