Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional propagation of values and densities

Jarek Duda

REVIEW 4 major objections 2 minor 2 cited by

Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional propagation of values and densities

T0 review · 4 major / 2 minor · reviewed 2026-05-24 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Joint distribution neurons model local densities to enable bidirectional propagation, moment-based uncertainty handling, and local training alternatives.

desk verdict This is a high-level sketch for joint-distribution neurons in a KAN extension that could support bidirectional flow and moment propagation, but the functional form is left too open to deliver on the claims without major additional work. read the letter →

arxiv 2405.05097 v9 pith:IIK37U7S submitted 2024-05-08 cs.LG stat.ML

Jarek Duda This is my paper · ORCID

classification cs.LGstat.ML

keywords jointdistributionneuronshierarchicalcorrelationreconstructionbidirectionalpropagationKolmogorov-Arnoldnetworksmomentlocaltraininginformationbottleneckneuralnetworkarchitecture

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes extending KAN-style neurons with an explicit model of the joint distribution over their inputs, written as a linear combination of basis functions over the unit hypercube. This model supports substituting observed values to recover conditional expectations or distributions for the remaining variables, propagating vectors of moments such as mean and variance, and training through direct fitting, tensor methods, or an information-bottleneck objective. A sympathetic reader would care because the construction directly targets three gaps between current artificial networks and biological ones: unidirectional flow, deterministic activation, and global back-propagation.

What carries the argument

The joint distribution representation ρ(x) = sum_{j in B} a_j f_j(x) that encodes correlations among inputs and supplies conditional values or moments on demand.

What would settle it

A controlled benchmark in which networks built from these neurons require substantially more parameters or training time than MLPs or KANs to reach the same test accuracy on a standard classification or regression task.

Watch

Extended reading notes

Core claim

Neurons containing the joint-density model ρ(x) = sum a_j f_j(x) for x in [0,1]^d allow repair of missing inputs by conditional evaluation, propagate distributions via moment vectors, and admit local training procedures including direct optimization and information-bottleneck updates, while remaining compatible with existing architectures such as transformers.

Load-bearing premise

The joint distribution model can be trained and evaluated at practical cost while preserving accuracy comparable to standard layers.

Editorial extensions

If this is right

Inputs can be repaired on the fly by solving for the conditional distribution given the observed coordinates.
Uncertainty can be propagated forward by carrying vectors of moments rather than single point estimates.
Training rules other than back-propagation become available, including direct fitting of the coefficients a_j and local information-bottleneck objectives.
The same representation can replace softmax layers in embedding models by treating learned features as mixed moments of an underlying joint density.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Such neurons could support decentralized or continual learning scenarios where only local statistics are updated.
Interpreting transformer features as moments suggests a route to uncertainty-aware attention mechanisms.
The approach opens a concrete path for testing whether explicit joint-density modeling improves robustness on tasks that reward risk sensitivity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main takeaway is that the paper outlines an idea for neurons that store a local joint density as a linear combination of basis functions, then use substitution to get conditionals and moments to carry distributions forward. This is positioned as a way to add three bio-inspired traits to current networks: bidirectional propagation, variance handling, and local training options like information bottleneck. The framing is general enough that it could also apply to embeddings or other layers. That part is clear and connects to existing work on KAN and probabilistic models without obvious circularity. What the paper does reasonably is name concrete gaps in modern architectures relative to biology and suggest how a density model might close them. The stress-test concern lands: the given form ρ(x) = sum a_j f_j(x) does not come with any choice of f_j, any product structure that would make marginalization cheap, or any procedure for normalization and training. Conditioning therefore stays formal rather than practical, especially past low dimensions, and nothing in the text shows the three properties actually appear at reasonable cost. There are also no derivations, toy calculations, or experiments to test feasibility. The paper is therefore an idea rather than a method. Readers already working on bio-plausible or probabilistic extensions of KAN might find it useful as a starting point for discussion. It shows honest engagement with the cited biology references and does not overclaim results. I would bring it to a reading group to talk through whether any basis choice could make the operations tractable. I would not cite it in its current form. It deserves peer review because the direction is worth exploring and feedback could push the authors toward concrete realizations or bounds.

Referee Report

4 major / 2 minor

Summary. The manuscript proposes joint distribution neurons as an extension of Kolmogorov-Arnold Networks, in which each neuron models a local joint distribution via the linear expansion ρ(x)=∑_{j∈B} a_j f_j(x) for x∈[0,1]^d. The central claim is that this form supplies three missing biological properties: (1) bidirectional propagation obtained by variable substitution to produce conditionals, (2) propagation of full distributions represented as moment vectors (mean, variance, …), and (3) local training routes including direct fitting, tensor decomposition, and the information bottleneck. The same construction is suggested as a drop-in replacement for softmax layers in transformers.

Significance. If the functional form could be equipped with concrete, tractable basis functions and training procedures that realize the three listed properties at scale, the work would supply a principled probabilistic primitive that unifies interpretation, uncertainty propagation, and locality of learning—potentially improving robustness and sample efficiency over standard MLPs or KANs. The absence of any such concrete realization, however, leaves the significance prospective rather than demonstrated.

major comments (4)

[Abstract] Abstract: the claim that substitution of variables directly yields conditional distributions omits the marginalization integrals required for normalization; without an explicit product or separable structure on the unspecified f_j, these integrals are intractable for d>3 and therefore load-bearing for the bidirectional-propagation claim.
[Abstract] Abstract: no choice of basis functions f_j, multi-index set B, non-negativity constraint, or normalization procedure for the coefficients a_j is supplied, rendering the three biological properties formal possibilities rather than demonstrated capabilities of the given expansion.
[Abstract] Abstract: the information-bottleneck training route is asserted to be “local and promising,” yet no algorithm, objective, or complexity bound is derived that would show how the bottleneck can be optimized using only the linear coefficients a_j and the (unspecified) f_j.
[Abstract] Abstract: the manuscript contains neither derivations, pseudocode, complexity analysis, nor any empirical result that would substantiate that the proposed neuron can be trained or evaluated at practical cost while preserving the claimed moment-propagation and conditioning properties.

minor comments (2)

[Abstract] Abstract: grammatical phrasing “a million of biological neurons” and “the proposed can predict” should be corrected.
[Abstract] Abstract: citation markers (e.g., ~cite{RL}, ~cite{axon}) appear without an accompanying reference list or context.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for the constructive critique. The manuscript is a concise conceptual proposal introducing the joint-distribution neuron form and arguing that it formally enables three biological properties. We respond point-by-point below, acknowledging where the current text is limited to the general expansion and where concrete realizations remain future work.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that substitution of variables directly yields conditional distributions omits the marginalization integrals required for normalization; without an explicit product or separable structure on the unspecified f_j, these integrals are intractable for d>3 and therefore load-bearing for the bidirectional-propagation claim.

Authors: We agree that obtaining a properly normalized conditional from the joint expansion generally requires marginalization integrals. The manuscript states that substitution yields conditionals, but does not claim this is automatic for arbitrary bases; the intent is that, once a concrete basis admitting closed-form or efficient marginals is chosen, the same linear coefficients allow both forward and backward propagation. The current text leaves the required structure on f_j implicit, which is a limitation of the presentation. revision: no
Referee: [Abstract] Abstract: no choice of basis functions f_j, multi-index set B, non-negativity constraint, or normalization procedure for the coefficients a_j is supplied, rendering the three biological properties formal possibilities rather than demonstrated capabilities of the given expansion.

Authors: The manuscript deliberately presents the most general linear expansion that still permits the three listed operations (variable substitution, moment-vector propagation, and local coefficient updates). Specific bases (e.g., multivariate polynomials or wavelets on [0,1]^d), non-negativity constraints, and normalization schemes are indeed omitted because the paper’s scope is to establish the functional form and its qualitative advantages over standard KAN neurons. Concrete instantiations are required for implementation and are noted as future work. revision: no
Referee: [Abstract] Abstract: the information-bottleneck training route is asserted to be “local and promising,” yet no algorithm, objective, or complexity bound is derived that would show how the bottleneck can be optimized using only the linear coefficients a_j and the (unspecified) f_j.

Authors: The claim is that the information-bottleneck objective can be expressed directly in terms of the coefficients a_j once the basis is fixed, because the modeled density is linear in those coefficients; this would in principle allow a local update without back-propagation through the rest of the network. No explicit algorithm or complexity analysis is supplied, as the manuscript only identifies the route as conceptually local. Deriving a practical optimizer is left for subsequent development. revision: no
Referee: [Abstract] Abstract: the manuscript contains neither derivations, pseudocode, complexity analysis, nor any empirical result that would substantiate that the proposed neuron can be trained or evaluated at practical cost while preserving the claimed moment-propagation and conditioning properties.

Authors: The manuscript is a short conceptual note whose contribution is the identification of the linear joint-density expansion and the three formal properties it enables. It therefore contains no empirical results, pseudocode, or complexity bounds. We accept that demonstrating practical cost and preservation of the properties requires concrete bases, training procedures, and experiments, none of which are present. revision: no

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: forward architectural proposal with independent claims

full rationale

The manuscript defines the joint density model ρ(x)=∑_{j∈B} a_j f_j(x) directly as an extension of KAN and then enumerates three biological properties (bidirectional repair via substitution, moment-vector propagation, and multiple local training routes) as consequences of that functional form. No step equates a claimed prediction or uniqueness result to a fitted parameter or prior self-citation; the conditioning argument is presented as a formal possibility of variable substitution without any reduction to an input equation or self-referential theorem. External citations (axon, risk, RL) are used only for motivation, not as load-bearing justification. The derivation chain therefore remains self-contained and non-circular.

Assumptions & free parameters 1 free parameters · 1 assumptions · 1 invented entities

The proposal rests on the modeling assumption that local joint distributions can be expressed in the given linear combination form and that this enables the claimed capabilities; no independent evidence for practical performance is provided.

free parameters (1)

a_j coefficients
The expansion coefficients in the joint density model ρ(x)=sum a_j f_j(x) would need to be determined during training.

assumptions (1)

domain assumption The local joint distribution can be expressed as a linear combination of basis functions f_j(x)
This is the core modeling assumption stated in the abstract for the neuron definition.

invented entities (1)

joint distribution neuron
purpose: To model local joint distributions enabling bidirectional propagation and probabilistic outputs
New neuron type introduced in the proposal.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional propagation of values and densities." pith.science (2026). https://pith.science/paper/IIK37U7S

@misc{pith2026240505097,
  author       = {Pith},
  title        = {Pith review of: Biology-inspired joint distribution neurons based on Hierarchical Correlation Reconstruction allowing for multidirectional propagation of values and densities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IIK37U7S}},
  note         = {Machine review of arXiv:2405.05097}
}

abstract

Recently a million of biological neurons (BNN) has turned out better from modern RL methods in playing Pong~\cite{RL}, reminding they are still qualitatively superior e.g. in learning, flexibility and robustness - suggesting to try to improve current artificial e.g. MLP/KAN for better agreement with biological. There is proposed extension of KAN approach to neurons containing model of local joint distribution: $\rho(\mathbf{x})=\sum_{\mathbf{j}\in B} a_\mathbf{j} f_\mathbf{j}(\mathbf{x})$ for $\mathbf{x} \in [0,1]^d$, adding interpretation and information flow control to KAN, and allowing to gradually add missing 3 basic properties of biological: 1) biological axons propagate in both directions~\cite{axon}, while current artificial are focused on unidirectional propagation - joint distribution neurons can repair by substituting some variables to get conditional values/distributions for the remaining. 2) Animals show risk avoidance~\cite{risk} requiring to process variance, and generally real world rather needs probabilistic models - the proposed can predict and propagate also distributions as vectors of moments: (expected value, variance) or higher. 3) biological neurons require local training, and beside backpropagation, the proposed allows many additional ways, like direct training, through tensor decomposition, or finally local and promising: information bottleneck. Proposed approach is very general, can be also used as extension of softmax in embeddings of e.g. transformer, JEPA, Mamba, suggesting interpretation that features are mixed moments of joint density of real-world properties.

Figures

Figures reproduced from arXiv: 2405.05097 by the authors.

**Figure 1.** uni-directional vs bi-directional propagation of biological axons [2], working only on values vs also on distributions - observed in animals e.g. as risk avoidance [3], and finally BNNs need local training approaches like looking the most promising: information bottleneck ([5], [8], [9]). 1https://www.newscientist.com/article/2517389-human-brain-cells-on-a-chiplearned-to-play-doom-in-a-week/ [PITH_FULL_IMAGE:figur… view at source ↗

**Figure 2.** Basic formulas and example for d = 2 variables HCR neuron, using convenient variable normalization to nearly uniform in [0, 1]. Neuron contains matrix of moments: aij (generally order d tensor), allowing to propagate in various directions by substituting some variables and normalizing to get estimated conditional density for the remaining - just permuting indexes to change propagation direction. For value propagatio… view at source ↗

**Figure 3.** Summary of differences between artificial (ANN) and bi [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

Figures from the paper (10 more)

**Figure 5.** Figure 5: The proposed HCR neuron and neural network (HCRN, HCRNN) [PITH_FULL_IMAGE:figures/full_fig_p002_5.png]

**Figure 4.** Figure 4: Simple 2/3D examples from HCRNN Wolfram notebook of propagation [PITH_FULL_IMAGE:figures/full_fig_p002_4.png]

**Figure 6.** Figure 6: 2D example comparison of local basis KDE (kernel density estimation) vs global basis HCR (available code in HCR Wolfram notebook) modelling joint density for dataset as points shown on the right. Assuming ρ = 1 trivial joint density, we would get 0 log-likelihood evalu…

**Figure 7.** Figure 7: Top: Visualized part of HCR polynomial [0, 1] basis in d = 1 dimension and fj(x) = Qd i=1 fji (xi) product bases for d = 2, 3. E.g. for d = 3 the assumed joint density becomes ρ(x, y, z) = P ijk aijkfi(x)fj (y)fk(z). As f0 = 1, zero index in aijk means independence fro…

**Figure 8.** Figure 8: KAN-like example with code from HCRNN Wolfram notebook: trying [PITH_FULL_IMAGE:figures/full_fig_p004_8.png]

**Figure 10.** Figure 10: Example of direct prediction of conditional distribution from [22] [PITH_FULL_IMAGE:figures/full_fig_p005_10.png]

**Figure 12.** Figure 12: Independence test HSIC vs HCR comparison from [25] - we independently generate 2 data samples from bimodal distribution and introduce dependence by rotating it 0, 1, 2, 3, 4, 5 degrees (top). In HCR we model their joint distribution as polynomial - there are shown suc…

**Figure 14.** Figure 14: Top: even not having ground truth for properties, we might be able to enforce network to work on their probability densities e.g. through replacement of softmax, un-embedding. There is now mainly used softmax for values (top row). HCR approach allows to replace one or…

**Figure 13.** Figure 13: Top: embeddings are the basic tools of modern neural networks like transformers, representing various objects e.g. words as vectors, of parameters hopefully corresponding to real properties, like age. However, e.g. word ”adult” represents much larger age variance than…

**Figure 15.** Figure 15: Omnidirectional HCR neuron proposed in [10] - getting any [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Robustness and Invariance of Hybrid Metaheuristics under Objective Function Transformations
cs.NE 2025-09 reject novelty 4.0 of 10

A CEC-2017 benchmark study claims DE-based hybrid optimizers remain robust under translation, scaling, rotation, and additive shifts, but the reported data are internally inconsistent and contain duplicated rows.
Improving KAN with CDF normalization to quantiles
cs.LG 2025-07 conditional novelty 4.0 of 10

CDF normalization to quantiles improves MNIST test accuracy and training speed of Legendre-KAN compared with min-max scaling.

Pith tools

Reviewed May 24, 2026 · model on record in the stance chip above.