REVIEW 3 major objections 5 minor 16 references
Training Neural Networks by Optimizing Neuron Positions
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Training networks by moving neuron positions instead of learning weights cuts parameters from $O(n^2)$ to $O(n)$ and keeps MNIST accuracy past 80% pruning
desk verdict A genuinely new weight-parameterization idea with honest reporting, but the paper's central 'competitive performance' claim is undermined by a parameter-matched baseline that matches counts while badly mismatching capacity. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the wiring rule of Eq. (1), $w_{ij}=1/\|p_i-p_j\|_2$, combined with a fixed per-layer z-coordinate and a learnable continuous inhibition mask that assigns each neuron an excitatory or inhibitory sign. Because every weight in a fully connected layer is a function of only the two endpoint positions, backpropagation moves points instead of scalars; a layer with $n_l$ neurons is stored with $O(n_l)$ position parameters rather than $O(n_l n_{l+1})$ weights. A single step of backpropagation therefore updates all connections incident to a neuron at once, which is the parameter saving and also the main source of weight interdependence.
What would settle it
Re-run the pruning experiments with a baseline that has the same 2,048 hidden neurons as the 3D MLP but whose per-connection weights are free and whose total trainable parameter count is forced to about 10,500, for instance by weight sharing across connections; if this width-matched free-weight baseline matches or outperforms the 3D MLP at every pruning level, the inverse-distance wiring rule is not what produces the observed pruning robustness.
Extended reading notes
Core claim
The central discovery is that a distance-geometry wiring rule can carry most of the learning burden: instead of storing and updating an $O(n^2)$ weight matrix, the network stores $d-1$ coordinates per neuron plus a bias and an inhibition sign, computes weights as $w_{ij}=1/\|p_i-p_j\|_2$, and learns by moving points. On MNIST, spatially embedded MLPs and SNNs reach competitive accuracy, and both retain accuracy under magnitude-based pruning above 80% sparsity; at 95% post-training pruning the 3D MLP even beats a wide baseline with far more parameters. The same experiments show that relaxing the layer structure and raising embedding dimensionality improve accuracy, and that dense MLP accuracy stays below conventional MLPs, which the authors explain by the restricted set of weight configurations reachable through positions.
Load-bearing premise
The evaluation assumes that matching a conventional network only by total parameter count is a fair comparison: the 3D MLP has 2,048 hidden neurons and about 1.63 million effective connections but only 10,576 trainable parameters, while the parameter-matched MLP Baseline 14 has 14 hidden neurons and about 11,116 weights; if this premise is wrong, the claims of competitive performance and pruning superiority are not established.
Editorial extensions
If this is right
- A fully connected network can be deployed with weights reconstructed from stored positions, so connection memory falls from $O(n^2)$ to $O(n)$, a direct benefit for edge-device and real-time inference.
- Magnitude-based pruning has a simple geometric reading: removing the smallest weights is the same as cutting the longest physical connections, so pruning can be specified in space rather than in weight space.
- The framework composes with other compression techniques, such as quantization and pruning, and can be dropped into existing MLP and SNN architectures without changing the forward-pass structure.
- Spiking versions outperform their parameter-matched conventional baseline, indicating that temporal spike timing can compensate for reduced weight flexibility; the same may hold for other temporally coded or recurrent models.
- Allowing richer geometry such as learned z-coordinates and higher-dimensional embedding spaces monotonically improves MNIST accuracy, so the geometry itself is a tunable resource for trading parameters against expressiveness.
Reading between the lines
- An editorial extension: the inverse-distance rule is one instance of a broader prior, monotone weight decay with spatial separation; testing other monotone parameterizations such as Gaussian or exponential kernels of distance would show whether the pruning robustness comes from the geometric prior or from the specific $1/d$ form.
- An editorial extension: the overlap of learned z-distributions across layers suggests the rigid layer code is not essential, so a fully unconstrained position embedding plus a routing mechanism might improve accuracy further, at the cost of feedforward ordering.
- An editorial extension: because each position update changes many weights, the effective number of independent degrees of freedom in a spatially embedded layer is far below its raw parameter count; measuring the rank of the Jacobian of the weight matrix with respect to positions would quantify that gap directly.
- An editorial extension: the same position-to-weight map transfers naturally to graph neural networks, where node coordinates could be learned jointly with edge weights and the inverse-distance rule would supply an inductive bias aligned with spatially structured graphs.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a neural architecture in which each neuron is assigned a position in Euclidean space and connection weights are computed as the inverse of the Euclidean distance between connected neurons. Instead of learning individual weights, the model optimizes neuron positions (plus a small number of auxiliary parameters such as layer distances and inhibition values). The authors argue that this reduces the parameter complexity from O(n^2) to O(n) and provides a biologically motivated inductive bias. They validate the approach on MNIST for both MLPs and fully connected SNNs, comparing against conventional baselines matched either by neuron count or by parameter count. They also report pruning experiments and an extension in which z-coordinates are learned, as well as embeddings in higher-dimensional spaces. The main claim is that spatially embedded networks are competitive with conventional architectures and maintain accuracy at high pruning rates.
Significance. The core idea is interesting and potentially useful: if a weight matrix can be replaced by a distance-dependent wiring rule, then a compact parameterization with a clear geometric interpretation becomes possible. The experiments are transparent, report multiple random seeds, and use an external benchmark, which is a strength. However, the current evidence does not establish the central claim of competitive performance: the paper's own MLP results consistently show the spatial models underperforming conventional baselines with comparable parameter counts but far fewer neurons, and the pruning advantage rests on comparisons that confound total parameter count with the number of effective connections. The significance of the work therefore depends on whether the claims can be appropriately reframed and whether the pruning robustness can be demonstrated under genuinely matched conditions.
major comments (3)
- [Abstract and Section 3.1, Table 1, Fig. 5] The abstract claims that spatially embedded networks 'achieve a performance competitive with conventional architectures,' but the paper's own experiments contradict this for every MLP configuration tested. In Table 1, the 3D MLP reaches 0.9217 accuracy, well below MLP Baseline 14 with 0.9429 and MLP Baseline 2048 with 0.9745. Table 3 shows the Relaxed 3D MLP at 0.9337 versus MLP Baseline 17 at 0.9507, and Fig. 5 shows every higher-dimensional spatial MLP below its corresponding parameter-matched baseline. Section 4 explicitly concedes that spatially embedded MLPs 'were consistently outperformed by conventional MLPs.' The only MLP result that favors the spatial model is the single 95%-pruning point in Fig. 3a. The central empirical claim should either be substantially weakened or supported by additional evidence beyond this one point.
- [Section 3.2, Fig. 3 and Table 1] The pruning comparison conflates parameter count with capacity and with the number of effective connections. The 3D MLP has 2,048 hidden neurons and 1,626,112 weights, while MLP Baseline 14 has 14 hidden neurons and 11,116 weights. When 80% of the 3D MLP's weights are pruned, roughly 325,000 weights remain, which is about 30 times the baseline's entire network. The abstract's phrase 'outperforming traditional networks with the same number of parameters under similar conditions' is therefore not an accurate description of what Fig. 3 shows. The width-matched control, MLP Baseline 2048, outperforms the 3D MLP at most pruning levels. A fair assessment of pruning robustness would need to compare networks with matched numbers of remaining connections, or otherwise control for the large difference in network width.
- [Section 2, Eq. (1)] Equation (1) defines the synaptic weight as the inverse of the Euclidean distance between connected neurons, but this quantity is undefined when the distance is zero. The paper does not specify how neuron positions are initialized, or what constraint, regularization, or projection step prevents two connected neurons from occupying the same position during optimization. Since positions are updated by gradient descent, a collision is not only possible but also not addressed by any stated mechanism. The model definition is therefore incomplete without a minimum-distance safeguard or an explicit justification for why collisions cannot occur.
minor comments (5)
- [Abstract and Section 4] The abstract says spatially embedded models 'outperform traditional networks with the same number of parameters under similar conditions,' but Section 4 states the opposite for MLPs. This inconsistency should be resolved by aligning the abstract with the actual results.
- [Section 2, Eqs. (3) and (4)] The parameter count in Eq. (4) includes positions for the input layer, but the text does not explain why input neurons need learnable positions. Clarifying the role of input positions would make the parameter-counting convention easier to interpret.
- [Section 3.1, Tables 1 and 2] The learning rates differ between the spatial models (0.005) and the baselines (0.001). Since no learning-rate sweep is reported, it is unclear whether the performance gap could be reduced by more careful hyperparameter tuning of the spatial models.
- [Section 3.2] The paper describes pruning the longest connections as mimicking the brain's tendency to minimize wiring length, but removing long connections is not the same as actively minimizing total wiring length. The biological analogy would be more accurate if framed as a heuristic consistent with wiring minimization.
- [Section 5] The conclusion mentions 'Recurrent Neural Networks' as a possible extension in the introduction, but the experiments only cover MLPs and SNNs. The paper would benefit from stating this scope limitation explicitly in the relevant sections.
Circularity Check
No circular derivation chain found; all results are empirical comparisons on an external benchmark.
full rationale
This paper does not derive a predicted constant or falsifiable quantity from fitted inputs; rather, it defines a parameterization and reports measured accuracies on MNIST. Equation (1) defines each synaptic weight as the inverse Euclidean distance between neuron positions, and Equation (4) counts position coordinates, biases, and inhibition values as parameters. These are definitions and parameter-counting identities, not predictions fitted to data. The experimental comparisons use an external benchmark and baselines constructed by the authors, which raises a question of evaluation fairness but does not make any claim equivalent to its own inputs by construction. Self-citations to Boccato et al. appear only as general related work on network topologies and are not load-bearing for the proposed method. No quoted passage exhibits a reduction of a claimed result to a fitted parameter or to a self-cited uniqueness theorem. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- Per-neuron spatial coordinates (x,y) in each layer =
learned from data; values not reported
- Learnable layer distances =
two distances for the 3-layer 3D MLP and 3D SNN; not individually reported
- Per-neuron inhibition values =
learned continuous values; not reported
- Learnable z-coordinates in the Relaxed 3D MLP =
learned per neuron; not reported
assumptions (5)
- domain assumption Synaptic weight between two neurons is exactly the inverse Euclidean distance between their positions (Eq. 1).
- domain assumption Neurons are organized into layers by fixing the z-coordinate to the layer index, preserving feedforward connectivity.
- domain assumption The sign of a neuron's contribution is controlled by a continuous inhibition value passed through a steep sigmoid; positions alone cannot produce negative weights.
- ad hoc to paper No two connected neurons occupy the same position, since Eq. (1) diverges at zero distance; the paper does not specify an initialization or regularization that guarantees this.
- domain assumption MNIST on three-layer networks is a sufficient probe of the architecture's general properties.
Cite this review
Pith. "Pith review of Training Neural Networks by Optimizing Neuron Positions." pith.science (2026). https://pith.science/paper/4XLP7H4V
@misc{pith2026250613410,
author = {Pith},
title = {Pith review of: Training Neural Networks by Optimizing Neuron Positions},
year = {2026},
howpublished = {\url{https://pith.science/paper/4XLP7H4V}},
note = {Machine review of arXiv:2506.13410}
}
read the original abstract
The high computational complexity and increasing parameter counts of deep neural networks pose significant challenges for deployment in resource-constrained environments, such as edge devices or real-time systems. To address this, we propose a parameter-efficient neural architecture where neurons are embedded in Euclidean space. During training, their positions are optimized and synaptic weights are determined as the inverse of the spatial distance between connected neurons. These distance-dependent wiring rules replace traditional learnable weight matrices and significantly reduce the number of parameters while introducing a biologically inspired inductive bias: connection strength decreases with spatial distance, reflecting the brain's embedding in three-dimensional space where connections tend to minimize wiring length. We validate this approach for both multi-layer perceptrons and spiking neural networks. Through a series of experiments, we demonstrate that these spatially embedded neural networks achieve a performance competitive with conventional architectures on the MNIST dataset. Additionally, the models maintain performance even at pruning rates exceeding 80% sparsity, outperforming traditional networks with the same number of parameters under similar conditions. Finally, the spatial embedding framework offers an intuitive visualization of the network structure.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Nature Machine Intelligence 5(12), 1369–1381 (2023)
Achterberg, J., Akarca, D., Strouse, D.J., Duncan, J., Astle, D.E.: Spatially em- bedded recurrent neural networks reveal widespread links between structural and functional neuroscience findings. Nature Machine Intelligence 5(12), 1369–1381 (2023). https://doi.org/10.1038/s42256-023-00748-9 12 Laura Erb et al
-
[2]
Nature Communications14(1) (2023)
Barabási, D.L., Beynon, T., Katona, A., Perez-Nieves, N.: Complex computation from developmental priors. Nature Communications14(1) (2023). https://doi. org/10.1038/s41467-023-37980-1
-
[3]
Nature Communications15(1), 364 (2024)
Barabási, D.L., Schuhknecht, G.F.P., Engert, F.: Functional neuronal circuits emerge in the absence of developmental activity. Nature Communications15(1), 364 (2024). https://doi.org/10.1038/s41467-023-44681-2
-
[4]
Nature Communications 13 (2022)
Bartolozzi, C., Indiveri, G., Donati, E.: Embodied neuromorphic intel- ligence. Nature Communications 13 (2022). https://doi.org/10.1038/ s41467-022-28487-2
work page 2022
-
[5]
Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gulcehre, C., Song, F., Ballard, A., Gilmer, J., Dahl, G., Vaswani, A., Allen, K., Nash, C., Langston, V., Dyer, C., Heess, N., Wierstra, D., Kohli, P., Botvinick, M., Vinyals, O., Li, Y., Pascanu, R.: Relati...
work page 2018
-
[6]
Proceedings of the National Academy of Sciences 119(3), e2112566119 (2022)
Blauch, N.M., Behrmann, M., Plaut, D.C.: A connectivity-constrained computa- tional account of topographic organization in primate high-level visual cortex. Proceedings of the National Academy of Sciences 119(3), e2112566119 (2022). https://doi.org/10.1073/pnas.2112566119
-
[7]
Neurocomput- ing 568, 127058 (2024).https://doi.org/10.1016/j.neucom.2023.127058
Boccato, T., Ferrante, M., Duggento, A., Toschi, N.: 4ward: A relayering strategy for efficient training of arbitrarily complex directed acyclic graphs. Neurocomput- ing 568, 127058 (2024).https://doi.org/10.1016/j.neucom.2023.127058
arXiv 2024
-
[8]
Neural Networks171, 215–228 (2024)
Boccato, T., Ferrante, M., Duggento, A., Toschi, N.: Beyond multilayer percep- trons: Investigating complex topologies in neural networks. Neural Networks171, 215–228 (2024). https://doi.org/10.1016/j.neunet.2023.12.012
Show all 16 references
-
[9]
The Journal of Neuroscience 14(4) (1994)
Cherniak, C.: Component placement optimization in the brain. The Journal of Neuroscience 14(4) (1994). https://doi.org/10.1523/JNEUROSCI.14-04-02418. 1994
1994 doi
-
[10]
Proceedings of the IEEE111(9), 1016–1054 (2023)
Eshraghian, J.K., Ward, M., Neftci, E.O., Wang, X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D.S., Lu, W.D.: Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE111(9), 1016–1054 (2023). https://doi.org/10.1109/JPROC.2023.3308088
2023
-
[11]
Journal of Cognitive Neuroscience 4(4), 323–336 (1992)
Jacobs, R.A., Jordan, M.I.: Computational consequences of a bias toward short connections. Journal of Cognitive Neuroscience 4(4), 323–336 (1992). https:// doi.org/10.1162/jocn.1992.4.4.323
1992 doi
-
[12]
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2017)
2017
-
[13]
Proceedings of the IEEE86(11), 2278–2323 (1998).https: //doi.org/10.1109/5.726791
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE86(11), 2278–2323 (1998).https: //doi.org/10.1109/5.726791
1998 doi
-
[14]
In: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services
Liu,S.,Lin,Y.,Zhou,Z.,Nan,K.,Liu,H.,Du,J.:On-demanddeepmodelcompres- sion for mobile devices: A usage-driven model selection framework. In: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. p. 389–400 (2018).https://doi.org/...
2018
-
[15]
Proceedings of the National Academy of Sciences121(38) (2024)
Shuvaev, S., Lachi, D., Koulakov, A., Zador, A.: Encoding innate ability through a genomic bottleneck. Proceedings of the National Academy of Sciences121(38) (2024). https://doi.org/10.1073/pnas.2409160121
2024 doi
-
[16]
Trends in Cognitive Sciences22(12), 1127–1142 (2018)
Stiso,J.,Bassett,D.S.:Spatialembeddingimposesconstraintsonneuronalnetwork architectures. Trends in Cognitive Sciences22(12), 1127–1142 (2018). https:// doi.org/10.1016/j.tics.2018.09.007
2018 doi
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.