HalfNet: Randomized Neural Networks with Learned Subspace Geometry

Ethem Alpaydin

REVIEW 3 major objections 2 minor 24 references

HalfNet: Randomized Neural Networks with Learned Subspace Geometry

T0 review · 3 major / 2 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HalfNet draws random weights from a learned low-rank covariance and matches fully trained networks with far fewer parameters.

desk verdict HalfNet learns low-rank covariance for random weights to match full MLPs on MNIST/CIFAR with fewer trained params, but the geometry claim risks being circular with the supervised fitting of Σ. read the letter →

arxiv 2606.04583 v1 pith:GK66SS4M submitted 2026-06-03 cs.LG

Ethem Alpaydin This is my paper · ORCID

classification cs.LG

keywords randomizedneuralnetworkslow-rankfactorizationsubspacegeometryrandomembeddingMNISTCIFAR-10metriclearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that random weights drawn from a normal distribution whose covariance matrix has a low-rank factorization learned from data can recover the performance of standard multilayer perceptrons. Experiments on MNIST and CIFAR-10 show comparable accuracy while training only a small head instead of all weights. Spectral analysis supports the claim that geometry of the weight space, rather than precise individual values, carries most of the predictive power. Accuracy increases smoothly as the rank of the factorization grows. The construction is presented as a data-dependent random embedding interpretable also as supervised metric learning or random features.

What carries the argument

The low-rank factorization of the covariance Σ that defines the distribution from which random weights are drawn, producing a data-dependent random embedding.

What would settle it

An experiment on a new dataset or architecture in which HalfNet with the learned Σ fails to reach the accuracy of a fully trained network of comparable size would falsify the claim that the learned geometry captures the essential structure.

Watch

Extended reading notes

Core claim

HalfNet draws random weights from N(0, Σ) where Σ has a learned low-rank factorization. This yields networks that match the accuracy of fully trained multilayer perceptrons on MNIST and CIFAR-10 while using substantially fewer parameters. Spectral analysis indicates that predictive power resides primarily in the geometry of the weight space rather than in the exact values of individual parameters, and accuracy scales smoothly with rank.

Load-bearing premise

That a low-rank factorization of Σ learned from data produces a random embedding whose fixed weights, when combined with a small trainable head, recover full network performance without the learned geometry being an artifact of the training procedure or dataset choice.

Editorial extensions

If this is right

Accuracy scales smoothly with the rank of the learned factorization.
The method can be viewed through supervised metric learning or random-feature and kernel perspectives.
Much of the predictive power of neural networks lies in the geometry of their weight space rather than precise parameter values.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The approach could be tested on sequence or graph tasks to check whether the same learned-geometry principle transfers beyond image data.
If the low-rank Σ can be estimated once and reused across related tasks, it would imply a form of transferable random embedding.
The smooth rank-accuracy curve suggests an inexpensive way to trade parameter count for performance without retraining the entire network.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main takeaway is that this paper takes randomized-weight networks and learns a low-rank factorization of the covariance Σ from data so that fixed random weights plus a small trainable head can reach the accuracy of a fully trained MLP. Experiments are reported on MNIST and CIFAR-10, with an added spectral analysis and the note that accuracy scales with rank.

What is actually new is the explicit data-driven low-rank structure on the sampling distribution rather than using a fixed isotropic Gaussian. The multiple framings (random features, supervised metric learning, kernel view) are a reasonable way to situate the method. The rank-scaling observation is concrete and could be practically useful.

The paper does the basic job of showing the idea on standard small datasets. If the full text supplies the exact learning procedure for the factors, the baselines used, and the quantitative gaps to fully trained models, that would be the useful part.

The soft spot is the evidence for the stronger claim that geometry carries the predictive power. Because Σ is fit directly on labeled data, the learned directions could simply be task-specific rather than revealing something general about neural weight spaces. The stress-test concern about circularity holds up on the abstract: the spectral analysis does not yet show that the geometry is independent of the supervised procedure that produced it. No error bars or exclusion details are mentioned either.

This is for people working on parameter-efficient or randomized networks. A reader already following that literature might extract the rank-scaling result or the embedding perspective.

It deserves a serious referee because the proposal is straightforward to test and the datasets are accessible, even if the analysis section will need tightening.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes HalfNet, in which a subset of network weights are drawn randomly from N(0, Σ) where Σ admits a learned low-rank factorization obtained from the labeled training data. Experiments on MNIST and CIFAR-10 are reported to show that the resulting model matches the accuracy of fully trained MLPs while using substantially fewer trainable parameters. Spectral analysis is presented to argue that predictive power resides primarily in the geometry of the weight-space distribution rather than in the precise values of individual weights, with accuracy observed to scale smoothly with the rank of the factorization. The construction is also interpreted as a data-dependent random embedding, a supervised metric-learning procedure, or a random-feature model.

Significance. If the central empirical and spectral claims hold after the circularity concern is addressed, the work would provide concrete evidence that a learned low-dimensional subspace geometry can substitute for full weight optimization in MLPs. This would strengthen the random-feature and kernel perspectives on neural networks and offer a practical route to parameter-efficient training. The smooth rank-accuracy scaling and the explicit low-rank factorization are potentially falsifiable predictions that could be tested on additional architectures and datasets.

major comments (3)

[§3] §3 (learning procedure for Σ): the low-rank factors of Σ are obtained by supervised optimization on the target labeled data. This procedure can embed task-specific directions into the geometry, so the subsequent claim that 'much of the predictive power lies in the geometry rather than the precise values' risks circularity; the geometry is co-optimized with the embedding. A control experiment that learns Σ without labels (or from a disjoint task) and then freezes it is required to separate intrinsic geometry from supervised artifact.
[Experiments] Experiments section and Table 1/2: the manuscript states that HalfNet 'matches' fully trained MLPs on MNIST and CIFAR-10, yet supplies no numerical accuracies, standard deviations across runs, baseline architectures with matched parameter counts, or exclusion criteria. Without these quantities the scaling-with-rank observation cannot be assessed for statistical reliability or compared to the reader's weakest assumption.
[§4] Spectral analysis (presumably §4): the claim that geometry dominates is supported only by post-hoc analysis of the learned Σ. Because Σ is itself the output of the same supervised procedure, the spectral decomposition may simply recover the directions already favored by gradient descent rather than revealing an independent property of neural weight spaces.

minor comments (2)

[Abstract] Abstract: the phrase 'substantially fewer parameters' is used without a concrete ratio or reference to the corresponding table/figure.
[§2] Notation: the precise definition of the low-rank factorization (e.g., Σ = AAᵀ with A ∈ ℝ^{d×r}) and how the random weights are sampled and frozen should be stated once in a single equation block for reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below and indicate where revisions to the manuscript are planned.

read point-by-point responses

Referee: [§3] §3 (learning procedure for Σ): the low-rank factors of Σ are obtained by supervised optimization on the target labeled data. This procedure can embed task-specific directions into the geometry, so the subsequent claim that 'much of the predictive power lies in the geometry rather than the precise values' risks circularity; the geometry is co-optimized with the embedding. A control experiment that learns Σ without labels (or from a disjoint task) and then freezes it is required to separate intrinsic geometry from supervised artifact.

Authors: We acknowledge the validity of the circularity concern. The supervised optimization of the low-rank factors of Σ does incorporate task-specific information. To separate the contribution of the learned geometry from supervised artifacts, we will add a control experiment in the revised manuscript in which Σ is learned either from unlabeled data statistics or from a disjoint task and then held fixed while sampling the random weights for HalfNet. This addition will directly test whether the geometry carries predictive power independent of the specific supervised optimization used in the main experiments. revision: yes
Referee: [Experiments] Experiments section and Table 1/2: the manuscript states that HalfNet 'matches' fully trained MLPs on MNIST and CIFAR-10, yet supplies no numerical accuracies, standard deviations across runs, baseline architectures with matched parameter counts, or exclusion criteria. Without these quantities the scaling-with-rank observation cannot be assessed for statistical reliability or compared to the reader's weakest assumption.

Authors: We agree that the experimental reporting requires greater detail for proper evaluation. Although the manuscript contains the underlying results, the presentation in the Experiments section and Tables 1/2 will be revised to include explicit mean accuracies, standard deviations computed over multiple independent runs, additional baseline models whose total parameter counts are matched to those of the HalfNet variants, and any relevant exclusion criteria. These changes will allow readers to assess the statistical reliability of the rank-accuracy scaling. revision: yes
Referee: [§4] Spectral analysis (presumably §4): the claim that geometry dominates is supported only by post-hoc analysis of the learned Σ. Because Σ is itself the output of the same supervised procedure, the spectral decomposition may simply recover the directions already favored by gradient descent rather than revealing an independent property of neural weight spaces.

Authors: The spectral analysis is indeed performed after Σ has been learned via the supervised procedure. Nevertheless, the central empirical support for the geometry claim is the observed smooth scaling of accuracy with the rank of the factorization, which holds even though only the subspace geometry (not the precise weight values) is retained. We will expand the discussion in §4 to clarify this distinction and to note the limitation that the recovered directions may align with those favored by gradient descent. We will also add a brief comparison against low-rank factorizations initialized without supervision to provide further context, though the primary scaling result remains unchanged. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: method is explicitly data-driven with empirical validation

full rationale

The paper proposes learning a low-rank factorization of Σ directly from data to define the distribution for random weights, then evaluates the resulting HalfNet empirically on MNIST and CIFAR-10. No derivation, equation, or claim reduces a performance result to a fitted quantity by construction, nor does any load-bearing step rely on self-citation or imported uniqueness. The spectral analysis and scaling observations are presented as post-hoc experimental findings rather than tautological consequences of the fitting procedure itself. The approach is framed as a data-dependent random embedding interpretable via metric learning or kernels, making the central claims self-contained against external benchmarks.

Assumptions & free parameters 1 free parameters · 1 assumptions · 0 invented entities

The method rests on the domain assumption that geometry of the random distribution is the dominant source of predictive power; the rank of the factorization is an implicit free parameter whose value is not specified in the abstract.

free parameters (1)

rank of low-rank factorization
Determines the dimensionality of the learned subspace and must be chosen or tuned.

assumptions (1)

domain assumption The geometry of the weight distribution, rather than precise parameter values, carries most predictive power
Invoked to interpret the spectral analysis and performance results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HalfNet: Randomized Neural Networks with Learned Subspace Geometry." pith.science (2026). https://pith.science/paper/GK66SS4M

@misc{pith2026260604583,
  author       = {Pith},
  title        = {Pith review of: HalfNet: Randomized Neural Networks with Learned Subspace Geometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GK66SS4M}},
  note         = {Machine review of arXiv:2606.04583}
}

abstract

Many researchers investigated neural networks with some of their weights fixed to values randomly drawn from a given distribution, e.g., $N(0, I)$. Our proposed HalfNet draws random weights from $N(0, \Sigma)$, where $\Sigma$, which defines the geometry of the distribution, has a low-rank factorization that we learn from data. Experiments on MNIST and CIFAR-10 demonstrate that HalfNet can match the performance of fully trained multilayer perceptrons while using substantially fewer parameters. Spectral analysis indicates that much of the predictive power of neural networks lies in the geometry of their weight space rather than in the precise values of individual parameters, and we observe that accuracy scales smoothly with rank. HalfNet is not a neural architecture trick for low-rank structure; it implements a data-dependent random embedding that can also be interpreted through supervised metric learning, or random-feature and kernel perspectives.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 2 canonical work pages

[1]

This leads to a simpler model and a simpler optimization problem

INTRODUCTION As an alternative to the usual approach of neural networks with all their weights trained on data, one can fix some of the weights to random values [1, 2, 3, 4, 5, 6]. This leads to a simpler model and a simpler optimization problem. The Johnson-Lindenstrauss lemma states that a high-dimensional space can be embedded into a space of much lowe...
[2]

We introduce HalfNet, which learns the covariance structure of random projections rather than the individ- ual weights
[3]

Accuracy scales smoothly with rank, indicating that much of the predictive power of trained layers is cap- tured by the low-dimensional geometric structure
[4]

We show that the framework can be extended to (i) mul- tiple layers, (ii) convolutional front ends, and (ii) binary weights
[5]

This paper is organized as follows: We present our model in Section 2 and our experimental results on MNIST in Sec- tion 3

HalfNet admits probabilistic, metric-learning, and ker- nel interpretations. This paper is organized as follows: We present our model in Section 2 and our experimental results on MNIST in Sec- tion 3. In Section 4, we show how our proposed model can also be adapted to learn binary weights. We give our experi- mental results on CIFAR-10 in Section 5. Secti...
[6]

MODEL FORMULA TION We have hidden unithwithx, w h ∈R d andy h, wh0 ∈R yh =g(w ⊤ h x+w h0)(1) where typically,w h, wh0, are trained on a dataset. The weights may be drawn randomly, then kept fixed and not updated during learning: yh =g(r ⊤ h x), r h ∼ N d(0, I).(2) arXiv:2606.04583v1 [cs.LG] 3 Jun 2026 With HalfNet, our aim is to learn the distribution ofr...

work page Pith review arXiv 2026
[7]

We use ReLU as the activation function at the hidden units and softmax at the output; training is done to minimize the cross-entropy using Adam

EXPERIMENTS ON MNIST We use the MNIST dataset which has28×28 = 784in- puts and ten classes, with 60,000 training and 10,000 test in- stances. We use ReLU as the activation function at the hidden units and softmax at the output; training is done to minimize the cross-entropy using Adam. The weights to the outputs are trained as usual; only the hidden units...
[8]

We can generate binary hyperplanes by placing the sign function after Equation 3

HALF LA YER NETWORKS WITH BINARY WEIGHTS The model that we discuss above can be adapted to gener- ate binary (±1) weights, which are attractive because they can lead to major savings in terms of memory and computa- tion. We can generate binary hyperplanes by placing the sign function after Equation 3. This allows us to mapN k(0, I)to ad-dimensional multiv...

Show all 24 references

[9]

CIFAR-10 consists of 60,000 color images of size32×32belonging to ten classes, with 50,000 training and 10,000 test instances

EXPERIMENTS ON CIFAR-10 To evaluate the applicability of HalfNet beyond relatively simple grayscale datasets such as MNIST, we conduct ad- ditional experiments on the CIFAR-10 dataset. CIFAR-10 consists of 60,000 color images of size32×32belonging to ten classes, with 50,000 t...
[10]

Empirical and theoretical stud- ies have shown that deep networks often learn representa- tions that are effectively low-dimensional [7, 8]

RELA TIONSHIP TO LOW-RANK MODELS, METRIC LEARNING, AND KERNELS A large body of work has explored low-rank structure in neu- ral networks, both as an implicit bias and as an explicit pa- rameter reduction technique. Empirical and theoretical stud- ies have shown that deep netwo...
[11]

The rank of the co- variance factor provides a direct control over model capacity, yielding a smooth trade-off between fixed random features and fully trainable neural networks

CONCLUSIONS We introduced HalfNet, a stochastic parameterization in which random weights are drawn from a distribution whose covariance structure is learned from data. The rank of the co- variance factor provides a direct control over model capacity, yielding a smooth trade-of...
[12]

Random features for large- scale kernel machines,

A. Rahimi and B. Recht, “Random features for large- scale kernel machines,” inAdvances in Neural Informa- tion Processing Systems, J. Platt, D. Koller, Y . Singer, and S. Roweis, Eds. (NIPS), 2007, Curran Associates, Inc

2007
[13]

Extreme learning machine for regression and multiclass classifi- cation,

G. Huang, H. Zhou, X. Ding, and R. Zhang, “Extreme learning machine for regression and multiclass classifi- cation,”IEEE Transactions on Systems, Man, and Cy- bernetics, vol. 42, pp. 513–529, 2012

2012
[14]

A review on neural networks with random weights,

W. Cao, X. Wang, Z. Ming, and J. Gao, “A review on neural networks with random weights,”Neurocomput- ing, vol. 275, pp. 278–287, 2017

2017
[15]

Randomness in neural net- works: An overview,

S. Scardapane and D Wang, “Randomness in neural net- works: An overview,”WIREs Data Mining Knowledge Discovery, vol. 7:, 2017

2017
[16]

Deep randomized neural networks,

C. Gallicchio and S. Scardapane, “Deep randomized neural networks,” inRecent Trends in Learning From Data, L. Oneto, N. Navarin, A. Sperduti, and D. An- guita, Eds., pp. 43–68. Springer, 2020

2020
[17]

Half-layered neural networks,

E. Alpaydın, “Half-layered neural networks,”arXiv, vol. 2506.04352, 2025

2025
[18]

The low-rank simplicity bias in deep networks,

Minyoung Huh, Hossein Mobahi, and Richard Zhang, “The low-rank simplicity bias in deep networks,” in ICML, 2021

2021
[19]

Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks,

Samet Oymak and Mahdi Soltanolkotabi, “Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks,”IEEE Transactions on Information Theory, 2019

2019
[20]

Pre- dicting parameters in deep learning,

Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas, “Pre- dicting parameters in deep learning,” inNeurIPS, 2013

2013
[21]

Speeding up convolutional neural networks with low rank expansions,

Max Jaderberg, Andrea Vedaldi, and Andrew Zisser- man, “Speeding up convolutional neural networks with low rank expansions,” inBMVC, 2014

2014
[22]

Sharp analysis of low-rank kernel ma- trix approximations,

Francis Bach, “Sharp analysis of low-rank kernel ma- trix approximations,”Conference on Learning Theory, 2013

2013
[23]

Neal,Bayesian Learning for Neural Net- works, Springer, 1996

Radford M. Neal,Bayesian Learning for Neural Net- works, Springer, 1996

1996
[24]

Deep neural networks as gaus- sian processes,

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington, “Deep neural networks as gaus- sian processes,” inICLR, 2018. A. RESULTS ON MNIST Table A1. Results on MNIST with one hidden layer: 784– 64–10. Model Hyperpa...

2018

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.

[1] [1]

This leads to a simpler model and a simpler optimization problem

INTRODUCTION As an alternative to the usual approach of neural networks with all their weights trained on data, one can fix some of the weights to random values [1, 2, 3, 4, 5, 6]. This leads to a simpler model and a simpler optimization problem. The Johnson-Lindenstrauss lemma states that a high-dimensional space can be embedded into a space of much lowe...

[2] [2]

We introduce HalfNet, which learns the covariance structure of random projections rather than the individ- ual weights

[3] [3]

Accuracy scales smoothly with rank, indicating that much of the predictive power of trained layers is cap- tured by the low-dimensional geometric structure

[4] [4]

We show that the framework can be extended to (i) mul- tiple layers, (ii) convolutional front ends, and (ii) binary weights

[5] [5]

This paper is organized as follows: We present our model in Section 2 and our experimental results on MNIST in Sec- tion 3

HalfNet admits probabilistic, metric-learning, and ker- nel interpretations. This paper is organized as follows: We present our model in Section 2 and our experimental results on MNIST in Sec- tion 3. In Section 4, we show how our proposed model can also be adapted to learn binary weights. We give our experi- mental results on CIFAR-10 in Section 5. Secti...

[6] [6]

MODEL FORMULA TION We have hidden unithwithx, w h ∈R d andy h, wh0 ∈R yh =g(w ⊤ h x+w h0)(1) where typically,w h, wh0, are trained on a dataset. The weights may be drawn randomly, then kept fixed and not updated during learning: yh =g(r ⊤ h x), r h ∼ N d(0, I).(2) arXiv:2606.04583v1 [cs.LG] 3 Jun 2026 With HalfNet, our aim is to learn the distribution ofr...

work page Pith review arXiv 2026

[7] [7]

We use ReLU as the activation function at the hidden units and softmax at the output; training is done to minimize the cross-entropy using Adam

EXPERIMENTS ON MNIST We use the MNIST dataset which has28×28 = 784in- puts and ten classes, with 60,000 training and 10,000 test in- stances. We use ReLU as the activation function at the hidden units and softmax at the output; training is done to minimize the cross-entropy using Adam. The weights to the outputs are trained as usual; only the hidden units...

[8] [8]

We can generate binary hyperplanes by placing the sign function after Equation 3

HALF LA YER NETWORKS WITH BINARY WEIGHTS The model that we discuss above can be adapted to gener- ate binary (±1) weights, which are attractive because they can lead to major savings in terms of memory and computa- tion. We can generate binary hyperplanes by placing the sign function after Equation 3. This allows us to mapN k(0, I)to ad-dimensional multiv...