Half-Layered Neural Networks

Ethem Alpaydin

REVIEW 3 major objections 6 minor 5 references

Half-Layered Neural Networks

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hidden unit with fixed random first-stage weights and only two trained parameters per unit can match the test accuracy of fully trained networks of similar parameter count.

desk verdict A clean, honest proposal for random-weight layers with trained per-unit rescaling, but the missing plain-ELM control leaves its main empirical claim unverified. read the letter →

arxiv 2506.04352 v1 pith:HEBPTJLN submitted 2025-06-04 cs.LG

Ethem Alpaydin This is my paper

classification cs.LG

keywords halflayerrandomweightsdeeplearningrepresentationfeatureextractionregularizationMNISTFashion

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a "half" hidden unit that splits a neuron's computation into two stages: a fixed random projection of the input, followed by a trainable scaling and shifting before the nonlinearity. Each half unit therefore has only two trainable parameters per hidden unit, a scale $u_h$ and a shift $u_{h0}$, regardless of how many inputs it receives. The central claim is that layers of such units can replace ordinary fully connected layers and reach test accuracy comparable to fully trained networks with the same number of trainable parameters, while using far fewer trainable weights overall. The support comes from MNIST and FashionMNIST experiments comparing several random-projection schemes, where half-layer networks match or approach full MLPs and convolutional baselines. The author interprets the random connections as a regularizer, since many noisy units acting together reduce variance, at the risk of adding bias.

What carries the argument

The half unit is the central object. It first computes a dot product $a_h$ using fixed random weights $r_{hj}$, then applies a trainable scale $u_h$ and shift $u_{h0}$ before the activation $g$, giving $z_h = g(u_h a_h + u_{h0})$. Four schemes are proposed for drawing the fixed weights: unit-normal values, binary $\{-1,+1\}$ values, random Mexican-hat filters, and random training instances used as templates. This decomposition is what makes the number of trainable parameters per hidden unit two, independent of the fan-in, and it is also what lets the random projection be rescaled to the data at hand.

What would settle it

Take a half-layer network with unit-normal fixed weights trained on normalized MNIST pixels, then train the same architecture on the same images with pixel values multiplied by a large factor such as 1000 (or shifted by a large constant) while keeping the random weights fixed; if test accuracy falls far below the normalized case, the trained scale and shift have not adapted the activation, contradicting the claim that $r_{hj}$ can be drawn at any scale.

Watch

Extended reading notes

Core claim

The constituent discovery is that the two trained parameters placed between a random projection and the activation suffice to produce useful hidden representations. Writing $a_h = \sum_j r_{hj} x_j$ for the fixed random projection and $z_h = g(u_h a_h + u_{h0})$ for the output, the pair $(u_h, u_{h0})$ learns to adapt the scale and offset of the random activation to the data, so the random weights $r_{hj}$ themselves never need updating. Because $r_{hj}$ is frozen, the trainable parameter count of a half layer is independent of the input dimensionality. Experiments on MNIST show that half-layer networks with 1024 or 2048 random units reach about 95% test accuracy, comparable to fully trained MLPs with similar parameter counts, and that a half layer inserted after convolutional layers also works, though with a larger accuracy gap on FashionMNIST. The paper frames each half unit as a noisy base-learner and the layer as an ensemble whose averaging reduces variance but can smooth too much when too many random units are used.

Load-bearing premise

The load-bearing premise is that a fixed random projection produces useful features once the trained scale and shift adapt its total activation to the data, so the random weights can be drawn without carefully matching input scale or dimensionality; if the two parameters cannot compensate for a given nonlinearity or data distribution, the half layer contributes no useful representation.

Editorial extensions

If this is right

Fully connected layers, which usually dominate a deep network's parameter count, can be replaced by half layers with only two trained parameters per unit, drastically cutting trainable weights with small accuracy cost on MNIST and FashionMNIST.
Half layers can be inserted after convolutional layers, and the output error can be back-propagated through the fixed random projection to train the earlier convolutional filters.
Binary random weights work nearly as well as normal random weights, suggesting that fixed projections can be stored in one bit and computed by additions and subtractions.
Learning curves of half-layer networks show training and test accuracy moving together, indicating that random units act as regularizers and reduce overfitting relative to full networks.
The author is cautious: the accuracy gain is not universal, and too many random units can over-smooth and introduce bias, so the practical benefit depends on the data set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One consequence the author leaves implicit is that half layers offer a cheap way to adapt a frozen random feature extractor by training only per-neuron scale and shift, without backpropagating through the fixed weights.
The binary-weight result implies a hardware-oriented reading: the fixed projection becomes additions and subtractions with one-bit weights, and the only trained values are two scalars per neuron.
The template-matching variant invites a connection to random-feature kernel methods, with the trained scale and shift acting as a per-template lengthscale and offset.
A natural extension consistent with the paper's own caution is to test half layers on larger datasets like CIFAR-10 with deeper architectures to see whether the regularizing effect persists.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Your instinct to flag the missing ELM baseline is right, and it is the main thing to know about this paper. The half unit is exactly a fixed random projection plus a trained affine rescale before the nonlinearity; without a control that has the former but not the latter, the paper cannot actually show that the rescale is what buys the accuracy. That doesn't sink the paper, because the claims are modest and the architecture is clean, but it turns the paper from a demonstration into a proposal.

What is genuinely new here is small but real: the random-weight literature the author cites (ELM, random features, randomized networks) fixes the whole pre-activation, including scale and bias. Breaking that pre-activation into a random part and a trained affine part, so that each hidden unit has exactly two trainable parameters regardless of fan-in, is a specific combination I have not seen in those references. The paper also does a useful thing by testing several random weight schemes, including one-bit binary weights, and showing the binary variant often matches the Gaussian one. The learning curves that show train/test accuracy moving together for the random layers are a nice qualitative illustration of the regularizing effect.

Where the paper is soft, in proportion: first, the missing plain ELM baseline is not cosmetic. A standard ELM with the same random projections and a random bias has even fewer trainable parameters, and if it matches rnd-H-N or rnd-H-B within noise, the two trained parameters are not earning their keep. The paper needs that experiment to support the word 'half' rather than just 'random layer with extra parameters.' Second, the paper never states the training protocol—optimizer, learning rate, epochs, normalization, weight initialization for the second stage—which makes the results hard to assess or reproduce. Third, there are prose/table errors in the FashionMNIST section (a duplicate rnd-1024-B row where the T variant is presumably intended). Minor: two small image benchmarks, so the generality claims are appropriately limited.

Bottom line: this is a serious, readable paper by someone who knows the literature; it just needs the decisive control and some reporting cleanup. I would send it to a referee rather than desk-reject, and request the ELM baseline and training details as major revisions. It is not going to reshape the field, but it is a legitimate architecture variant that is cheap to test.

Referee Report

3 major / 6 minor

Summary. The paper proposes "half" layers for neural networks, in which each hidden unit has fixed random input weights r_hj followed by two trained parameters (scale u_h and offset u_h0) before the nonlinearity, so that the number of trainable parameters per hidden unit is independent of the fan-in. The author formalizes the construction in Section 2, gives four random-weight schemes (Normal, Binary, Mexican-hat, and Instance), and reports experiments on MNIST and FashionMNIST for one-hidden-layer MLPs and for convolutional networks with a half layer following the convolutional features. The central empirical claim is that such half layers achieve test accuracy comparable to fully trained MLPs at similar numbers of trainable parameters, with the regularizing effect of the random projection offsetting the reduction in trained parameters. The paper concludes cautiously that half layers show promise, especially as fully connected layers, while acknowledging the need for further experiments on larger datasets and deeper networks.

Significance. If the central claim is supported, the half-layer construction is a simple and potentially useful way to reduce the number of trainable parameters in fully connected layers, and the binary-weight variant additionally offers memory and arithmetic savings. The paper is clearly written, compares several random-weight schemes, reports mean and standard deviation over five runs, and gives a restrained conclusion that does not overstate the results. However, the significance is currently limited by the absence of the natural control experiment: a classical random-weight/ELM baseline with the same random projections and no trained scale or shift. Without that control, the reported accuracy cannot be attributed to the proposed mechanism rather than to random features in general. The contribution is therefore potentially interesting but not yet established.

major comments (3)

[Section 3.1, Tables 1-2 and Eq. (5)-(6)] The experiments do not include the decisive baseline for the proposed mechanism. In a half unit, the only new ingredients relative to classical random-weight networks are the trained scale u_h and offset u_h0 in Eq. (6). A standard ELM/random-feature model draws the same random first-stage weights r_hj and a fixed random bias, and trains only the output layer; it has fewer trainable parameters than rnd-H-N or rnd-H-B and the same random projections. If such an ELM achieves test accuracy within noise of the half-layer results at the same H, then the two trained parameters per unit are not the source of the reported performance, and the central architectural contribution is unsupported. If, instead, the half layer clearly outperforms the ELM, that isolates the effect of the trained scale and shift. This baseline should be added for at least the N and B schemes on both datasets.
[Section 3, Tables 1-4] The full training protocol is missing, which makes the empirical comparisons hard to reproduce and interpret. The paper reports averages over five seeds but does not specify the optimizer, learning rate schedule, number of epochs, batch size, initialization of u_h, u_h0 and output weights, or whether any regularization or early stopping is used. This matters because the claimed regularizing effect of the random units (for example, the learning curves in Figures 7-8) could be affected by optimization choices rather than by the architecture itself. Please provide the complete protocol for all baselines and proposed models, and state whether the same protocol is used for the full and half networks.
[Section 3.2, Tables 3-4] The convolutional experiments are compared only against clp and cmlp, which are fully trained models. The comparison therefore shows that replacing a fully connected layer with a half layer reduces trainable parameters, but it does not test whether the per-unit scale and shift are beneficial in this setting. The missing control is again a convolutional feature extractor followed by a fully random projection layer with fixed bias and only the output layer trained. Without this control, the statement that the half layer gives a regularizing effect after convolutional layers is not isolated from the general behavior of random features.

minor comments (6)

[Table 2] The first two full-layered rows are both labeled 'mlp-32' but the parameter counts (12,730 and 25,450) identify them as mlp-16 and mlp-32, respectively; please correct the label. Also, the row 'rnd-1024-B 86.02, 0.35 82.97, 0.20' appears twice; the second occurrence is presumably rnd-1024-T and should be labeled accordingly.
[Figures 3 and 4] The y-axis is labeled 'Test error' but the plotted values are test accuracies; the label should read 'Test accuracy' or the values should be converted to error rates. In Figure 4, the legend entries 'rnd-B(N)', 'rnd-N(N)', 'rnd-M(N)', and 'rnd-T(N)' appear to be typos for 'rnd-B', 'rnd-N', 'rnd-M', and 'rnd-T'.
[Section 3.2] The text says 'On FashionMNIST (see Table 3 and Figure 5)', but the FashionMNIST convolutional results are in Table 4 and Figure 6. Also, the sentence 'M and T variants are not applicable here' needs an explanation of why those random-weight schemes cannot be used after the convolutional feature extractor.
[Eq. (1)-(6)] The desired output is denoted r^t in Eq. (1)-(2) while the first-stage weights are also denoted r_hj in Eq. (4)-(5); this creates a notation conflict. Please rename the desired output, for example to t^t.
[Abstract and Section 3.2] The abstract says the paper shows 'how such half units can be used in the first or any later layer in a deep network', but the experiments cover only a first hidden layer and a half layer after convolutional features; there is no experiment with multiple half layers in a deep stack. Please soften the wording or add an experiment that demonstrates the claim.
[Section 3.2] The description of the convolutional experiments should state whether the convolutional layers are trained end-to-end from scratch or pre-trained and then fine-tuned, since the text only says their parameters 'are trained as usual' while the error is back-propagated through the half layer.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the half-layer accuracy results are empirical, and no equation or fitted parameter is reused as a prediction.

full rationale

The paper's contribution is an architecture, not a derivation. Equation (5) defines the random projection a_h^t, and Equation (6) defines the half-unit output z_h^t = g(u_h a_h^t + u_h0). The trained parameter count per hidden layer is computed arithmetically from this definition, but no accuracy value is derived from that count; all accuracies are measured and reported in Tables 1-4 on the external MNIST and FashionMNIST benchmarks. There is no fitted-then-predicted quantity, no self-citation used as load-bearing evidence, and no invoked uniqueness theorem. The paper's explanation of the random units' regularizing effect is an interpretation of the learning curves, not a step that reduces Equations (5)-(6) to themselves. The absence of a plain ELM baseline is a test of whether the trained scale and shift contribute causally, but that is an experimental control issue, not a circularity under the seven enumerated patterns. The limitation paragraph explicitly states that the findings rest on simple datasets and need further experimentation, so the paper does not claim a derived guarantee. Consequently, the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The half-layer proposal introduces no new forces, particles, or external entities. Its central claim rests on the empirical assumption that random projections plus a learned affine transform yield useful features, on unspecified training details, and on the choice of layer width H.

free parameters (1)

hidden layer size H = 256, 512, 1024, 2048
The paper sweeps H across four values to demonstrate the parameter/accuracy trade-off; the chosen values are ad hoc, and the conclusions about reduced parameters depend on this choice.

assumptions (4)

domain assumption Random first-stage weights with learned per-unit scale and shift produce useful hidden representations.
This is the core inductive bias of the half-layer; it is not proven, only tested on two datasets.
domain assumption The nonlinearity g is chosen so that the composition with a random affine map is trainable by backpropagation; the paper does not specify g in experiments.
Standard ML practice, but the activation function is left unspecified, which affects the validity of the empirical comparisons.
domain assumption Backpropagation converges to a sufficiently good local minimum for the half-layer networks.
The paper notes non-convexity but assumes optimization succeeds; no convergence analysis is given.
domain assumption Results on MNIST and FashionMNIST with simple architectures generalize to larger datasets and deeper networks.
The paper itself limits its conclusions, but the promise of the method implicitly relies on this extrapolation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Half-Layered Neural Networks." pith.science (2026). https://pith.science/paper/HEBPTJLN

@misc{pith2026250604352,
  author       = {Pith},
  title        = {Pith review of: Half-Layered Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HEBPTJLN}},
  note         = {Machine review of arXiv:2506.04352}
}

read the original abstract

We propose a ``half'' layer of hidden units that has some of its weights randomly set and some of them trained. A half unit is composed of two stages: First, it takes a weighted sum of its inputs with fixed random weights, and second, the total activation is multiplied and then translated using two modifiable weights, before the result is passed through a nonlinearity. The number of modifiable weights of each hidden unit is thus two and does not depend on the fan-in. We show how such half units can be used in the first or any later layer in a deep network, possibly following convolutional layers. Our experiments on MNIST and FashionMNIST data sets indicate the promise of half layers, where we can achieve reasonable accuracy with a reduced number of parameters due to the regularizing effect of the randomized connections.

Figures

Figures reproduced from arXiv: 2506.04352 by the authors.

**Figure 2.** Different ways of randomly setting the weights of a half layer. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Results on MNIST with networks having one full or half hidden layer. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Results on FashionMNIST with networks having one full or half hidden layer. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Convolutional network results on MNIST. The convolutional version of our method, which we name crnd-H, keeps the same convolutional structure and adds a half layer of 𝐻 units between the two layers of 1,152 and 10 units; this adds 2 × 𝐻 + (𝐻 + 1) × 10 weights. The para…

**Figure 6.** Figure 6: Convolutional network results on FashionMNIST. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Learning curves of (a) mlp32 and (b) conv2 on FashionMNIST. 0 5 10 15 20 Epochs 65 70 75 80 85 Accuracy FASHION_LPR_(0_1024) Tra Tes 0 5 10 Epochs 81 82 83 84 85 86 87 88 Accuracy Fashion_CNN2_LPR(1024) Tra Tes [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Learning curves of (a) rnd-B-1024 and (b) convrnd-B-1024 on FashionMNIST. network. Because the number of trained parameters does not depend on the fan-in, we believe that such structures are especially interesting as fully-connected layers, which, in general, make the …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

5 extracted references · 5 canonical work pages

[1]

W. Cao, X. Wang, Z. Ming, and J. Gao. 2017. A Review on Neural Networks with Random Weights.Neurocomputing275 (2017), 278–287

work page 2017
[2]

Gallicchio and S

C. Gallicchio and S. Scardapane. 2020. Deep Randomized Neural Networks. InRecent Trends in Learning From Data, L. Oneto, N. Navarin, A. Sperduti, and D. Anguita (Eds.). Springer, 43–68. Half-Layered Neural Networks 11

work page 2020
[3]

Huang, H

G. Huang, H. Zhou, X. Ding, and R. Zhang. 2012. Extreme Learning Machine for Regression and Multiclass Classification.IEEE Transactions on Systems, Man, and Cybernetics42 (2012), 513–529. Issue 2

work page 2012
[4]

Rahimi and B

A. Rahimi and B. Recht. 2007. Random Features for Large-Scale Kernel Machines. InAdvances in Neural Information Processing Systems, J. Platt, D. Koller, Y. Singer, and S. Roweis (Eds.). (NIPS), Curran Associates, Inc

work page 2007
[5]

Scardapane and D Wang

S. Scardapane and D Wang. 2017. Randomness in Neural Networks: An Overview.WIREs Data Mining Knowledge Discovery7:e1200 (2017). https://doi.org/10.1002/widm.1200

work page doi:10.1002/widm.1200 2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

W. Cao, X. Wang, Z. Ming, and J. Gao. 2017. A Review on Neural Networks with Random Weights.Neurocomputing275 (2017), 278–287

work page 2017

[2] [2]

Gallicchio and S

C. Gallicchio and S. Scardapane. 2020. Deep Randomized Neural Networks. InRecent Trends in Learning From Data, L. Oneto, N. Navarin, A. Sperduti, and D. Anguita (Eds.). Springer, 43–68. Half-Layered Neural Networks 11

work page 2020

[3] [3]

Huang, H

G. Huang, H. Zhou, X. Ding, and R. Zhang. 2012. Extreme Learning Machine for Regression and Multiclass Classification.IEEE Transactions on Systems, Man, and Cybernetics42 (2012), 513–529. Issue 2

work page 2012

[4] [4]

Rahimi and B

A. Rahimi and B. Recht. 2007. Random Features for Large-Scale Kernel Machines. InAdvances in Neural Information Processing Systems, J. Platt, D. Koller, Y. Singer, and S. Roweis (Eds.). (NIPS), Curran Associates, Inc

work page 2007

[5] [5]

Scardapane and D Wang

S. Scardapane and D Wang. 2017. Randomness in Neural Networks: An Overview.WIREs Data Mining Knowledge Discovery7:e1200 (2017). https://doi.org/10.1002/widm.1200

work page doi:10.1002/widm.1200 2017