Network with Sub-Networks

Hakaru Tamukoh; Ninnart Fuengfusin

REVIEW 3 major objections 6 minor 5 references

Network with Sub-Networks

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A copy-and-share training scheme produces a neural network whose hidden layers can be detached at inference time without a significant accuracy drop.

desk verdict Depth-wise slimmable-net training idea, clearly described, but the test-accuracy parity claim is confounded by a 10x effective learning-rate difference between the proposed and baseline optimizers. read the letter →

arxiv 1908.00763 v2 pith:W3NODEYP submitted 2019-08-02 cs.LG cs.CV

Ninnart Fuengfusin , Hakaru Tamukoh This is my paper · ORCID

classification cs.LGcs.CV

keywords modelcompressionnetworkwithsub-networksmultilayerperceptrongradientsharingparametercopyingdepth-wiseslimminginference-timeadaptivityhandwrittendigitrecognition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces network with sub-networks (NSNs), a training scheme that turns one base neural network and its smaller relatives into a single shared parameter set. The aim is to make depth a runtime choice: after training, the network can drop its earliest hidden layers and keep working as a smaller network, so one stored model can serve devices with different compute budgets. The central evidence is on a handwritten-digit task, where the two-hidden-layer base model reaches 0.989 test accuracy, essentially matching the 0.9886 of a regularly trained same-size model, while its one-hidden-layer sub-model reaches 0.9843 and its softmax sub-model 0.926. The paper thus claims that the flexibility of multiple model sizes can be bought with training-time changes alone, without sacrificing base-model accuracy or storing several copies.

What carries the argument

The carrying mechanism is a paired weight-sharing rule applied every mini-batch. Copying learnable parameters enforces exact equality: the weight tensor of the first layer of a sub-model is copied to the second layer of the next larger model ($W_{o+1,m+1} = W_{o,m}$), so a detached layer's weights are never new or unused. Sharing gradient then keeps those shared tensors adapted to both roles: each pair of models is forward-propagated separately, and the parameter update uses the average of the two gradients ($W_{m,o} \leftarrow W_{m,o} - \frac{lr}{2}(\partial L_m/\partial W_{m,o} + \partial L_{m+1}/\partial W_{m+1,o+1})$). The one weight layer without a partner, the base model's input layer, updates by ordinary backpropagation. Together these operations turn depth itself into a tunable resource.

What would settle it

Train the same NSN architecture and sub-models using the standard momentum update (Eq. 3) instead of the modified one (Eq. 4), with hyperparameters and the copying/sharing procedure otherwise fixed, across several random seeds. If the base model's accuracy falls clearly below the 0.9886 regular-model baseline while detached sub-models also lose accuracy, the mechanism alone does not deliver the claimed parity.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that one set of weights can be trained to serve a deep network and its sub-networks at once, where each sub-network is formed by deleting the earliest hidden layers. The training procedure first copies parameters from smaller models into larger ones, then forward-propagates every model separately and updates each adjacent pair with averaged gradients. In the reported experiments, the two-hidden-layer base model reaches 0.989 test accuracy versus 0.9886 for the regular two-hidden-layer baseline; the one-hidden-layer sub-model keeps 0.9843, and the softmax sub-model keeps 0.926. The authors also report that the shared-weight constraint acts as a regularizer: it hurts the one-hidden-layer base model slightly but helps the two-hidden-layer base model, which edges out its baseline.

Load-bearing premise

The central comparison assumes NSNs may use a modified momentum update while baselines use the standard one, even though the authors chose the modified update because NSNs performed better with it; if that optimizer difference, rather than the copying and gradient-sharing mechanism, is what produces the matching accuracy, the core claim collapses.

Editorial extensions

If this is right

A single trained network can be stored once and then truncated at runtime, so devices of different capability share one checkpoint instead of many model copies.
Removing a hidden layer no longer collapses performance: with the two-hidden-layer model, deleting the first hidden layer leaves 0.9843 accuracy rather than a near-random result.
Depth-wise layer removal can be combined with width-wise slimming, so a future design could adjust both depth and width from one parameter set.
The extra flexibility is paid for in training: all models are forward-propagated separately and paired gradients are computed, increasing per-step training cost.
The method's regularization effect is visible: it slightly improves the two-hidden-layer base over the unconstrained baseline but slightly degrades the one-hidden-layer case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

This is effectively shared-weight ensembling: averaging gradients from a full and a truncated network biases the solution toward features that work at both depths, a mechanism with a natural family resemblance to knowledge distillation run in both directions. If that reading is right, a teacher-student objective may reproduce the parity more cheaply.
The balance between base-model and sub-model accuracy is controlled by which layers are left unshared; adjusting that count could give users a dial between full-model quality and detached-model quality.
The crucial scaling test is whether the parity survives convolutional and residual architectures on larger image benchmarks; the paper itself notes this is future work.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is a depth-wise counterpart to Slimmable Neural Networks: copy sub-model parameters into the base model and average gradients pairwise during training. That is a simple and reasonable extension, and the authors are upfront that SNNs are the only related work. The mechanism is clean and the writing is honest about the small scale.

What the paper does well: the method is described with clear equations, the motivation is practical, and the authors acknowledge that their approach actually hurts the one-hidden-layer base model. They also report submodel accuracies, which show the detached networks are functional. The hypothesis that gradient sharing acts as a regularizer is offered as a possible explanation, not overclaimed.

The soft spots are real and load-bearing. The central claim is that the NSN base model reaches test accuracy comparable to regular training—0.989 vs 0.9886 for the two-hidden-layer model. But the comparison is not controlled. NSNs are trained with Eq. 4, V_{t+1} = αV_t + (1−α)G, while the regular baselines use Eq. 3, V_{t+1} = αV_t + G. With α=0.9, Eq. 4 is effectively Eq. 3 with the gradient scaled by 0.1, so the accumulation term is about 10x smaller. That is a large change in effective learning rate, and the authors state they picked Eq. 4 because NSNs performed better with it. So the observed accuracy equality could easily come from the optimizer change, not from the copying-and-gradient-sharing mechanism. The stress-test note is correct: the decisive confound is uncontrolled.

Also, only one run is reported and the 'best test accuracy' is selected on the test set during training, so the point estimate has unknown variance and is optimistically biased. The paper limits itself to MNIST and MLPs, which the authors admit. These issues are enough to make the main claim conditional rather than established. The idea may work, but the current evidence doesn't show it.

The citation pattern is thin—only one related work—but that is not a problem for such a focused workshop-level contribution. The paper is coherent, not sloppy, and the authors are honest about limitations.

Who is this for? People working on flexible or slimmable networks, model compression, or multi-capacity inference. It is a minor incremental idea that could become a decent workshop paper if the experiments are redone with matched optimizers, multiple seeds, and validation-based model selection.

Recommendation: send to peer review rather than desk reject—the idea deserves referee time—but the authors need to fix the experimental protocol before publication.

Referee Report

3 major / 6 minor

Summary. The paper introduces "Network with Sub-Networks" (NSNs), a neural network whose weight layers can be detached during inference to yield shallower sub-networks. The training procedure first copies parameters from smaller sub-models into the base model (Eq. 1) and then updates shared weights by averaging gradients from pairs of networks (Eq. 2). Experiments on MNIST with one- and two-hidden-layer MLPs report that the base model achieves test accuracy comparable to regularly trained baselines (0.989 vs 0.9886 for the two-hidden-layer model), while retaining the ability to remove weight layers at inference time. The authors explicitly acknowledge that the demonstration is small-scale and leave larger-scale validation to future work.

Significance. If the central empirical claim is accepted, the idea is a modest but useful contribution to model deployment: a single model that can trade depth for speed at inference time, avoiding the memory cost of storing multiple separately trained models. The training procedure is simple, clearly described, and easily reproducible, and the paper is honest about the limited scale of the experiments. However, the load-bearing parity claim is not currently supported because the proposed and baseline models are trained under different optimizer configurations and with different regularization strengths, and only a single run with test-set-based selection is reported. These are correctness risks for the main claim, not mere presentation issues; they need to be resolved by additional experiments under a common protocol.

major comments (3)

[Section 4, Eqs. (3)-(4), Table 3] The central parity claim is confounded by the optimizer mismatch. With alpha=0.9, Eq. (4) is exactly Eq. (3) with the gradient G scaled by (1-alpha)=0.1, so the NSN models use an effective learning rate of 0.03 instead of the 0.3 used for the regular baselines under the stated step schedule. The authors also state that Eq. (4) was chosen after observing that it improved NSN performance and slightly worsened regular DNNs. Consequently, the reported 0.989 vs 0.9886 accuracy equality in Table 3 could be an artifact of per-method optimizer selection rather than evidence for the copying-and-gradient-sharing mechanism.
[Section 4, experimental protocol] The experiments report the best test accuracy over the 600-epoch training run, and no multiple-seed runs or error bars are provided. Selecting the best test accuracy during training both introduces optimistic bias and leaves the variance of the point estimate unknown, so the 0.0004 gap between model2 and ref-model2 is statistically unassessable. The paper should report mean +/- standard deviation over at least five independent runs, with a fixed epoch schedule or a validation-based early-stopping criterion.
[Section 4.1, 4.2 and Table 1] The L2 regularization coefficients differ between the proposed models and their baselines: the one-hidden-layer NSN uses 9e-6 while ref-model1 uses 5e-6, and the two-hidden-layer NSN uses 9e-5 while ref-model2 uses 1e-5. This is a second uncontrolled variable, so the claimed 'comparable' base-model accuracy could be partly due to differing regularization strengths rather than to the copying-and-sharing training procedure itself.

minor comments (6)

[Abstract] There is a typo: 'it's weight layers' should be 'its weight layers'.
[Conclusion] The verb 'purpose' should be 'propose', and 'purposed method' should be 'proposed method' throughout the manuscript.
[Introduction and Section 2] The phrase 'on fly' should be 'on the fly', and 'Tensorflow' should be 'TensorFlow'.
[Section 3.1, Eq. (1)] The indexing in Eq. (1) is inconsistent with the text and Fig. 1: the equation relates adjacent pairs (W_{o+1,m+1} = W_{o,m}), while the figure caption says that W1,1, W2,2, and W3,3 are exactly the same variable. Please clarify the notation and state explicitly which parameters are shared and which are merely copied at initialization or each iteration.
[Section 4.2, Table 3] The accuracy values are reported with inconsistent precision: '0.989' and '0.926' in Table 3 versus '0.9886' and '0.9241' in Table 1. Use the same number of significant digits for all entries.
[Section 4.1, Table 2] The table caption is missing a period and a space: 'model0-1' should be 'model0-1.' and the caption should be a complete sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical comparisons and a construction method, not a derivation that reduces to its own inputs.

full rationale

The paper does not present a derivation chain from assumptions to predicted results. Its central claim is that the proposed training procedure (copying learn-able parameters and sharing gradients) yields a base model with test accuracy comparable to regularly trained models while permitting layer detachment. The detachment property is true by construction: Eq. (1) literally copies the sub-model weights into the base model, and the paper labels this as the method rather than as a derived prediction. No fitted parameter is later renamed as a prediction; the reported accuracies are measured from held-out MNIST test data, and the baseline comparisons are externally trained models. The only questionable choice is the use of a different momentum update (Eq. 4) for NSNs than for baselines (Eq. 3), which the authors selected after observing better NSN performance. This is a potential experimental confound or fairness issue, not circularity: Eq. 4 is not defined in terms of the headline accuracy, and the comparison is still an empirical measurement. There are no load-bearing self-citations; the sole related-work citation (Slimmable Neural Networks) is background. The conclusion's own admission that the demonstration is small-scale is a scope limitation, not a circular step. Therefore no circularity is established under the standard that requires a quoted reduction of a claimed derivation to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests on standard deep learning machinery and on the paper's own design choices. The only free parameters are the regularization coefficients and the optimizer variant, both hand-tuned; the core assumption is that pairwise gradient averaging (Eq. 2) is a valid way to train shared weights across depths.

free parameters (3)

L2 regularization coefficient, model0-1 base model = 9e-6
Chosen for the NSN base model; differs from the baseline model's 5e-6, so the comparison is not fully controlled.
L2 regularization coefficient, model0-1-2 base model = 9e-5
Chosen for the NSN base model; a tuned hyperparameter that affects the reported accuracy.
Momentum update variant = Equation 4 for NSNs, Equation 3 for baselines
Adopted after observing NSNs perform better with Eq. 4; a post hoc modeling choice that favors the proposed method.

assumptions (3)

standard math Backpropagation via TensorFlow computes correct gradients for the cross-entropy loss with log-softmax.
Relied on by all training runs; not stated in the paper.
domain assumption The MNIST training and test sets are drawn from the same distribution.
Standard supervised learning assumption; the paper does not discuss distribution shift.
ad hoc to paper Averaging the gradients from a pair of networks of different depths (Eq. 2) yields a valid and effective update for the shared weights.
This is the core design choice of NSNs; the paper asserts it works empirically but provides no theoretical justification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Network with Sub-Networks." pith.science (2026). https://pith.science/paper/W3NODEYP

@misc{pith2026190800763,
  author       = {Pith},
  title        = {Pith review of: Network with Sub-Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W3NODEYP}},
  note         = {Machine review of arXiv:1908.00763}
}

read the original abstract

We introduce network with sub-networks, a neural network which its weight layers could be detached into sub-neural networks during inference. To develop weights and biases which could be inserted in both base and sub-neural networks, firstly, the parameters are copied from sub-model to base-model. Each model is forward-propagated separately. Gradients from a pair of networks are averaged and, used to update both networks. Our base model achieves the test-accuracy which is comparable to the regularly trained models, while the model maintains the ability to detach weight layers.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 4 canonical work pages

[1]

Krizhevsky, I

A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in neural information proces sing systems , pp. 1097–1105, 2012

work page 2012
[2]

K. He, X. Zhang, S. Ren, a nd J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016

work page 2016
[3]

Slimmable neural networks,

J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang, “Slimmable neural networks,” arXiv preprint arXiv:1812.08928, 2018

arXiv 2018
[4]

Mnist hand-written digit database,

Y. LeCun, C. Cortes, and C. Burges, “Mnist hand-written digit database,” AT&T Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, vol. 2, p. 18, 2010

work page 2010
[5]

Dropout: a simple way to prevent neural networks from overfitting

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting ”, The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davi s, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., “Tensorflow: A syste...

work page 1929

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Krizhevsky, I

A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in neural information proces sing systems , pp. 1097–1105, 2012

work page 2012

[2] [2]

K. He, X. Zhang, S. Ren, a nd J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016

work page 2016

[3] [3]

Slimmable neural networks,

J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang, “Slimmable neural networks,” arXiv preprint arXiv:1812.08928, 2018

arXiv 2018

[4] [4]

Mnist hand-written digit database,

Y. LeCun, C. Cortes, and C. Burges, “Mnist hand-written digit database,” AT&T Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, vol. 2, p. 18, 2010

work page 2010

[5] [5]

Dropout: a simple way to prevent neural networks from overfitting

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting ”, The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davi s, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., “Tensorflow: A syste...

work page 1929