{"id":"b75ee48a-935c-4a7f-aa28-daea52c58a60","arxiv_id":"1908.00763","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Network with Sub-Networks trains a base network and its layer-pruned sub-networks together by copying parameters and averaging gradients, so the base model can shed layers at runtime with little accuracy loss.","lead":"A training method for neural networks that lets a model drop its early layers at inference time while keeping accuracy close to the full network. The authors demonstrate the approach on small MNIST classifiers and report accuracy within a fraction of a percent of regular training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Parity claim is not yet established: NSNs were trained with an effectively 10x smaller learning rate (Eq.4 vs Eq.3) and evaluated as a single best-test run, so the 0.989 vs 0.9886 margin is confounded.","rationale":"The paper's only support for the central claim is the MNIST experiment. The reader correctly identified the optimizer mismatch as the weakest assumption: the authors explicitly chose Eq.4 for NSNs and Eq.3 for baselines after observing that this pairing favored their method, and Eq.4 is mathematically equivalent to Eq.3 with a 10x smaller effective learning rate at alpha=0.9. This alone makes the 0.989 versus 0.9886 comparison uninterpretable as evidence for the proposed mechanism. The additional problems of a single run and test-set-based epoch selection reinforce the same conclusion: the empirical parity could be due to uncontrolled experimental choices. My recommended concrete test directly removes the confound by fixing the optimizer rule and sweeping learning rate, which is the minimal experiment needed to decide whether the central claim has empirical support. Since this is the same condition the reader imposed, the verdict remains conditional rather than accept or reject.","tokens_in":4735,"tokens_out":15095,"duration_ms":154239,"concrete_test":"Run model0-1-2 and ref-model2 under a single optimizer rule: use Eq.3 for both, with a learning-rate sweep over {0.3, 0.1, 0.03, 0.01} and the same schedule; use at least 10 random seeds, select the epoch by validation accuracy, and report mean and standard deviation of test accuracy. If the best same-rule NSN base accuracy is within 0.2 percentage points of the best same-rule baseline, the parity claim survives; if the gap exceeds this or the NSN is only competitive under Eq.4, the Table 3 comparison is not evidence for the mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: Table 3's model2 (0.989) is said to match ref-model2 (0.9886). For this comparison to be load-bearing, the two models must be trained under a common protocol. They are not. Section 4 states that NSNs use Eq.4, V_{t+1}=alpha*V_t+(1-alpha)*G, while regularly-trained baselines use Eq.3, V_{t+1}=alpha*V_t+G. With alpha=0.9, Eq.4 is exactly Eq.3 with the gradient scaled by 0.1, i.e. an effective learning rate of 0.03 instead of 0.3 in the first 200 epochs. The authors also state they selected Eq.4 because NSNs performed better with it and it was slightly worse for regular DNNs; this is post-hoc per-method optimizer selection. Thus the 0.0004 advantage could be an artifact of the effective-learning-rate pairing rather than evidence for the copying/gradient-sharing mechanism. In addition, only one run is reported and the 'best test accuracy' is selected on the test set during training, so the point estimate has unknown variance and is optimistically biased. The paper itself limits the demonstration to MNIST with a few hidden layers (Conclusion). The claim may be true, but the current experiment does not control the decisive confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces \"Network with Sub-Networks\" (NSNs), a neural network whose weight layers can be detached during inference to yield shallower sub-networks. The training procedure first copies parameters from smaller sub-models into the base model (Eq. 1) and then updates shared weights by averaging gradients from pairs of networks (Eq. 2). Experiments on MNIST with one- and two-hidden-layer MLPs report that the base model achieves test accuracy comparable to regularly trained baselines (0.989 vs 0.9886 for the two-hidden-layer model), while retaining the ability to remove weight layers at inference time. The authors explicitly acknowledge that the demonstration is small-scale and leave larger-scale validation to future work.","tokens_in":5049,"tokens_out":3795,"duration_ms":37358,"significance":"If the central empirical claim is accepted, the idea is a modest but useful contribution to model deployment: a single model that can trade depth for speed at inference time, avoiding the memory cost of storing multiple separately trained models. The training procedure is simple, clearly described, and easily reproducible, and the paper is honest about the limited scale of the experiments. However, the load-bearing parity claim is not currently supported because the proposed and baseline models are trained under different optimizer configurations and with different regularization strengths, and only a single run with test-set-based selection is reported. These are correctness risks for the main claim, not mere presentation issues; they need to be resolved by additional experiments under a common protocol.","major_comments":[{"comment":"The central parity claim is confounded by the optimizer mismatch. With alpha=0.9, Eq. (4) is exactly Eq. (3) with the gradient G scaled by (1-alpha)=0.1, so the NSN models use an effective learning rate of 0.03 instead of the 0.3 used for the regular baselines under the stated step schedule. The authors also state that Eq. (4) was chosen after observing that it improved NSN performance and slightly worsened regular DNNs. Consequently, the reported 0.989 vs 0.9886 accuracy equality in Table 3 could be an artifact of per-method optimizer selection rather than evidence for the copying-and-gradient-sharing mechanism.","section":"Section 4, Eqs. (3)-(4), Table 3"},{"comment":"The experiments report the best test accuracy over the 600-epoch training run, and no multiple-seed runs or error bars are provided. Selecting the best test accuracy during training both introduces optimistic bias and leaves the variance of the point estimate unknown, so the 0.0004 gap between model2 and ref-model2 is statistically unassessable. The paper should report mean +/- standard deviation over at least five independent runs, with a fixed epoch schedule or a validation-based early-stopping criterion.","section":"Section 4, experimental protocol"},{"comment":"The L2 regularization coefficients differ between the proposed models and their baselines: the one-hidden-layer NSN uses 9e-6 while ref-model1 uses 5e-6, and the two-hidden-layer NSN uses 9e-5 while ref-model2 uses 1e-5. This is a second uncontrolled variable, so the claimed 'comparable' base-model accuracy could be partly due to differing regularization strengths rather than to the copying-and-sharing training procedure itself.","section":"Section 4.1, 4.2 and Table 1"}],"minor_comments":[{"comment":"There is a typo: 'it's weight layers' should be 'its weight layers'.","section":"Abstract"},{"comment":"The verb 'purpose' should be 'propose', and 'purposed method' should be 'proposed method' throughout the manuscript.","section":"Conclusion"},{"comment":"The phrase 'on fly' should be 'on the fly', and 'Tensorflow' should be 'TensorFlow'.","section":"Introduction and Section 2"},{"comment":"The indexing in Eq. (1) is inconsistent with the text and Fig. 1: the equation relates adjacent pairs (W_{o+1,m+1} = W_{o,m}), while the figure caption says that W1,1, W2,2, and W3,3 are exactly the same variable. Please clarify the notation and state explicitly which parameters are shared and which are merely copied at initialization or each iteration.","section":"Section 3.1, Eq. (1)"},{"comment":"The accuracy values are reported with inconsistent precision: '0.989' and '0.926' in Table 3 versus '0.9886' and '0.9241' in Table 1. Use the same number of significant digits for all entries.","section":"Section 4.2, Table 3"},{"comment":"The table caption is missing a period and a space: 'model0-1' should be 'model0-1.' and the caption should be a complete sentence.","section":"Section 4.1, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's idea is clear and the authors are appropriately cautious about scope. The main issue is the confounded experimental protocol; because the defect is fixable with additional experiments (same optimizer and regularization for both groups, multiple seeds, validation-based model selection), I recommend major_revision rather than rejection. I would also encourage the editor to consider whether the journal's standards for empirical contributions require stronger baselines and statistical rigor than this short paper currently provides."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is a depth-wise counterpart to Slimmable Neural Networks: copy sub-model parameters into the base model and average gradients pairwise during training. That is a simple and reasonable extension, and the authors are upfront that SNNs are the only related work. The mechanism is clean and the writing is honest about the small scale.\n\nWhat the paper does well: the method is described with clear equations, the motivation is practical, and the authors acknowledge that their approach actually hurts the one-hidden-layer base model. They also report submodel accuracies, which show the detached networks are functional. The hypothesis that gradient sharing acts as a regularizer is offered as a possible explanation, not overclaimed.\n\nThe soft spots are real and load-bearing. The central claim is that the NSN base model reaches test accuracy comparable to regular training—0.989 vs 0.9886 for the two-hidden-layer model. But the comparison is not controlled. NSNs are trained with Eq. 4, V_{t+1} = αV_t + (1−α)G, while the regular baselines use Eq. 3, V_{t+1} = αV_t + G. With α=0.9, Eq. 4 is effectively Eq. 3 with the gradient scaled by 0.1, so the accumulation term is about 10x smaller. That is a large change in effective learning rate, and the authors state they picked Eq. 4 because NSNs performed better with it. So the observed accuracy equality could easily come from the optimizer change, not from the copying-and-gradient-sharing mechanism. The stress-test note is correct: the decisive confound is uncontrolled.\n\nAlso, only one run is reported and the 'best test accuracy' is selected on the test set during training, so the point estimate has unknown variance and is optimistically biased. The paper limits itself to MNIST and MLPs, which the authors admit. These issues are enough to make the main claim conditional rather than established. The idea may work, but the current evidence doesn't show it.\n\nThe citation pattern is thin—only one related work—but that is not a problem for such a focused workshop-level contribution. The paper is coherent, not sloppy, and the authors are honest about limitations.\n\nWho is this for? People working on flexible or slimmable networks, model compression, or multi-capacity inference. It is a minor incremental idea that could become a decent workshop paper if the experiments are redone with matched optimizers, multiple seeds, and validation-based model selection.\n\nRecommendation: send to peer review rather than desk reject—the idea deserves referee time—but the authors need to fix the experimental protocol before publication.","headline":"Depth-wise slimmable-net training idea, clearly described, but the test-accuracy parity claim is confounded by a 10x effective learning-rate difference between the proposed and baseline optimizers.","tokens_in":5544,"tokens_out":2477,"would_cite":false,"duration_ms":27622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A copy-and-share training scheme produces a neural network whose hidden layers can be detached at inference time without a significant accuracy drop.","keywords":["model compression","network with sub-networks","multilayer perceptron","gradient sharing","parameter copying","depth-wise slimming","inference-time adaptivity","handwritten digit recognition"],"falsifier":"Train the same NSN architecture and sub-models using the standard momentum update (Eq. 3) instead of the modified one (Eq. 4), with hyperparameters and the copying/sharing procedure otherwise fixed, across several random seeds. If the base model's accuracy falls clearly below the 0.9886 regular-model baseline while detached sub-models also lose accuracy, the mechanism alone does not deliver the claimed parity.","tokens_in":4516,"feed_emoji":"🧠","tokens_out":9092,"duration_ms":88841,"temperature":0.7,"pith_summary":"The paper introduces network with sub-networks (NSNs), a training scheme that turns one base neural network and its smaller relatives into a single shared parameter set. The aim is to make depth a runtime choice: after training, the network can drop its earliest hidden layers and keep working as a smaller network, so one stored model can serve devices with different compute budgets. The central evidence is on a handwritten-digit task, where the two-hidden-layer base model reaches 0.989 test accuracy, essentially matching the 0.9886 of a regularly trained same-size model, while its one-hidden-layer sub-model reaches 0.9843 and its softmax sub-model 0.926. The paper thus claims that the flexibility of multiple model sizes can be bought with training-time changes alone, without sacrificing base-model accuracy or storing several copies.","feed_headline":"One neural net sheds layers at run time and keeps its accuracy","feed_subtitle":"A copy-and-share training rule lets one model act as its own smaller sub-networks while matching single-model accuracy.","key_machinery":"The carrying mechanism is a paired weight-sharing rule applied every mini-batch. Copying learnable parameters enforces exact equality: the weight tensor of the first layer of a sub-model is copied to the second layer of the next larger model ($W_{o+1,m+1} = W_{o,m}$), so a detached layer's weights are never new or unused. Sharing gradient then keeps those shared tensors adapted to both roles: each pair of models is forward-propagated separately, and the parameter update uses the average of the two gradients ($W_{m,o} \\leftarrow W_{m,o} - \\frac{lr}{2}(\\partial L_m/\\partial W_{m,o} + \\partial L_{m+1}/\\partial W_{m+1,o+1})$). The one weight layer without a partner, the base model's input layer, updates by ordinary backpropagation. Together these operations turn depth itself into a tunable resource.","core_discovery":"On its own terms, the discovery is that one set of weights can be trained to serve a deep network and its sub-networks at once, where each sub-network is formed by deleting the earliest hidden layers. The training procedure first copies parameters from smaller models into larger ones, then forward-propagates every model separately and updates each adjacent pair with averaged gradients. In the reported experiments, the two-hidden-layer base model reaches 0.989 test accuracy versus 0.9886 for the regular two-hidden-layer baseline; the one-hidden-layer sub-model keeps 0.9843, and the softmax sub-model keeps 0.926. The authors also report that the shared-weight constraint acts as a regularizer: it hurts the one-hidden-layer base model slightly but helps the two-hidden-layer base model, which edges out its baseline.","pith_inferences":["This is effectively shared-weight ensembling: averaging gradients from a full and a truncated network biases the solution toward features that work at both depths, a mechanism with a natural family resemblance to knowledge distillation run in both directions. If that reading is right, a teacher-student objective may reproduce the parity more cheaply.","The balance between base-model and sub-model accuracy is controlled by which layers are left unshared; adjusting that count could give users a dial between full-model quality and detached-model quality.","The crucial scaling test is whether the parity survives convolutional and residual architectures on larger image benchmarks; the paper itself notes this is future work."],"forward_implications":["A single trained network can be stored once and then truncated at runtime, so devices of different capability share one checkpoint instead of many model copies.","Removing a hidden layer no longer collapses performance: with the two-hidden-layer model, deleting the first hidden layer leaves 0.9843 accuracy rather than a near-random result.","Depth-wise layer removal can be combined with width-wise slimming, so a future design could adjust both depth and width from one parameter set.","The extra flexibility is paid for in training: all models are forward-propagated separately and paired gradients are computed, increasing per-step training cost.","The method's regularization effect is visible: it slightly improves the two-hidden-layer base over the unconstrained baseline but slightly degrades the one-hidden-layer case."],"supporting_citations":[{"why":"supplies the width-wise slimming baseline that NSNs transpose to depth-wise layer removal.","marker":"3"},{"why":"provides the handwritten-digit benchmark from which all reported accuracies are drawn.","marker":"4"},{"why":"supplies the dropout regularization applied to the base models' input and hidden layers.","marker":"5"},{"why":"gives the standard momentum-update baseline (Eq. 3) that NSNs modify for their own training.","marker":"6"}],"fun_headline_variants":["Detachable layers: one net, many sizes, accuracy holds","Shared weights let a net act as its own subnets","Copy-share training yields detachable sub-networks","Single model, multiple depths: trained to detach","Shed layers at inference, keep accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes NSNs may use a modified momentum update while baselines use the standard one, even though the authors chose the modified update because NSNs performed better with it; if that optimizer difference, rather than the copying and gradient-sharing mechanism, is what produces the matching accuracy, the core claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Detachable layers: one net, many sizes, accuracy holds","Shared weights let a net act as its own subnets","Copy-share training yields detachable sub-networks","Single model, multiple depths: trained to detach","Shed layers at inference, keep accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000847,"raw_usage":{"total_tokens":3607,"prompt_tokens":786,"completion_tokens":2821,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":402,"completion_tokens_details":{"reasoning_tokens":2744}},"tokens_in":402,"tokens_out":2821,"duration_ms":22950,"temperature":1.0,"reasoning_tokens":2744,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:32:33.825413+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same NSN architecture and sub-models using the standard momentum update (Eq. 3) instead of the modified one (Eq. 4), with hyperparameters and the copying/sharing procedure otherwise fixed, across several random seeds. If the base model's accuracy falls clearly below the 0.9886 regular-model baseline while detached sub-models also lose accuracy, the mechanism alone does not deliver the claimed parity.","supporting_citations":[{"cited_title":"Mnist hand-written digit database,","cited_arxiv_id":null,"evidence_quote":"provides the handwritten-digit benchmark from which all reported accuracies are drawn."},{"cited_title":"Dropout: a simple way to prevent neural networks from overfitting","cited_arxiv_id":null,"evidence_quote":"supplies the dropout regularization applied to the base models' input and hidden layers."}],"review_version":1}