{"id":"5b6149d5-9189-42f9-bdeb-e90c20598633","arxiv_id":"1908.06022","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Replacing skip connections with a learnable 1x1 convolution (ELS) during weight-sharing supernet training stabilizes the supernet and improves architecture ranking, yielding the SCARLET family of ImageNet models.","lead":"This paper proposes a small learnable fix, a 1x1 convolution called an Equivariant Learnable Stabilizer, that makes weight-sharing neural architecture search stable when the search space includes skip connections. The result is a family of mobile networks, with SCARLET-A reaching 76.9% top-1 ImageNet accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 1's proof only covers channel-matched layers, yet S1/S2 include skips that change channel count (Fig. 11); the ELS equivalence claim is therefore unsupported exactly where it matters for one-shot ranking.","rationale":"My reading: the paper's empirical contribution is real and reproducible. ELS visibly stabilizes training, improves ranking on a channel-matched NAS-Bench-101 subspace, and SCARLET-A/B/C train to strong ImageNet accuracies. I do not dispute those observations. The load-bearing weakness is theoretical: the exact equivalence guaranteed by Lemma 1 is the only thing that lets the authors remove ELS and claim the supernet searched the same architecture class that was later trained. The Appendix proof is algebraic and valid for a single convolution with matching channel bookkeeping, but the search space intentionally includes skips at channel-changing layers, as confirmed by Fig. 11 and by the fact that SCARLET-B/C use identity operations. At those layers Eq. 3 either has a type mismatch (the shared next op expects c_{l+1} channels) or requires folding the ELS into the next block's weights, which breaks weight sharing. The paper's own ranking validation avoids this by adding a projection. This is not a reason to reject the empirical results; the final models are trained from scratch and are strong. It is, however, a reason the central 'same search space' claim is conditional. The proposed control experiment would settle whether ELS ranking survives the mismatched case; if it does, the concern is moot.","tokens_in":14793,"tokens_out":8610,"duration_ms":87537,"concrete_test":"Re-run the NAS-Bench-101 ranking experiment of §6.2 without the 1x1 projection added before the first node (Fig. 6), so that a skip connection can change the channel count and produce the c_l vs c_{l+1} mismatch present in S1/S2. Compare Kendall tau for supernets trained with ELS and without ELS. If tau with ELS drops to the level of the w/o-ELS baseline (≈0.01), the ranking benefit rests on the channel-matched equivalence Lemma 1 proves, not on the mismatched case used in the actual scalable search; if tau remains high, the paper's stabilization claim is more robust than its proof.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on Lemma 1 / Eq. 3: inserting a 1x1 ELS in place of a skip must not change the represented subnetwork. Appendix A proves only that a single convolution Conv(c_l, m, k) can be factored as Conv(c_{l+1}, m, k) after a 1x1 Conv(c_l, c_{l+1}, 1); that is, the proof applies to a linear layer whose direct input already has c_l channels. In the scalable spaces S1/S2 the skip emits c_l channels while every other choice block emits c_{l+1}; Fig. 11's caption states the post-skip feature map is half the width of the others (16 vs 32). At such a layer the left side of Eq. 3, f^o_{l+1}(x^{c_l}_l), is not a well-defined input to a shared next block whose first conv is shaped for c_{l+1} channels, and the 'iteratively apply' argument cannot pass through MB nonlinearities (expansion, depthwise, SE) because folding ELS into the next block creates a different weight tensor than the shared one. Since SCARLET-B and -C are reported to contain identity operations, including at late stages where channel counts change, the supernet may have ranked architectures that are not the architectures later trained from scratch. The only benchmark experiment validating ELS ranking (NAS-Bench-101, §6.2) deliberately adds a 1x1 projection 'to avoid channel mismatch', which sidesteps exactly this case.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses weight-sharing one-shot NAS in search spaces with variable depth, where skip connections are added to the usual MobileNetV2-style blocks. The authors observe that a vanilla scalable supernet suffers from training instability and poor subnetwork ranking, and they attribute this to feature inconsistency between the skip connection and the other choice blocks. They propose replacing the skip with a learnable 1x1 convolution, the Equivariant Learnable Stabilizer (ELS), during supernet training. They claim (Lemma 1, Eq. 3) that this stabilizer does not change the representational capacity of any subnetwork, so it can be removed after training. Experiments show lower variance in training curves, improved Kendall Tau on NAS-Bench-101 (from 0.0118 to 0.421), and a family of SCARLET models with competitive ImageNet accuracies, the best of which (SCARLET-A) reaches 76.9% top-1 accuracy. The final models are trained from scratch after the search, so the headline accuracies do not depend on the supernet proxy.","tokens_in":15085,"tokens_out":9315,"duration_ms":95273,"significance":"If the equivalence claim holds, the paper contributes a practical fix for a real failure mode in one-shot NAS with variable depth: the ELS mechanism is simple, cost-free during search, and demonstrably improves ranking on an external benchmark. The empirical backbone is generally solid: the NAS-Bench-101 ranking experiment uses ground-truth accuracies; the final ImageNet models are re-trained from scratch; and the ablations (training curves, feature similarity, constrained optimization) directly support the main narrative. The central risk is the gap between the proof of Lemma 1 and the actual search spaces, which contain layers where the skip output channel count differs from the other choices. Since the entire ranking guarantee rests on Eq. (3), this gap is load-bearing and must be closed or the search space clarified before the claims can be accepted.","major_comments":[{"comment":"Lemma 1's proof covers only a 1x1 convolution followed by a kxk convolution with matching input channels. In the search spaces S1/S2, at channel-changing layers the skip output has c_l channels while every other choice block outputs c_{l+1} channels; the Fig. 11 caption explicitly states 'the channel size of feature map after Choice 6 ... is half of others'. For these layers the right-hand side of Eq. (3), f^o_{l+1}(x^{c_l}_l), is not a well-defined input to a shared next block whose first convolution is shaped for c_{l+1} channels, and the 'iteratively apply' argument cannot pass through the MB nonlinearities (expansion, depthwise, SE) because folding ELS into the next block would require changing that block's weights to a tensor different from the shared one. Since SCARLET-B and -C are reported to contain identity operations, including at late stages where channel counts change, the supernet may have ranked architectures that are not the same as those later trained from scratch. The only direct ranking benchmark in Section 6.2 deliberately adds a 1x1 projection to avoid channel mismatch, which sidesteps exactly this case. Please either extend the equivalence proof to mismatched layers, or state that skip connections are only allowed when c_l = c_{l+1} and rerun or justify the affected experiments.","section":"Section 4.1 and Appendix A, Eq. (3)"},{"comment":"The text repeatedly calls the skip connection 'parameter-free', yet the vignette in Fig. 11 and the search-space description imply that a bare identity can be selected at layers where c_l differs from c_{l+1}. A bare identity cannot be followed by a next-layer block whose first convolution expects c_{l+1} input channels; the resulting computational graph is ill-defined. This makes it unclear how the vanilla scalable supernets in Fig. 3 were trained at all. Please specify the exact skip implementation (for example, a 1x1 projection at downsampling layers, or a restriction of skip choices to channel-matched layers) and confirm that the training curves and ranking experiments use that implementation. Without this clarification, the observed 'instability' could be an artifact of an invalid architecture rather than a genuine feature-inconsistency effect.","section":"Section 3.1 and Appendix C.1"}],"minor_comments":[{"comment":"There are several small wording issues, such as 'we are mainly concerned about finding models' (Introduction) and 'it creates a large range of perturbation' (Abstract); these should be polished.","section":"Abstract and Introduction"},{"comment":"The notation f^o_{l+1} is used for the next-layer operation on both sides of Eq. (3), but if f^o_{l+1} is a shared block that expects c_{l+1} input channels, the expression on the right side is ambiguous at mismatched layers; the paper should define the input channel dimension of f^o_{l+1} explicitly.","section":"Section 4.1, Eq. (3)"},{"comment":"The NAS-Bench-101 experiment samples 100 models for the Kendall Tau comparison; it would be helpful to report the variance across multiple seeds or at least the number of supernet training runs, since a single run at tau=0.421 could be optimistic.","section":"Section 6.2, Fig. 7"},{"comment":"Several hyperparameters in Table 7 (prm, pre, ppr, pM, pK-M) are not defined in the text or pseudocode; please add a brief description of the mutation/crossover probabilities.","section":"Appendix C.2, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper builds heavily on the authors' own FairNAS work, including the feature-similarity diagnosis and the training procedure. This is a significant amount of self-citation, but it is not circular: the ELS mechanism is clearly distinguished and the benchmark evaluation is external. The main concern is the channel-mismatch gap in the equivalence proof, which is a central correctness issue rather than a presentation point. I believe the authors can address it in revision by clarifying the search-space implementation and either proving the mismatched-layer case or restricting the claims accordingly, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading: it is the first to seriously address what happens when you add skip connections to a single-path one-shot supernet, and the proposed fix—a learnable 1x1 convolution (ELS) that stands in for the skip during training—is simple and clearly effective. The empirical story is genuinely good: training curves stabilize, the Kendall tau on NAS-Bench-101 jumps from 0.01 to 0.42, and the final SCARLET models are trained from scratch, so the headline ImageNet numbers do not depend on the supernet proxy. That is real evidence, not hype.\n\nThe soft spot is the theory. Lemma 1 claims Equivalence, not just helpful stabilization, and the proof in Appendix A only covers a 1x1 conv followed by a kxk conv with matching channel counts. But the search spaces S1 and S2 include skip connections that change channel counts—Fig. 11 explicitly shows the skip output having half the width of the other choices—and the next block's first conv is shared across all choices. The \"iteratively apply\" line in the proof does not survive contact with MB nonlinearities, SE, or consecutive skips. The NAS-Bench-101 experiment, which is the one clean ranking test, adds a 1x1 projection to avoid channel mismatch, so it sidesteps exactly the case where the proof is missing. This is not a fatal flaw: the method clearly helps even if the equivalence is inexact, and the final architectures are not trained through the supernet. But the paper overclaims when it says ELS leaves the search space unchanged. A serious referee should ask the authors to either extend the proof to the channel-mismatched case or soften the claim to \"approximate equivalence.\"\n\nAlso, the 0.6% gain over EfficientNet-B0 is a single training run with no error bars; not a big deal for this area, but worth noting.\n\nBottom line: this deserves peer review. The core idea is likely correct, the empirical work is solid, and the gaps are addressable. I would bring it to a reading group and probably cite it if I did NAS work. It just needs a more honest statement of what is proven and what is empirical.","headline":"A simple, effective stabilizer for skip connections in one-shot NAS, with a mathematical equivalence claim that is stronger than the proof supports.","tokens_in":15687,"tokens_out":4088,"would_cite":true,"duration_ms":42151,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper identifies skip connections as the destabilizing element in one-shot weight-sharing supernets and shows that replacing them during training with a removable learned 1x1 convolution restores ranking and yields a 76.9% ImageNet…","keywords":["neural architecture search","weight-sharing supernet","one-shot NAS","skip connection","feature inconsistency","equivariant learnable stabilizer","evolutionary search","image classification"],"falsifier":"A direct check is to take a layer where the skip's output has $c_l$ channels while the other operations produce $c_{l+1}\\neq c_l$ channels, feed one batch of random inputs through both the raw skip and the ELS into every next-layer operation, and compare $f^{l+1}_o(x^{c_l}_l)$ with $f^{l+1}_o(f^l_{ELS}(x^{c_l}_l))$; a difference above numerical precision at any mismatched layer would show that Lemma 1 does not cover the actual search space. Alternatively, one can compare the supernet's rank correlation on benchmark models when ELS is used only at matched-channel layers versus everywhere; a drop in the all-layers case would show the mismatch is load-bearing.","tokens_in":14553,"feed_emoji":"🤖","tokens_out":13142,"duration_ms":119513,"temperature":0.7,"pith_summary":"The paper tries to establish that weight-sharing neural architecture search can be made scalable to variable network depths by fixing how skip connections are represented inside the supernet. Earlier single-path one-shot supernets deliberately leave skip connections out, because adding them destabilizes training and destroys the supernet's ability to rank candidate models. The paper diagnoses the cause: outputs of a skip connection are much less similar to outputs of the other choices, and the resulting feature inconsistency cascades through the layers. It then proposes the Equivariant Learnable Stabilizer (ELS), a $1\\times1$ convolution with no batch-normalization or activation, trained in place of each skip and folded away afterward so that representational capacity is unchanged. With this stabilizer, supernet training converges reliably and ranking improves, and an evolutionary search yields the SCARLET models, of which SCARLET-A reaches 76.9% top-1 on ImageNet.","feed_headline":"A 1x1 convolution stabilizes one-shot NAS, finding 76.9%","feed_subtitle":"Replacing skip jumps with a removable learned 1x1 conv restores supernet ranking and yields a 76.9% ImageNet model.","key_machinery":"The load-bearing object is the Equivariant Learnable Stabilizer (ELS), a learned $1\\times1$ convolution with no batch-normalization and no activation, placed where the search space would put an identity skip. Its defining equivalence is Eq. (3): for the input $x^{c_l}_l$ to layer $l$ and every operation $o$ in layer $l+1$, $f^{l+1}_o(x^{c_l}_l)=f^{l+1}_o(f^l_{ELS}(x^{c_l}_l))$. The $1\\times1$ map satisfies this because linear convolution commutes with composition: its weights can be folded into the following convolution's weight tensor, so the stabilizer changes no representational capacity. ELS therefore gives the search a homogeneous-feature training signal while leaving the final architecture free of the extra layer.","core_discovery":"On its own terms, the paper's discovery is that the obstacle to one-shot search over variable depths is not skip connections as such but the feature heterogeneity they introduce. In a layer whose choices are inverted-bottleneck blocks and an identity skip, the skip's feature maps have dramatically lower cosine similarity with the other blocks' outputs; when the next layer samples one path, it receives inputs with different statistics depending on the previous choice, and the disturbance accumulates. Replacing the skip during supernet training with an equivariant learnable stabilizer removes the heterogeneity while preserving exactly the same representational capacity, because a $1\\times1$ convolution composed with any following convolution can be absorbed into that convolution's weights (Lemma 1). The paper reports that the stabilized supernet is a far better evaluator: on a benchmark with known ground-truth labels, rank correlation rises from near zero to about 0.42 with ELS. Searching this supernet evolutionarily produces SCARLET-A, which reaches 76.9% ImageNet top-1 at 365M multiply-adds, and the paper shows transfer results on CIFAR-10 and object detection.","pith_inferences":["Extension: the same linear-absorption argument should transfer to any search space where a candidate operation's input features can be homogenized by a linear map; one could, for example, use a learned $1\\times1$ stabilizer for variable-width choices rather than only variable-depth ones.","Extension: measuring cross-choice feature similarity early in supernet training could serve as a cheap diagnostic for whether a given search space will rank well; the paper's cosine-similarity curves make that a testable preprocessing step rather than a post hoc explanation.","Extension: the channel-mismatch worry in Eq. (3) suggests a targeted experiment: restrict ELS to layers whose channel counts match and compare final search quality; if quality does not drop, the shortcut is harmless in practice, and if it does, the proof needs a refined statement.","Extension: the ranking improvement from near zero to about 0.42 implies that weight-sharing evaluations can be made informative with a much smaller fix than retraining every subnetwork, which may make one-shot search competitive as a cheap ranking oracle for broader model-selection tasks."],"forward_implications":["Skip connections can be included in one-shot search spaces without destabilizing supernet training, so searched models can differ in depth rather than only in per-cell choices.","Because ELS is removed after training, final architectures carry no extra parameter cost from the stabilizer, so the accuracy gain is not bought with added runtime.","Better supernet ranking makes the evolutionary search stage more reliable; the paper's ablation puts the gain from ELS in ranking, not in the search algorithm.","The reported result implies that competitive mobile-scale architectures can be found without proxy tasks at modest compute: SCARLET-A's 76.9% top-1 exceeds a 390M-FLOPs strong baseline using 365M FLOPs.","Non-equivariant stabilizers (e.g., adding ReLU) overestimate subnetworks, so the equivalence condition is not a formality but an experimentally visible requirement."],"supporting_citations":[{"why":"It supplies the single-path one-shot supernet formulation that the paper extends by adding skip connections.","marker":"[12]"},{"why":"It provides the competing one-shot training scheme and the observation that cross-block feature similarity is what makes single-path training work.","marker":"[6]"},{"why":"It contributes the inverted-bottleneck choice space with skip connections from which S1 and S2 are built.","marker":"[2]"},{"why":"It defines the scaled-model comparison point whose reported results the SCARLET models are measured against.","marker":"[29]"},{"why":"It supplies ground-truth accuracies used to measure the supernet's ranking correlation with and without ELS.","marker":"[33]"},{"why":"It provides the training recipe and mobile target objectives used to train and position the final SCARLET models.","marker":"[28]"},{"why":"It defines the inverted-residual blocks that constitute the non-skip choices in the search space.","marker":"[24]"}],"fun_headline_variants":["Taming skip-connection chaos yields 76.9% NAS model","1x1 conv stabilizes variable-depth supernet, hits 76.9%","Calming feature shifts in one-shot NAS scores 76.9%","Smoothing skip paths in NAS supernet achieves 76.9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Eq. (3) holds at every layer of the supernet, including layers where the skip carries a different number of output channels into the next block than the other choices do; the paper's proof covers only a $1\\times1$ convolution followed by a $k\\times k$ convolution with matching input channels, so a channel mismatch would leave the trained supernet evaluating a different architecture class than the final models.","fun_headline_variants_meta":{"raw":{"variants":["Taming skip-connection chaos yields 76.9% NAS model","1x1 conv stabilizes variable-depth supernet, hits 76.9%","Calming feature shifts in one-shot NAS scores 76.9%","Smoothing skip paths in NAS supernet achieves 76.9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1625,"prompt_tokens":988,"completion_tokens":637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":566}},"tokens_in":604,"tokens_out":637,"duration_ms":5495,"temperature":1.0,"reasoning_tokens":566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:58:47.042120+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is to take a layer where the skip's output has $c_l$ channels while the other operations produce $c_{l+1}\\neq c_l$ channels, feed one batch of random inputs through both the raw skip and the ELS into every next-layer operation, and compare $f^{l+1}_o(x^{c_l}_l)$ with $f^{l+1}_o(f^l_{ELS}(x^{c_l}_l))$; a difference above numerical precision at any mismatched layer would show that Lemma 1 does not cover the actual search space. Alternatively, one can compare the supernet's rank correlation on benchmark models when ELS is used only at matched-channel layers versus everywhere; a drop in the all-layers case would show the mismatch is load-bearing.","supporting_citations":[{"cited_title":"In: ICLR (2019)","cited_arxiv_id":null,"evidence_quote":"It contributes the inverted-bottleneck choice space with skip connections from which S1 and S2 are built."},{"cited_title":"In: ICML (2019)","cited_arxiv_id":null,"evidence_quote":"It defines the scaled-model comparison point whose reported results the SCARLET models are measured against."},{"cited_title":"In: ICML","cited_arxiv_id":null,"evidence_quote":"It supplies ground-truth accuracies used to measure the supernet's ranking correlation with and without ELS."},{"cited_title":"In: CVPR (2019)","cited_arxiv_id":null,"evidence_quote":"It provides the training recipe and mobile target objectives used to train and position the final SCARLET models."},{"cited_title":"In: CVPR","cited_arxiv_id":null,"evidence_quote":"It defines the inverted-residual blocks that constitute the non-skip choices in the search space."}],"review_version":1}