{"id":"6ce13c11-331c-41d8-9767-35d82d34465e","arxiv_id":"2411.09475","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-phase training algorithm alternating droppath steps with frozen-path steps gives modest accuracy improvements on small image datasets, with inconsistent ImageNet results.","lead":"This paper proposes ResidualDroppath, a training schedule that alternates between droppath updates and updates that train only the dropped paths while freezing the rest, to encourage feature reuse in residual networks. Tests on ResNet50 and ResNet50d show small accuracy gains on CIFAR-10 and MNIST, and mixed single-seed results on ImageNet.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.1's Euclidean-distance feature similarity is not scale-normalized, so the claimed 'feature reuse deficiency' in Fig. 3 may be an artifact of activation magnitude rather than a real reuse failure.","rationale":"The reader's CONDITIONAL verdict is sensible. My stress-test identifies a sharper soft spot inside the toy analysis: the one quantitative metric (Euclidean distance, Fig. 3) is not normalized, so the observed non-monotonic 'similarity' may be an artifact. This is load-bearing because the algorithm is explicitly designed to fix a deficiency diagnosed only in that toy experiment; if that diagnosis is an artifact, the explanatory claim of the paper is not supported, even if the CIFAR10 numbers survive. I do not move the verdict because there is still a credible 5-seed CIFAR10 improvement that could make the method a useful empirical heuristic, and a normalized re-analysis could in principle confirm the qualitative observation. The reader's weakest_assumption focused on toy-to-ResNet transferability; I agree that direct feature-reuse measurement in ResNet is missing, but the more urgent check is whether the toy experiment itself measures what it claims. Hence 'partial' agreement. A scale-normalized CKA re-run of Fig. 3 is cheap and decisive.","tokens_in":17072,"tokens_out":9649,"duration_ms":94818,"concrete_test":"Recompute the Section 3.1 similarity analysis on the same 32-layer MLP using centered, per-layer standardized features: subtract each node's mean over the 2,500 grid inputs and divide by its standard deviation, then recompute the Figure 3 pairwise Euclidean distances and a scale-invariant measure such as linear CKA between all layer pairs. If the non-monotonic similarity pattern disappears or becomes monotone, the 'feature reuse deficiency' is an artifact of activation scale and the paper's motivating diagnosis fails. If it persists under normalization, the toy observation is robust, and the remaining question becomes whether a ResNet trained with ResidualDroppath actually increases layer-wise feature reuse as measured by CKA.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is the Section 3.1 diagnosis that vanilla residual connections cause inefficient feature reuse across non-consecutive layers, which is then used to justify the ResidualDroppath design. The quantitative support is Figure 3: the Euclidean distance between the 2D input and each layer's features, plus pairwise similarity heatmaps. This metric is not normalized; it conflates geometric similarity with feature scale. In a 32-layer MLP with residual adds and GELU nonlinearities, activation norms can shrink or grow substantially across layers, so a layer whose feature directions are close to the input can nonetheless appear 'far' if its norm is large, and vice versa. The reported rise-and-fall of similarity over depth may therefore track magnitude changes rather than repeated transformations. The contour plots are qualitative and are not supplemented by a null model or a normalized similarity statistic. Consequently, the paper does not establish, even in the toy setting, that residual connections impede feature reuse; the only direct test of the mechanism in ResNet is likewise absent. If the premise is an artifact, the central claim that ResidualDroppath 'facilitates feature reuse' is unsupported, although the empirical CIFAR10 gain would remain a standalone training heuristic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that vanilla residual connections limit feature reuse across non-consecutive layers, supports this claim with visualizations of a toy MLP trained on a 2D spiral dataset, and proposes ResidualDroppath, a training procedure that alternates between droppath-style iterations and iterations that freeze the non-dropped residual paths. The method is evaluated on MNIST, CIFAR-10, and ImageNet-1k with ResNet50 and ResNet50d. The authors report the strongest gains on CIFAR-10 (around +1.3% top-1 accuracy over both standard training and droppath), small gains on MNIST, and mixed or marginal results on ImageNet-1k, where the ResNet50 result is worse than droppath and ResNet50d improves only slightly.","tokens_in":17281,"tokens_out":5191,"duration_ms":49538,"significance":"If the CIFAR-10 result is reproducible, ResidualDroppath offers a simple, training-time accuracy improvement for residual networks with no change at inference. The algorithm is clearly specified and the experimental setup is mostly standard. However, the paper's central mechanistic claim—that the method improves feature reuse—is not directly tested in the ResNet experiments, and the toy-analysis evidence for the initial deficiency is weakened by the use of a scale-sensitive similarity measure. The ImageNet results lack error bars and are inconsistent across architectures. The paper is honest about its limitations, but the gap between the claimed mechanism and the empirical evidence is the main weakness.","major_comments":[{"comment":"The Euclidean-distance similarity measure used in Figure 3 is not scale-normalized. Activation magnitudes can vary substantially across layers in a 32-layer residual MLP, so a rise-and-fall in Euclidean distance may track changes in feature norm rather than a genuine absence of feature reuse. The claim that vanilla residual connections produce a 'feature reuse deficiency' is therefore not established by the presented evidence. Please recompute the analysis with a scale-invariant measure (e.g., cosine similarity or centered kernel alignment) and, ideally, compare against a non-residual baseline or a null model.","section":"Section 3.1, Figure 3"},{"comment":"The ImageNet-1k results are from a single seed with no error bars. For ResNet50, ResidualDroppath (75.79 top-1) is worse than Droppath (76.11), and for ResNet50d the gain over Droppath is only 0.08 top-1. The abstract and introduction claim 'significant improvement' on MNIST and CIFAR-10; the MNIST differences (0.06% with a standard deviation of 0.05%) are within noise. The only statistically solid result is CIFAR-10. Please either restrict the claims to CIFAR-10 or provide multi-seed ImageNet results with error bars that support the stronger conclusion.","section":"Section 4.2, Table 1"},{"comment":"The second iteration (M % 2 == 1) reuses the mask variable 'maskd' sampled in the previous iteration, but this is not stated in the algorithm or the text. More importantly, the paper provides no measurement of feature reuse in the trained ResNet models, so the mechanism claimed in the title and Section 1—that ResidualDroppath 'facilitates feature reuse across multiple layers'—remains untested. Please add a direct feature-reuse analysis (e.g., layer-wise similarity or CKA on the actual trained models with and without ResidualDroppath) to support the mechanistic interpretation.","section":"Algorithm 1, line 16"},{"comment":"The toy analysis trains MLPs with GELU activations on a 2D spiral dataset and interprets layer-wise feature similarity as evidence for the behavior of deep convolutional residual networks on natural images. This transfer is assumed but not demonstrated. Without evidence that the same pattern appears in ResNet blocks, the motivation for the algorithm is weakened. Please either provide a similar analysis for a small ResNet on a natural-image subset or explicitly reframe the toy analysis as illustrative rather than diagnostic.","section":"Section 3.1"},{"comment":"The drop rate (0.1) and the alternation period (every two iterations) are fixed hyperparameters, and no sensitivity analysis is reported. Since these are the only free parameters introduced by the method, their robustness is load-bearing for the reproducibility of the claimed gains. Please include an ablation over at least a small range of drop rates and alternation periods.","section":"Section 4.1"}],"minor_comments":[{"comment":"The baseline description contains the typo 'ResidaulDroppath'; should read 'ResidualDroppath'.","section":"Section 4.1"},{"comment":"The text says 'In this session' but should say 'In this section'.","section":"Section 3.1"},{"comment":"The dataset column header uses 'Imagenet-1k' while the body text uses 'ImageNet-1k'; please be consistent.","section":"Table 1"},{"comment":"The caption states that 'similarity decreases and increases' but does not state whether higher heatmap values indicate higher or lower similarity; clarify the color scale.","section":"Figure 3"},{"comment":"Reference [1] is marked 'under review' and is not a stable archival reference; please replace it with a citable version or remove it.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The CIFAR-10 result is the strongest part of the paper and, if confirmed, could be a useful empirical contribution. The main weakness is the disconnect between the asserted feature-reuse mechanism and the evidence: the toy analysis is not scale-normalized and the ResNet experiments do not measure feature reuse. These issues are fixable in revision. I would also encourage the editor to require multi-seed ImageNet results or temper the claims accordingly. The reference list is unusually long and contains many peripheral items; the authors should trim it to focus on the relevant literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is a two-iteration training schedule for residual nets: a droppath forward pass followed by an update that trains only the dropped paths while freezing the rest. That alternation is genuinely new as far as I can tell—it's not in the stochastic-depth, dropout, or DenseNet line. The paper is honest too: it reports the ImageNet ResNet50 regression as a limitation and doesn't oversell the mixed results.\n\nThe strongest evidence is the CIFAR-10 ResNet50d result (90.84 ± 0.18 vs 89.55 ± 0.25 for standard), with non-overlapping error bars. That's a real, if modest, gain. The algorithm is simple and clearly specified, and the authors didn't tune parameters to match the test sets.\n\nThe soft spots are substantial. The motivating analysis in Section 3.1 is the weakest link: the Euclidean-distance similarity in Figure 3 is scale-sensitive, so the rise-and-fall pattern could just track activation norms rather than any feature-reuse deficiency. The contour plots are suggestive but not quantified, and there's no null model. The leap from a 32-layer GELU MLP on a 2D spiral to ResNet50 on ImageNet is unsupported—no direct measurement of feature reuse in the real experiments is provided. MNIST gains are within noise, and ImageNet is single-seed with a regression on the base ResNet50. The drop rate for ResidualDroppath is never stated, which is a minor but easy-to-fix omission. The abstract's 'significant improvement' on MNIST overstates what Table 1 actually shows.\n\nWho's this for? Someone working on cheap training-time tweaks for residual networks. It's a small contribution, not a paradigm shift. If the CIFAR-10 effect replicates, it's a useful data point even if the feature-reuse story is wrong.\n\nThis deserves a serious referee, not a desk reject—the new combination and the positive result justify review time. In its current form I'd push for major revision: multi-seed ImageNet or a clear explanation of why that's not feasible, the drop-rate disclosure, an ablation of the alternation period, and either a normalized similarity analysis or a direct test of the mechanism in the ResNet setting. The empirical result might stand on its own; the story needs to catch up.","headline":"New training trick, one solid CIFAR-10 number, and a motivation that doesn't survive close reading—worth a look but needs work.","tokens_in":17857,"tokens_out":2944,"would_cite":false,"duration_ms":28811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that alternating droppath iterations with iterations that train only the dropped residual paths makes residual networks reuse features as identity mappings and improves image-classification accuracy.","keywords":["feature reuse","residual connections","droppath","stochastic depth","image classification","identity mapping","training algorithm","information retention"],"falsifier":"Train ResNet50d on CIFAR-10 with and without ResidualDroppath under matched compute (same total gradient updates) and measure the average cosine similarity between each block's input and output on the test set. If the ResidualDroppath model does not show a higher fraction of blocks whose output is close to their input compared with the droppath baseline, the claimed identity-reuse mechanism is not what drives the accuracy improvement.","tokens_in":16804,"feed_emoji":"🧩","tokens_out":5962,"duration_ms":50303,"temperature":0.7,"pith_summary":"Residual connections are widely credited with letting deep networks reuse features, but this paper argues that vanilla residual training does not actually encourage the network to pass features through unchanged. Using a toy spiral dataset and a visualization tool, the author shows that MLPs with residual connections tend to transform features repeatedly, producing similar distributions at non-adjacent layers and losing information. To counter this, the paper proposes ResidualDroppath, a training scheme that alternates between droppath iterations (which randomly drop some residual paths, forcing the model to rely on identity shortcuts) and iterations that train only the newly dropped paths while freezing the undropped ones. On CIFAR-10, MNIST, and ImageNet-1k, applying this scheme to ResNet50 and ResNet50d improves Top-1/Top-5 accuracy in most settings, with the largest gain on CIFAR-10. The paper's own limitation is that ResNet50 on ImageNet-1k does not beat the plain droppath baseline.","feed_headline":"Two-step training scheme lifts ResNet accuracy on CIFAR-10","feed_subtitle":"Training-time alternation improves feature reuse with no change at inference.","key_machinery":"The central object is the ResidualDroppath training loop, which alternates two iteration types per block. Even iterations apply a droppath mask $m_d$ so that the output is $x + F(x) \\cdot m_d$, forcing dropped positions to pass the input $x$ unchanged. Odd iterations compute $x + F(x).\\mathrm{detach}() \\cdot m_d + F(x) \\cdot (1 - m_d)$, which detaches (freezes) the kept paths and trains only the dropped ones. This two-step alternation is designed to teach the residual branch $F$ to produce updates that leave features reusable in their identity form.","core_discovery":"Residual connections alone do not guarantee efficient feature reuse: even with a skip connection, intermediate layers keep applying transformations to features that could be passed through in identity form, and similar feature distributions reappear in non-consecutive layers, which risks information loss. The paper's proposed ResidualDroppath addresses this by alternating two training iterations. In the first, droppath randomly drops a subset of the residual path, so the block output reduces to the identity (the input) for dropped positions and the network must learn under forced feature reuse. In the second, the paths that were not dropped are frozen and the dropped parts are trained, so those residual branches learn outputs that are compatible with identity reuse. The result is a training-time modification that improves image-classification accuracy for ResNet50/ResNet50d on CIFAR-10 and MNIST, and for ResNet50d on ImageNet-1k, with no change at inference.","pith_inferences":["The toy-MLP analysis uses Euclidean distance between input and layer features as a proxy for feature reuse; a direct test on natural-image ResNet activations (e.g., CKA or cosine similarity between block input and output) would let us check whether the proposed mechanism really increases identity reuse in the trained ResNets.","The second iteration doubles the number of forward/backward passes per block; if the gains come mainly from the extra optimization steps or from the detach operation alone, a simpler scheme might produce the same result. Comparing ResidualDroppath against an equal-compute droppath baseline would isolate the alternation's contribution.","The interpretation of dataset information content—why CIFAR-10 gains are large, MNIST small, and ImageNet mixed—could be tested by ablating the number of classes or images per class while keeping the architecture fixed.","The paper's insight that residual training may over-transform features connects to a broader design space of training-time regularizers that encourage shortcut use, beyond the specific droppath mask."],"forward_implications":["If ResidualDroppath works as claimed, any residual network can be trained with a simple two-stage alternation at no extra inference cost, and the accuracy gains on CIFAR-10 (~1.3 points over the standard ResNet50d baseline) indicate the mechanism is most effective when per-class information is abundant.","The method suggests that feature reuse is not an automatic property of skip connections but a learnable behavior, so training that explicitly forces identity reuse should be part of a residual network's optimization.","Because the algorithm is architecture-agnostic in spirit, it may extend to other residual architectures (e.g., transformers) as the paper notes as future work.","On ImageNet-1k, the gains are smaller and the ResNet50 case loses to droppath, so the benefit appears to depend on the information richness of the dataset."],"supporting_citations":[{"why":"Supplies the residual connection architecture (ResNet) that the method modifies and evaluates.","marker":"[38]"},{"why":"Supplies droppath (stochastic depth), the technique the first iteration uses and the baseline it is compared against.","marker":"[46]"},{"why":"Supports the paper's premise that residual connections benefit from identity mappings.","marker":"[39]"},{"why":"Frames feature reuse as the central benefit of dense connectivity, which motivates the analysis.","marker":"[47]"},{"why":"Provides the visualization tool used to analyze feature reuse in the toy MLP experiments.","marker":"[44]"},{"why":"Supplies the ImageNet-1k dataset used in the large-scale evaluation.","marker":"[88]"},{"why":"Supplies the CIFAR-10 dataset where the largest gains are observed.","marker":"[58]"},{"why":"Supplies the MNIST dataset used in the small-scale evaluation.","marker":"[59]"},{"why":"Supplies the default training hyperparameters and model implementations used in the experiments.","marker":"[112]"}],"fun_headline_variants":["ResidualDroppath: training trick boosts ResNet accuracy","Alternating droppath and fine-tuning improves feature reuse","Training-time tweak forces residual nets to reuse features","Two-phase training lifts ResNet accuracy on multiple benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the feature-reuse deficiency observed in small MLPs trained on a 2D spiral toy dataset generalizes to deep convolutional residual networks trained on natural images.","fun_headline_variants_meta":{"raw":{"variants":["ResidualDroppath: training trick boosts ResNet accuracy","Alternating droppath and fine-tuning improves feature reuse","Training-time tweak forces residual nets to reuse features","Two-phase training lifts ResNet accuracy on multiple benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1820,"prompt_tokens":892,"completion_tokens":928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":861}},"tokens_in":508,"tokens_out":928,"duration_ms":8813,"temperature":1.0,"reasoning_tokens":861,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:35:46.153425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ResNet50d on CIFAR-10 with and without ResidualDroppath under matched compute (same total gradient updates) and measure the average cosine similarity between each block's input and output on the test set. If the ResidualDroppath model does not show a higher fraction of blocks whose output is close to their input compared with the droppath baseline, the claimed identity-reuse mechanism is not what drives the accuracy improvement.","supporting_citations":[{"cited_title":"From Tinkering to Engineering: Measurements in Tensorflow Playground","cited_arxiv_id":"2101.04141","evidence_quote":"Provides the visualization tool used to analyze feature reuse in the toy MLP experiments."},{"cited_title":"Pytorch image models","cited_arxiv_id":null,"evidence_quote":"Supplies the default training hyperparameters and model implementations used in the experiments."}],"review_version":1}