{"id":"8899632b-5289-4fbe-b3a6-734d94685c66","arxiv_id":"2504.16290","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ResNet18's residual stream can build scale invariance by summing a smaller-scale feature from the block input with a larger-scale feature from the pre-sum output, and ablating these channels mildly impairs scale-robust image classification.","lead":"Using feature visualization and ablations, this paper finds channels in ResNet18 where the residual connection adds a small-scale copy of a feature from the block input to a large-scale copy from the block output, creating a scale-invariant representation. It is a concrete step toward understanding how convolutional networks, and possibly biological vision, achieve scale-robust object recognition.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scale-invariance criteria in Eqs. (1)-(2) compare responses to two separately optimized feature visualizations, not to scale-transformed versions of the same image, so the central claim that residual summation computes scale invariance is not actually tested.","rationale":"The paper is an honest, exploratory study, and the reader's conditional verdict is appropriate. The most load-bearing weakness is not the ablation statistics but the logical gap in the channel-selection criteria: feature visualizations are optimization artifacts, not controlled stimuli, so Eq. (2)'s ratio of responses to two different FZs cannot certify scale invariance. The central claim requires that Post_c respond similarly to scaled versions of the same natural image, and that In_c and Pre_c carry the same feature at different scales. The paper explicitly defers natural-image tests to future work, which is commendable transparency, but that deferral leaves the main mechanistic assertion unsupported. The proposed concrete test directly settles the concern: if Post_c responses are not invariant under the same scale transforms used in the ablation, the identified channels cannot be interpreted as scale-invariant and the behavioral interpretation collapses. If the test passes, the paper's central observation is substantially strengthened. The reader already identified the same assumption, so agreement is full; the verdict remains conditional pending the natural-image check.","tokens_in":6885,"tokens_out":5325,"duration_ms":58740,"concrete_test":"For each criteria-passing channel in blocks 2.1 and 3.1, take the top-50 center-neuron-activating ImageNet validation images for Post_c. Apply the same scale transform family used in Section 4 (center-crop to 10-50% of 256 pixels, then resize to 224) to each image, and compute Post_c(S_p(I))/Post_c(I) for p=10,...,50. Also compute In_c and Pre_c activation curves across p for images that activate Post_c, and check whether In_c peaks at smaller p than Pre_c. If Post_c responses vary systematically with p (e.g., monotone decrease or increase) or In_c/Pre_c do not show the predicted peak ordering, Eqs. (1)-(2) do not identify scale invariance and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3's criteria do not establish scale invariance of the post-sum channel. Eq. (2) only requires Post_c to respond comparably to two independently optimized feature visualizations, \\hat X^{In}_c and \\hat X^{Pre}_c; it never applies a scale transform S to a fixed stimulus and checks that Post_c's response is approximately unchanged. Eq. (1) only shows that Pre_c's response to an enlarged version of In's feature visualization is larger than to the original; this is a preference for larger-scale features, not evidence that Pre_c encodes a larger-scale copy of the same feature or that the representation is scale-equivariant. The central mechanistic claim, that the residual sum adds a smaller-scale copy from In to a larger-scale copy from Pre to obtain a scale-invariant Post, therefore rests on an untested proxy. The paper's own Section 5 limitation ('criteria ... only via a channel's response to the FZs') makes this explicit, but the limitation is load-bearing because the same channel set is then used in the ablation experiments. If Eq. (2) is satisfied by broadly tuned channels that are not scale-invariant on natural images, both the identified channels and the behavioral link would be artifacts of feature-visualization optimization, especially given the layer-dependent jitter settings reported in Section 7.1.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates scale invariance in the residual stream of ResNet18. For each residual block, the authors obtain feature visualizations (FZs) for the block input (In), pre-sum (Pre), and post-sum (Post) layers, and propose two criteria (Eqs. (1)-(2)) to identify channels whose Post activation appears scale invariant, with In coding a smaller-scale copy and Pre coding a larger-scale copy. Across blocks 1.1 to 4.1, they find criteria-passing channels concentrated in intermediate blocks (23 in 2.1, 46 in 3.1). They then ablate these channels and report that scale-robust ImageNet accuracy degrades more than when random non-passing channels are ablated. The paper concludes that the residual stream computes scale invariance via element-wise summation of scale-equivariant representations, and tentatively suggests a behavioral role.","tokens_in":7137,"tokens_out":2742,"duration_ms":28673,"significance":"If the central claim were established, this would be a valuable mechanistic finding: it would extend invariance-circuit analysis from InceptionV1 to ResNet, identify a concrete computation in the residual stream, and connect a neural-level property to behavior through ablation. The paper is honest about its tentative nature and makes code available. However, the evidence as presented is indirect: the selection criteria use feature visualizations rather than scaled natural images, the ablation effect is small and lacks significance testing, and the results rely on hyperparameter choices that are not fully controlled. These issues currently leave the central claim unproven, though the methods are sufficiently concrete that targeted additional experiments could substantially strengthen the paper.","major_comments":[{"comment":"The criteria do not directly test scale invariance of the Post channel. Eq. (2) only requires that Post_c responds comparably to two independently optimized feature visualizations, Xhat_In and Xhat_Pre; it never applies the scale transform S to a fixed stimulus and checks that the response of Post_c is approximately unchanged. Eq. (1) only shows that Pre_c's response to an enlarged version of In's FZ is larger than to the original, which is a preference for larger-scale features, not evidence that Pre_c encodes a larger-scale copy in an equivariant sense. The paper's own Section 5 limitation ('criteria ... only via a channel's response to the FZs') acknowledges this, but the limitation is load-bearing because the same channel set is used in the ablation experiments. I request an additional validation: take fixed natural images (or the In-channel FZs), apply S, and check directly that Post_c's response is approximately scale-invariant while In and Pre responses shift appropriately. Without such validation, the identified channels may be artifacts of feature-visualization optimization rather than true scale-invariant computations.","section":"Section 4, ablation analysis"},{"comment":"The ablation evidence is not statistically supported. Figure 2 reports only mean ratios with standard error bars; no significance test (e.g., permutation test or bootstrap confidence interval) is performed for the claim that scale-transformed ratios are below the no-scale ratio. Moreover, the matching constraint for block 3.1 was relaxed (random ablation accuracy can be up to 1% greater in the no-scale condition), which may confound the comparison because the two channel sets are not matched in no-scale damage. I request reporting absolute top-1 accuracies for both ablation conditions, formal significance tests across the scale sweep, and a sensitivity analysis to the matching threshold. Without these, the conclusion that scale-invariant channel ablation disproportionately damages scale-robust accuracy is not established.","section":"Section 7.1, Feature Visualization Regularization"},{"comment":"The layer-dependent jitter settings create a potential confound. Feature visualizations are optimized with jitter omitted for blocks 1.1 and 2.0, jitter=4 for 2.1 Pre and the entire 3.0 block, and jitter=16 otherwise. Because the FZ optimization landscape changes with depth and jitter, the criteria-passing channel set may partly reflect which channels are easily optimizable at a given jitter rather than a genuine scale-invariant property. I request a robustness check: re-run the channel search with at least one alternative jitter setting (or a fixed jitter across all blocks) and report the overlap of passing channels, or validate the criteria on natural images so that the FZ optimization step is not the sole evidence for neural properties.","section":"Section 4, Eq. (3)"}],"minor_comments":[{"comment":"In Eq. (3), the subscript s_i is not defined; presumably it indexes the random ablation trial. Please define it.","section":"Section 4, Fig. 2"},{"comment":"The caption of Fig. 2 states 'mean ratios of top-1 ImageNet validation accuracy between the two ablation conditions' but does not explain that the blue line is the no-scale ratio. Please make the baseline explicit in the caption.","section":"Abstract and Section 6"},{"comment":"The abstract and conclusion use more assertive language ('how the residual stream computes scale invariance', 'these findings are crucial') than the 'tentative' framing in the Discussion. Please align the language with the strength of the evidence.","section":"Section 2.2"},{"comment":"The description of feature visualization says 'near-identical regularizations as prescribed by [14]' without listing the exact regularization terms; since the paper depends on FZ quality, a complete list of hyperparameters (beyond jitter) would aid reproducibility.","section":"General"},{"comment":"There are occasional spacing issues in the text (e.g., 'P re', 'P ost') and undefined notation like 'FZ' is used before first definition. A copyedit pass would improve readability.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is genuinely exploratory and the authors disclose their limitations, which I appreciate. However, the central mechanistic claim is currently supported only by a proxy criterion, and the behavioral ablation lacks inferential statistics. The revision path is clear: validate with scaled natural images, add significance tests, and report absolute accuracies. If these additions are made, the paper could be a solid contribution to a workshop or a niche journal; as it stands, the conclusions outrun the evidence. I would not recommend rejection if the authors are willing to add the requested experiments, because the question is interesting and the method is concrete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: the paper's central claim—that residual summation builds scale invariance in ResNet18—isn't actually tested by its own criteria. Eq. (2) only checks that the post-sum channel responds comparably to two separately optimized feature visualizations, one for the block input and one for the pre-sum output. It never applies a scale transform to a fixed image and checks that the response holds. So \"scale invariant\" is a proxy derived from FZ responses, not a measured property on scaled natural images. The paper admits this in the limitations, but the same proxy-selected channels are then used in the ablation, so the admitted limitation is load-bearing.\n\nWhat's genuinely new and worth credit: prior equivariance work was done on InceptionV1, which has no residual stream. This paper is the first to look at the residual stream specifically and report a consistent pattern: for some channels, the block input likes a smaller-scale texture, the pre-sum output likes a larger-scale version, and the post-sum responds to both. That pattern is visually plausible in the FZ grids, especially in block 3.1. The ablation is an honest attempt to link these channels to behavior, and the effect, though small, is consistent across scale percentages. The writing is clear and the hedging is appropriate; the limitations section is upfront.\n\nWhere it's soft: the criteria are not sufficient for scale invariance. A channel that responds to two different FZs could be broadly tuned or optimized to both, not invariant to scale. The ablation effect is small, shown as ratios with standard errors and no significance test; and the matching baseline has an ad hoc constraint for block 3.1. There's also a multiple-comparisons issue—they search all channels in several blocks and report percentages without correction. The jitter values change by layer, which may affect FZ quality. None of this is fatal to the overall observation, but it means the title and abstract claim more than the evidence supports.\n\nWho this is for: people working on mechanistic interpretability of residual networks, and anyone thinking about scale invariance in convolutional nets. It's the kind of paper that a good referee could push into a more defensible shape: use scaled natural images or scaled versions of the same FZ to define invariance, add significance tests, and tighten the language. It deserves a serious referee, not a desk reject, but it needs major revision before the central claim is convincing.","headline":"The paper's own criteria don't actually test scale invariance—they compare responses to two different feature visualizations—so the central mechanistic claim is overreaching, though the study is honest, the residual-stream observation is new, and it deserves a serious referee.","tokens_in":7645,"tokens_out":2595,"would_cite":false,"duration_ms":24284,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ResNet18's residual stream computes scale invariance by summing small- and large-scale copies of the same feature.","keywords":["mechanistic interpretability","scale invariance","scale equivariance","residual stream","ResNet18","feature visualization","ablation study","object recognition"],"falsifier":"Measure the post-sum responses of the criteria-passing channels in blocks 2.1 and 3.1 to natural images that contain the same feature at several controlled scales, for instance by rescaling each channel's top-activating ImageNet validation images by factors from 0.5x to 2x and recording the center-neuron activation curve; if the post-sum channel's activation is not roughly flat across that range within the 3/2 band while the input and pre-sum channels respond to the small and large ends respectively, the mechanism fails, and so does the ablation-based link to scale-robust behavior.","tokens_in":6668,"feed_emoji":"🔍","tokens_out":7816,"duration_ms":67382,"temperature":0.7,"pith_summary":"The paper asks where scale invariance comes from in a convolutional network that lacks the hand-designed invariance circuitry found in earlier interpretability studies. It claims that ResNet18's residual stream computes scale invariance directly: within a block, the same channel holds a smaller-scale copy of a feature at the block input and a larger-scale copy at the pre-sum output, and their element-wise addition yields a post-sum channel that responds to both scales. The evidence is a pair of quantitative criteria applied to feature visualizations, which flag such channels in blocks 1.1, 2.0, 2.1, and 3.1, most densely in the intermediate blocks 2.1 and 3.1. Ablating these channels degrades ImageNet accuracy under scale transformation more than ablating matched random channels, which the paper reads as a causal link from these neurons to scale-robust recognition. The author is careful to call the findings tentative, since the criteria rest on feature visualizations rather than scaled natural images.","feed_headline":"ResNet18's skip connections compute scale invariance","feed_subtitle":"The residual stream appears to sum small- and large-scale copies of features into scale-invariant outputs.","key_machinery":"The load-bearing object is the residual stream's summation node: in each ResNet block, the block input In bypasses the block's convolutions and is added element-wise to the pre-sum output Pre, yielding $\\mathrm{Post}=\\mathrm{ReLU}(\\mathrm{Pre}+\\mathrm{In})$. The paper's two criteria treat this node as a scale-invariance factory: the first requires that scaling up the input channel's feature visualization drive the pre-sum channel's positive activation upward, and the second requires the post-sum channel to respond to the input and pre-sum feature visualizations within a factor of 3/2 of each other. Together they single out channels where In encodes the smaller scale, Pre the larger scale, and the sum is scale-invariant. The ablation machinery then mean-substitutes the selected channels' post-sum activations across all spatial positions and compares the accuracy loss under increasing center-crop scaling against randomly selected non-passing channels.","core_discovery":"On the paper's own terms, the discovery is that scale invariance need not be assembled by a chain of hand-wired equivariant circuits, as previously found in InceptionV1; in ResNet18 it appears to emerge from the residual stream's summation operation itself. For a set of channels in blocks 1.1, 2.0, 2.1, and 3.1, the block input layer In holds a smaller-scale rendering of a feature, the pre-sum layer Pre holds a larger-scale rendering of the same feature, and the post-sum output Post, which is ReLU applied to Pre plus In, responds nearly equally to both. The paper identifies these channels with two inequalities: enlarging the input channel's feature visualization must boost the pre-sum channel's positive activation, and the post-sum channel's activation on the two feature visualizations must agree within a factor of 3/2. It then shows that mean-ablating only the post-sum channels of these sets damages top-1 accuracy on scale-transformed ImageNet images more than ablating matched random channels, while the two sets are comparable at the no-scale baseline. The author states plainly that these results are tentative and that the criteria test only feature-visualization responses, not scaled natural images.","pith_inferences":["A sharper test of the mechanism would replace the residual addition with concatenation in a matched-width network: the dense clustering of scale-invariant channels should disappear, and the scale-robustness gap between the two architectures should track that loss.","The neuroscience corollary is directly testable: record single-unit latencies in a scale-invariant ventral-stream population while presenting small versus large versions of the preferred feature; the bypass hypothesis predicts faster onsets for the smaller scales.","The 3/2 tolerance band in the second criterion would also admit channels that simply respond weakly to both feature visualizations, so requiring a high absolute post-sum activation and checking invariance across a continuous scale range (say 0.5x to 2x) would pin the claim down more tightly.","If superposition hides additional scale-equivariant features in the residual stream, as the paper suggests, then a dictionary-learning search over the block's activations should find a larger set of such features, and ablating that full set should produce a bigger behavioral effect than ablating the visible channels alone."],"forward_implications":["Scale-invariant recognition in ResNet18 is at least partly built by the residual stream itself, not only by the feed-forward hierarchy of filters.","Ablating the scale-invariant channels (23 in block 2.1, 46 in block 3.1) cuts scale-transformed top-1 accuracy at every tested center-crop percentage from 10 through 50 relative to matched random ablations.","Architectures without skip connections lack this summation point, so any scale invariance they exhibit must arise through a different mechanism, giving a concrete architectural difference to test.","The same two-criteria search can be run across the rest of the ResNet family and on recurrent weight-sharing networks, where the identical feature can be detected at multiple scales across timesteps.","If bypass connections in biological visual systems compute scale invariance, then scale-invariant deep ventral-stream neurons should respond faster to smaller-than-preferred scales than to larger ones."],"supporting_citations":[{"why":"Prior discovery of scale equivariance in InceptionV1; supplies the equivariance-to-invariance framing that this work transfers to ResNet18's residual stream.","marker":"[17]"},{"why":"Defines the feature-visualization method that generates the images used by the paper's two scale-invariance criteria.","marker":"[14]"},{"why":"Introduces the ResNet architecture and its residual stream, the object of study here.","marker":"[8]"},{"why":"Provides the ImageNet dataset and validation set used for the natural-image checks and all top-1 accuracy measurements.","marker":"[2]"},{"why":"Supplies the PyTorch/torchvision implementation and ImageNet-trained weights of ResNet18 used in all experiments.","marker":"[18]"},{"why":"Provides the activation-extraction toolbox used to obtain the In, Pre, and Post activations.","marker":"[12]"}],"fun_headline_variants":["Residual stream sums scales to grant invariance","Scale invariance emerges from skip-connection sums","ResNet18's residual sum fuses feature scales","Tentative: residual sum builds scale invariance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument turns on the assumption that a channel passing the two feature-visualization tests really is scale invariant, with the input channel holding the smaller scale and the pre-sum channel the larger scale; if that small-scale/large-scale split does not transfer to natural images, the flagged channels are not doing the claimed work and the ablation effect has another explanation.","fun_headline_variants_meta":{"raw":{"variants":["Residual stream sums scales to grant invariance","Scale invariance emerges from skip-connection sums","ResNet18's residual sum fuses feature scales","Tentative: residual sum builds scale invariance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3073,"prompt_tokens":968,"completion_tokens":2105,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2056}},"tokens_in":584,"tokens_out":2105,"duration_ms":15654,"temperature":1.0,"reasoning_tokens":2056,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:06:10.704854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the post-sum responses of the criteria-passing channels in blocks 2.1 and 3.1 to natural images that contain the same feature at several controlled scales, for instance by rescaling each channel's top-activating ImageNet validation images by factors from 0.5x to 2x and recording the center-neuron activation curve; if the post-sum channel's activation is not roughly flat across that range within the 3/2 band while the input and pre-sum channels respond to the small and large ends respectively, the mechanism fails, and so does the ablation-based link to scale-robust behavior.","supporting_citations":[{"cited_title":"Naturally occur- ring equivariance in neural networks","cited_arxiv_id":null,"evidence_quote":"Prior discovery of scale equivariance in InceptionV1; supplies the equivariance-to-invariance framing that this work transfers to ResNet18's residual stream."},{"cited_title":"Feature visualization","cited_arxiv_id":null,"evidence_quote":"Defines the feature-visualization method that generates the images used by the paper's two scale-invariance criteria."},{"cited_title":"Imagenet: A large-scale hierarchical image database","cited_arxiv_id":null,"evidence_quote":"Provides the ImageNet dataset and validation set used for the natural-image checks and all top-1 accuracy measurements."},{"cited_title":"Pytorch: An imperative style, high-performance deep learning library","cited_arxiv_id":null,"evidence_quote":"Supplies the PyTorch/torchvision implementation and ImageNet-trained weights of ResNet18 used in all experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the activation-extraction toolbox used to obtain the In, Pre, and Post activations."}],"review_version":1}