{"id":"f34ea8ab-bf54-4fc7-a210-04056ba5ef54","arxiv_id":"1908.06416","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new spatial orderness metric is proposed, and experiments show that convolutional layers lose their benefit when spatial order is destroyed at higher scales.","lead":"This paper introduces a statistical measure called spatial orderness that captures how much nearby pixels predict distant pixels, and uses it to study convolutional neural networks. It reports that adding convolutional layers can hurt accuracy when the input's spatial order is disrupted, and that CNN feature maps pass through a characteristic rise and fall in spatial orderness during training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Block-swapping confounds loss of spatial order with increased task difficulty; a label-noise control is needed before the depth-benefit claim can be interpreted causally.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: block-swapping may alter other aspects of the learning problem besides spatial orderness, and the paper lacks a control for increased task difficulty. This is the central threat to the strongest claim because the depth-benefit comparison in Figure 2 is used to attribute the benefit of depth to multi-scale spatial order. If a harder task alone can erase the benefit of depth, then the observed pattern does not demonstrate that convolutional depth specifically exploits spatial order. A label-noise control directly addresses this by increasing task difficulty while leaving input spatial orderness exactly unchanged. I do not see a separate, more fundamental flaw that would change the verdict: the spatial-order metric itself is well-defined, the block-swapping manipulation plausibly reduces orderness at higher scales, and the feature-map and kernel analyses are secondary to the main claim. The theory in Section 5 is weak and not load-bearing for the headline result, since the depth experiment stands independently. The existing conditional verdict already requires additional controls and repeated-run statistics, so my read does not move the verdict.","tokens_in":11543,"tokens_out":8280,"duration_ms":96147,"concrete_test":"For each dataset, build a label-noise control: randomly reassign a fraction p of training labels (p chosen so that a 1-conv-layer network achieves approximately the same test error as the Ns=30 block-swapped condition). Then train the same depth sweep as in Section 4.1.2 (1, 2, 3, and 4 conv layers) on both the control and the block-swapped data, repeating each condition with at least six random seeds and reporting mean ± std. If the depth-benefit curves on the label-noise control also flatten or go negative, the block-swap result is confounded by task difficulty; if depth continues to help under label noise at matched accuracy, the spatial-order interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.2's central conclusion—that extra convolutional depth stops paying off when data lacks high-scale spatial order—rests on the assumption that random block-swapping changes only the multi-scale spatial orderness of the data. That assumption is not controlled for. Swapping 6×6 or 8×8 blocks also rearranges class-discriminative parts (e.g., digit strokes) into atypical positions, changes the distribution of pixel pairs across block boundaries, and makes the classification task harder. The depth-benefit curves in Figure 2 therefore conflate 'less spatial order' with 'a harder task with a different image-to-label mapping.' A deeper network has more parameters; on a harder or noisier task, added capacity can overfit and produce the same diminishing or negative returns without any spatial-order mechanism being tested. The supplementary Table 1 repeats runs only for MNIST and still provides no control that raises task difficulty while holding input spatial orderness fixed. Without such a control, the paper's headline claim is not established; the observation is consistent with a pure task-difficulty/capacity story.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a multi-scale statistical metric called spatial orderness, which measures how well pixel values follow the underlying spatial grid, and uses it to investigate the behavior of CNNs. The authors block-swap images from MNIST, Fashion-MNIST, and CIFAR-10 to disrupt spatial orderness, and report that the performance improvement from adding convolutional layers shrinks and sometimes becomes negative as the amount of block-swapping increases. They also observe that feature-map spatial orderness rises then falls during training, with validation accuracy improving only after the falling phase. Finally, they present theorems (Appendix A) intended to show that kernel spatial orderness is related to input spatial orderness and that smaller kernels produce more spatially ordered kernels, with supporting experiments on MNIST-1000.","tokens_in":11819,"tokens_out":5068,"duration_ms":50364,"significance":"If the central finding were established, the paper would provide a useful data-centric criterion for when convolutional depth helps: the benefit of adding conv layers would depend on the presence of multi-scale spatial order in the input. The spatial orderness metric is simple and easy to compute, and the feature-map dynamics are an interesting descriptive observation that could motivate further work. However, the main empirical claim is confounded by task-difficulty changes induced by block-swapping, and the theoretical results in Appendix A are vacuous because the free constant alpha can absorb all scales and the second inequality is essentially the definition of spatial orderness. As a result, the paper's headline claims are not currently supported.","major_comments":[{"comment":"The block-swapping procedure changes not only the spatial orderness of the inputs but also the difficulty of the classification task, because it moves class-discriminative image parts into atypical positions and introduces boundary discontinuities. The paper reports no control condition that increases task difficulty while leaving spatial orderness intact (e.g., label noise or a permuted-label baseline), so the diminishing and negative depth gains in Figure 2 are also consistent with a pure task-difficulty or capacity story. This confound directly affects the abstract's headline claim that convolutional depth is counterproductive for data bereft of spatial order.","section":"Section 4.1.2, Figure 2"},{"comment":"The constant alpha = sum_n C_n^2 N_total is a free parameter that is not bounded below, and the proof only shows that the kernel difference is at most alpha times the input difference for some alpha. Since alpha can be chosen arbitrarily large after the fact, the inequality is always satisfiable and carries no falsifiable content. As a result, the stated conclusion that kernel spatial orderness is 'likely to be higher' for higher input orderness is not entailed by the theorem.","section":"Appendix A, Theorem 1, Eq. (10)"},{"comment":"The second inequality of Theorem 1 is circular: it substitutes the definition of spatial orderness, so(X)_1 = E|X(a)-X(c)|^2 / E|X(a)-X(b)|^2 - 1, into the upper bound, so the claimed relationship between input orderness and kernel orderness is an identity rather than a derivation. No independent bound on the kernel's spatial orderness is obtained, and the 'prediction' is therefore not testable in the form presented.","section":"Appendix A, Theorem 1, Eq. (11)"},{"comment":"The empirical claim that larger kernels yield lower kernel spatial orderness is not a valid test of the theory, because Corollary 1.1 introduces alpha(K) as an unspecified function of kernel size. Without a characterization of alpha(K), any observed trend can be absorbed into that constant, so the experiment in Figure 4(b) does not confirm the theoretical prediction. Additionally, the experiments use only 1000 MNIST examples and six random splits with no error bars shown, limiting confidence in the trend.","section":"Section 5.2, Corollary 1.1, Figure 4(b)"}],"minor_comments":[{"comment":"The spatial orderness metric is not restricted to [0,1]; the paper should state its range and discuss cases where the ratio E[(p-r)^2]/E[(p-q)^2] is less than 1, which would give negative values.","section":"Section 3, Eq. (6)"},{"comment":"The 'synchronized peaking' of feature-map spatial orderness is reported for a single training run per dataset; adding repeated trials with error bars would strengthen this descriptive claim.","section":"Section 4.2.1"},{"comment":"The symbol sigma is used both for the standard deviation in the diffusion model (Eqs. 1-3) and for the noise parameters in Corollary 1.1 (Eq. 13); this overloading is confusing and should be disambiguated.","section":"Section 2 and Appendix A"},{"comment":"The semilog plots make error-rate changes difficult to compare across datasets and conditions; consider plotting on a linear scale or reporting the actual error values with error bars.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has an interesting idea and the empirical setup is easy to follow, but the central depth-benefit claim needs a proper control for task difficulty, and the theoretical appendix is not mathematically substantive as written. I would encourage the editor to solicit a revision only if the authors can add the missing controls and either repair or substantially weaken the theoretical claims; otherwise the paper does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The spatial orderness metric is a simple ratio of squared intensity differences at one-hop versus two-hop lags, which is standard spatial statistics, but pointing it at CNN feature maps and kernels is new, and the block-swap experiments are a neat way to probe when depth stops paying. The three-dataset demonstration gives the empirical story some breadth.\n\nThe main observation—that adding conv layers yields diminishing or negative returns as block-swapping increases—is plausible, but the stress-test is right: block-swapping does not isolate spatial order. It also relocates class-discriminative structure, changes the boundary statistics, and makes the task objectively harder. A deeper network on a harder task can overfit and show the same curve. There is no control that raises task difficulty while preserving spatial orderness, so the headline claim is not established. The supplementary correlation in Table 1 inherits the same confound.\n\nThe feature-map dynamics (synchronized peaking then decline) are interesting descriptively, but they are not backed by error bars or repeated runs in the main text. The theory in Appendix A is the weakest part. Theorem 1's alpha is defined after the fact and can absorb any scale; the second inequality just substitutes the definition of spatial orderness, making it an identity, not a derivation. Corollaries 1.1 and 1.2 inherit the vacuity. This section should be removed or substantially rewritten, because as written it is not a theorem about CNNs—it is an algebraic restatement with free parameters.\n\nCredit where it is due: the metric is trivial to compute and the experiments are reproducible in principle. Section 5.2 does repeat six random splits for MNIST-1000, which is more than many papers at this level, but Figure 2 lacks such statistics.\n\nWho gets value: someone looking for a cheap diagnostic for when depth helps, or a discussion of the pitfalls of corruption-based ablation studies. It is not a paper that resolves the question, but it asks the right question.\n\nFor peer review: send it out. A serious referee should ask for the task-difficulty control and a rewritten or deleted theory section. It is a conditional accept at best, but the empirical kernel is worth discussing.","headline":"A plausible empirical diagnostic wrapped in a vacuous theory; the block-swap confound means the headline depth claim is not yet established.","tokens_in":12234,"tokens_out":1817,"would_cite":false,"duration_ms":22786,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding convolutional layers pays off only when the input has multi-scale spatial order; block-swapping that order shrinks and can reverse the benefit of depth on MNIST, Fashion-MNIST, and CIFAR-10.","keywords":["spatial orderness","convolutional neural networks","inductive bias","compositional structure","block-swapping","feature maps","kernel size","deep learning"],"falsifier":"Train the same CNN at depths 1, 2, and 3 on a block-swapped dataset and on a control that matches the swapped dataset's accuracy while preserving spatial orderness, for instance by injecting label noise or by translating whole digits inside a larger canvas. If the depth benefit, meaning the test-error drop from layer 2 to layer 3, stays large in the control but vanishes on swapped data, the spatial-orderness mechanism is confirmed; if the benefit shrinks in both, the observed effect is task difficulty, not lost spatial order.","tokens_in":11371,"feed_emoji":"🧩","tokens_out":9845,"duration_ms":85993,"temperature":0.7,"pith_summary":"The paper introduces a statistical metric, spatial orderness, that quantifies how much nearby pixels predict distant pixels' values at multiple scales. Using a block-swapping procedure to destroy higher-scale order while leaving low-scale order intact, the authors find that adding convolutional layers to a CNN becomes less useful, and can even raise test error, on MNIST, Fashion-MNIST, and CIFAR-10 once higher-scale order is gone. They also report that feature-map spatial orderness rises synchronously across layers at the start of training, with validation accuracy improving only after orderness begins to fall, and that trained kernels show greater orderness when kernel sizes are small. The paper uses these findings to argue that the inductive bias of convolution is not depth itself but the presence of spatially ordered, compositional structure in the data.","feed_headline":"More CNN layers can backfire when images lose spatial order","feed_subtitle":"New metric measures when image order makes extra CNN layers pay off; block-swapped data shows the cutoff.","key_machinery":"The central object is the multi-scale spatial orderness metric, so(I)_a. For a chosen scale a, it samples triples of locations p (a pixel), q (a one-hop neighbor), and r (a two-hop neighbor), and computes 1 subtracted from the ratio of the mean squared value difference between p and r to that between p and q. Under an ideal spatial-diffusion process the ratio is 2, giving so(I)_a = 1; under random permutation of pixel locations the two differences are equal, giving 0. Different scales are obtained by mean-downsampling the images by a x a blocks. The metric is what lets the paper separate scales: block-swapping with block size k reduces orderness above scale k while leaving lower scales nearly untouched, so depth gains and feature-map/kernel orderness can be tied to specific scales of spatial structure.","core_discovery":"The paper's central claim is that convolution layers exploit a measurable property of the input called spatial orderness, defined through a spatial-diffusion relation among pixels at two-hop distances: for ordered data, the expected squared difference between values two hops apart is twice that of values one hop apart, so the ratio-based metric equals 1, while randomly permuted data gives 0. On block-swapped versions of MNIST, Fashion-MNIST, and CIFAR-10, the test-error reduction from adding a convolution layer shrinks as the number of swaps grows, and at high swap levels an extra layer can increase error; the first layers still help because low-scale order survives. Feature maps mirror this: the spatial orderness of the layer whose output feeds a new convolution layer correlates with the error improvement that layer brings, with reported correlations above 0.98 on MNIST trials. Trained kernels also inherit input orderness, and larger kernels yield lower kernel orderness, with the kernel size equal to the input size behaving like a fully connected layer. Taken together, the paper claims that CNN depth is beneficial only over the range of scales at which the data remains spatially ordered.","pith_inferences":["A direct testable extension would apply the metric to non-image data by replacing pixel neighborhoods with adjacency, for example spectrogram time-frequency bins or graph edges; the prediction is that locality-preserving architectures gain depth only when that adjacency carries order.","The early-training rise in feature-map orderness, if it is robust across architectures, could serve as a training diagnostic: validation gains should not be expected until layer-wise orderness begins to decline, and early stopping before that phase may waste the useful part of training.","The kernel-size result suggests that layer-wise kernel orderness could act as a post-hoc measure of how convolutional a layer really is, bridging the gap between architectural labels and learned behavior.","If the mechanism is right, architecture selection should become data-dependent in a quantitative way: measure the data's spatial-order profile first, then choose how many convolutional stages to stack rather than defaulting to maximum depth."],"forward_implications":["On all three datasets, as block swapping increases, the test-error reduction from adding a convolutional layer becomes smaller and eventually negative.","Since low-scale spatial order survives block-swapping, the first convolutional layers remain useful even on heavily swapped data, isolating the depth effect to scales above the swap block.","During training, feature-map spatial orderness rises across layers in a synchronized early phase, and validation accuracy only begins to improve after the deepest feature maps' orderness starts falling.","Trained kernel orderness is controlled by the input's orderness and by kernel size: smaller kernels produce more spatially ordered filters, and very large kernels behave like fully connected layers.","The correlation between the final feature map's orderness and the improvement from adding the next convolution layer implies that each convolution layer's usefulness depends on the spatial order in the feature map it receives."],"supporting_citations":[{"why":"Provides the theoretical foundation that stacked convolutional layers are optimally biased toward compositional functions, which the paper uses to explain why destroying high-scale order removes the benefit of depth.","marker":"[10]"},{"why":"Supports the bag-of-words view of high-scale image organization, which the paper draws on to interpret the low spatial orderness it measures at higher scales.","marker":"[3]"},{"why":"Supplies the CIFAR-10 dataset used in the block-swapping depth experiments.","marker":"[7]"},{"why":"Supplies the MNIST dataset used in the depth, feature-map, and kernel experiments and in the six-split kernel trials.","marker":"[8]"},{"why":"Supplies the Fashion-MNIST dataset used in the block-swapping depth experiments.","marker":"[14]"},{"why":"Provides the comparison showing CNNs can outperform fully connected nets even when global translation symmetry is absent, motivating the question of what data property makes convolution useful.","marker":"[13]"},{"why":"Represents the standard empirical baseline that deeper CNNs improve accuracy, against which the paper's block-swap reversal is measured.","marker":"[4]"},{"why":"Shows CNNs fail to generalize under certain frequency-domain masks; the paper uses kernel spatial orderness to explain why low-pass filtered inputs behave better.","marker":"[5]"}],"fun_headline_variants":["Spatial order metric predicts when extra CNN layers hurt","CNN depth: only pays off when data stays spatially ordered","Why more convolution layers can backfire on shuffled images","New metric: spatial orderness flags when CNNs need fewer layers","CNN layers hurt if high-scale order is missing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, introduced in Section 4.1, is that block-swapping reduces spatial orderness at higher scales while leaving every other aspect of the classification problem unchanged; if the corruption also makes the task harder or moves class-relevant content out of its usual positions, the loss of depth benefit could be caused by task difficulty rather than by lost spatial order.","fun_headline_variants_meta":{"raw":{"variants":["Spatial order metric predicts when extra CNN layers hurt","CNN depth: only pays off when data stays spatially ordered","Why more convolution layers can backfire on shuffled images","New metric: spatial orderness flags when CNNs need fewer layers","CNN layers hurt if high-scale order is missing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001472,"raw_usage":{"total_tokens":5956,"prompt_tokens":1019,"completion_tokens":4937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":4857}},"tokens_in":635,"tokens_out":4937,"duration_ms":28885,"temperature":1.0,"reasoning_tokens":4857,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:46:07.101028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same CNN at depths 1, 2, and 3 on a block-swapped dataset and on a control that matches the swapped dataset's accuracy while preserving spatial orderness, for instance by injecting label noise or by translating whole digits inside a larger canvas. If the depth benefit, meaning the test-error drop from layer 2 to layer 3, stays large in the control but vanishes on swapped data, the spatial-orderness mechanism is confirmed; if the benefit shrinks in both, the observed effect is task difficulty, not lost spatial order.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the theoretical foundation that stacked convolutional layers are optimally biased toward compositional functions, which the paper uses to explain why destroying high-scale order removes the benefit of depth."},{"cited_title":"Approximating CNNs with bag-of-local- features models works surprisingly well on ImageNet","cited_arxiv_id":null,"evidence_quote":"Supports the bag-of-words view of high-scale image organization, which the paper draws on to interpret the low spatial orderness it measures at higher scales."},{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"Supplies the CIFAR-10 dataset used in the block-swapping depth experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MNIST dataset used in the depth, feature-map, and kernel experiments and in the six-split kernel trials."},{"cited_title":"Regularization of neural networks using dropconnect","cited_arxiv_id":null,"evidence_quote":"Provides the comparison showing CNNs can outperform fully connected nets even when global translation symmetry is absent, motivating the question of what data property makes convolution useful."}],"review_version":1}