{"id":"af7b2a91-e717-4138-9429-64c5ea6c1b9e","arxiv_id":"2507.17785","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A new regularizer that constrains a distance-based self-similarity score of hidden features improves accuracy on MLP and transformer models by up to 6 points, but the metric itself is not validated.","lead":"This paper builds a graph from hidden-layer activations of a neural network, measures its 'self-similarity' with a new score, and uses that score as a training penalty. On MLP and attention models, the penalty improves classification accuracy by up to about 6 percentage points, while convolutional models show little change.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SS-rate metric rests on an unvalidated proxy (Eq. 9) that ignores feature arrangement, so the paper's self-similarity claims are unsupported; a direct box-covering comparison would settle it.","rationale":"The paper's central claim is that a differentiable self-similarity metric, SS rate, can act as a regularizer and improve performance in intrinsically self-similar architectures. The entire chain of evidence—Algorithm 2, Eq. 6, and Table 2—depends on Eq. 9's Nθ being a faithful proxy for the box-covering count that defines self-similarity. Inspection shows Eq. 9 depends only on the scalar p(C≤θ) at each threshold, i.e., the cumulative histogram of pairwise distances, and discards all arrangement information that box-covering is designed to capture. No validation of this approximation against a real box-covering algorithm, or against any external notion of self-similarity, is provided. The regularizer could therefore be penalizing an arbitrary smooth function of pairwise distances, and the attribution of accuracy gains to 'self-similarity' is unsupported. The reported gains are real but are not compared with standard regularizers, so they cannot carry the claim by themselves. This is the same load-bearing weakness identified by the reader's weakest_assumption. A direct comparison of Eq. 9 with a greedy box-covering count on real feature matrices would settle the matter; until that is done, the rejection verdict is appropriate.","tokens_in":11702,"tokens_out":7798,"duration_ms":84929,"concrete_test":"Take the actual feature matrices used for Table 2 (e.g., ViT-Base on CIFAR-100) and compute, over a grid of θ values, the proxy Nθ from Eq. 9 and a faithful box-covering count Nθ_true using a standard greedy box-covering algorithm on the complete weighted graph with edge weights C_ij (the method described in Appendix A). If the Spearman rank correlation between Nθ and Nθ_true is below 0.8 or the mean relative error exceeds 20%, Eq. 9 is not a valid stand-in for box-covering and the self-similarity interpretation collapses. Report the same comparison for every model in Table 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation 9 defines Nθ = 1 + (D−1) log_D(D + (1−D)p(C≤θ)), where p(C≤θ) is the empirical CDF of all pairwise feature distances. For each θ, Nθ is a fixed monotone function of the single scalar p, so it contains no information about which nodes are close to which. Two feature configurations with identical pairwise-distance histograms yield identical Nθ(θ) curves and identical SS rates, even when their true box-covering counts (minimum number of diameter-θ clusters, the quantity Algorithm 2 is meant to approximate) differ. The paper calls Eq. 9 an 'approximate simulation' of the box-cover method (Appendix A), but no comparison to any real box-covering algorithm is reported. The SS rate is therefore an unvalidated functional of the distance distribution, not a demonstrated measure of feature-network self-similarity. Because the same quantity is used both as the training penalty in Eq. 6 and as the evidence of self-similarity in Table 2, the central narrative—that constraining self-similarity improves MLP and attention models—is unsupported unless Eq. 9 is shown to track true box-covering. The Table 3 accuracy gains could come from any smooth distance-distribution regularizer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a complex-network representation GM built from hidden-layer activations, defines a self-similarity rate (SS rate) through a differentiable proxy for box-covering, and adds an MSE regularizer that penalizes the squared difference between the SS rate and a target value gamma. Experiments on CIFAR-10, CIFAR-100, and Imagenette across MLP, CNN, and attention architectures report accuracy gains for ResMLP, MLP-Mixer, and ViT models, and the authors argue that models with intrinsic self-similarity benefit most from the constraint.","tokens_in":11964,"tokens_out":5876,"duration_ms":56423,"significance":"If the proposed metric were validated, the paper would offer a useful, architecture-agnostic tool for monitoring and steering the geometry of hidden representations during training. The experimental protocol is commendable in reporting means and standard deviations over ten runs and in providing public code, and the reported gains (e.g., ViT-Base CIFAR-100 from 61.40 to 67.15) are potentially interesting. However, the significance currently hinges on an unvalidated proxy for box-covering, and the claimed mechanism connecting self-similarity to accuracy is not established by the evidence presented.","major_comments":[{"comment":"The proposed proxy Nθ = 1 + (D−1) log_D(D + (1−D)p(C≤θ)) is a fixed monotone function of the single scalar p(C≤θ), so it contains no information about which nodes are close to which; two feature configurations with identical pairwise-distance histograms yield identical SS rates even when their true box-covering counts differ. The paper calls this an approximate simulation of the box-cover method but reports no comparison to any actual box-covering algorithm (such as the burning or greedy algorithms described in Appendix A). Because the same quantity is used both as the training penalty in Eq. (6) and as the evidence of self-similarity in Table 2, the central narrative that constraining self-similarity improves MLP and attention models is unsupported unless Eq. (9) is shown to track true box-covering. A direct comparison on small feature networks, or a counterexample with matched distance histograms, would settle this.","section":"Appendix A, Eq. (9); Algorithm 2"},{"comment":"The loss defines gamma as the inherent self-similarity level of feature networks after standard training without regularization constraints. If the regularizer works, the S2-Con SS rates should move toward the baseline values; instead, Table 3 reports moves away from baseline in many cases (e.g., ResMLP CIFAR-10 0.29 to 0.36, MLP-Mixer CIFAR-10 0.15 to 0.25, ViT-small CIFAR-10 0.49 to 0.51). The paper never reports the gamma values used, nor does it verify that the constraint actually reduces |SS_rate − gamma|. The accuracy gains in Table 3 are therefore not shown to be attributable to the stated self-similarity constraint; they could arise from any smooth auxiliary penalty on pairwise feature distances.","section":"Eq. (6), Table 3"},{"comment":"The claim that performance improvements occur precisely for models with better statistical and geometric invariance is not supported by the tables as presented. ResMLP has the third-worst statistical scale invariance (0.3420) yet shows consistent gains; PoolFormer has a better geometric-invariance score (0.2735) than ViT (0.2772) but degrades on CIFAR-10 and CIFAR-100; and ResNet34/50 show small gains despite poor invariance rankings. No correlation coefficient, significance test, or per-layer analysis is provided, so the proposed mechanism that intrinsic architectural self-similarity governs constraint efficacy is not established.","section":"Section 'Architectural Self-Similarity: The Decisive Factor', Tables 4–5"},{"comment":"The experimental protocol omits several quantities needed for reproducibility and for interpreting the effect size: the values of gamma per model and dataset, the regularization coefficient alpha (only 'order of magnitude 10−4' is given), the smoothing factor k and gradient scaling factor fac in Algorithm 2, the threshold epsilon in Eq. (1), the number of layers sampled in the random layer selection, and the batch size B used for GM construction. Without these details, the reported improvements cannot be reproduced, and it is unclear whether the results are sensitive to the auxiliary-loss hyperparameters.","section":"Experiments, Eq. (6)"}],"minor_comments":[{"comment":"The affiliation contains a typo: 'Inteligence' should be 'Intelligence'.","section":"Title page"},{"comment":"The phrase 'an self-similarity constraint' should be 'a self-similarity constraint'.","section":"Conclusion"},{"comment":"The text refers to 'the universal SS rate elevation trend observed in Table 1', but the relevant table is Table 2.","section":"After Table 2"},{"comment":"Figure 2 lacks axis labels and a clear legend, which makes the claimed clustering-versus-diffusion differences among architectures difficult to verify.","section":"Figure 2"},{"comment":"The meaning of 'Pre-train' and 'Post-train' in Table 2 should be stated explicitly, since it is unclear whether these denote random-initialization versus standard-trained models or pre-trained versus fine-tuned checkpoints.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a concrete experimental result but the central interpretation is not yet supported. The missing validation of Eq. (9) against true box-covering is an empirical gap that can be filled within the scope of a revision, so I do not recommend outright rejection. If the authors cannot provide such validation, the manuscript's claims would need to be substantially weakened to describe a distance-distribution regularizer rather than a self-similarity regularizer."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper gets some real accuracy gains from a new regularizer, but it rests on a metric that does not actually measure what it claims. The stress-test note is correct. Equation 9 defines Nθ as a function of p(C≤θ), the fraction of pairwise distances below θ. For each θ, that is a single scalar from the distance histogram, so the SS rate cannot distinguish between two feature configurations with the same pairwise distance distribution but different network geometry. Real box-covering counts depend on which nodes are close to which. The authors call this an 'approximate simulation' (Appendix A) but never compare it to any actual box-covering algorithm. Without that, the self-similarity story is unsupported.\n\nWhat is genuinely new here: the GM construction, the differentiable SS rate, and its use as a regularizer. The authors are also honest about their experimental setup—10 runs, mean/std, public code, multiple architectures and datasets. The main empirical observation, Table 3, shows consistent gains on ResMLP, MLP-Mixer, and ViT, and no gains on CNNs. Those numbers are worth checking.\n\nThe soft spots are load-bearing. The same metric is used as the training objective (Eq. 6) and as the evidence of self-similarity change (Table 2), so the claim 'constraining self-similarity improves performance' is partly circular. What is not circular is the accuracy improvement, but it is confounded by lack of comparison to simpler regularizers (e.g., penalizing distance histogram skewness, feature decorrelation). The PoolFormer results contradict the narrative—CIFAR-10/100 drop under the constraint—and the paper's explanation (local vs global) is plausible but post-hoc. The auxiliary invariance metrics in Tables 4 and 5 (power-law exponent std, correlation dimension fluctuation) are interesting but are different from SS rate and are also not validated.\n\nIf the authors add a direct box-covering comparison on small graphs and show Nθ tracks the true box count, the paper would be substantially stronger. If they also compare against a generic distance-distribution regularizer, they could separate the self-similarity effect from a generic smoothing effect. As is, I would not accept the central claim, but the paper deserves a serious referee because the empirical result is concrete and fixable.\n\nFor you: cite it only if you need a reference for 'differentiable self-similarity regularizer' with all caveats. I'd send it to review if I were an editor, but with a request for major revision and the box-covering validation.","headline":"Real accuracy gains from a self-similarity regularizer, but the metric is an unvalidated histogram statistic, so the paper's central claim is unsupported.","tokens_in":12520,"tokens_out":3474,"would_cite":false,"duration_ms":34917,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that treating hidden-layer features as a complex network and regularizing their self-similarity improves classification accuracy by up to 6 percentage points, but only in MLP and attention architectures.","keywords":["self-similarity","complex networks","feature networks","hidden-layer features","differentiable regularization","vision transformers","MLP-Mixer","SS rate"],"falsifier":"Take a small trained MLP or ViT on CIFAR-10 and compute the SS rate both with the paper's proxy and with an exact minimum-box-count algorithm, such as burning or cluster-growing, on the same thresholded feature graph. If the proxy disagrees with the exact box count in the ordering of layers or models, then the reported accuracy gains cannot be attributed to self-similarity.","tokens_in":11456,"feed_emoji":"📈","tokens_out":8850,"duration_ms":83632,"temperature":0.7,"pith_summary":"This paper argues that the internal feature geometry of a deep network can be viewed as a complex network, and that its degree of self-similarity is a trainable quantity. The authors build a graph from hidden-layer activation vectors, define a differentiable self-similarity rate (SS rate), and use it as a regularizer during training. They report that this constraint improves accuracy on CIFAR-10, CIFAR-100, and Imagenette for MLP and attention architectures, by up to 6 percentage points, while leaving convolutional networks essentially unchanged. The reason, they argue, is that networks with global information aggregation genuinely possess self-similar feature structure, so enforcing that structure helps rather than hurts.","feed_headline":"Self-similarity loss adds up to 6 points for ViT and MLP","feed_subtitle":"A differentiable self-similarity metric as a regularizer helps only networks that already show self-similar feature geometry.","key_machinery":"The central object is $G_M$, a graph whose $D$ nodes are hidden-layer feature maps after dimension reduction, with edges defined by Euclidean distance below a threshold. Self-similarity is quantified by the SS rate, which measures deviation of the log of a box-count proxy from the linear law expected for a fractal network, with values near 0 indicating strong self-similarity and values near 1 indicating weak self-similarity. The key enabling device is the differentiable proxy $N_\\theta = 1 + (D-1)\\log_D(D + (1-D)p(C \\le \\theta))$, where $p(C \\le \\theta)$ is the fraction of normalized feature distances below threshold $\\theta$; because it is differentiable through a sigmoid smoothing of the step count, it can enter the loss function directly and be optimized by backpropagation.","core_discovery":"During training, the feature networks $G_M$ of most architectures move from high self-similarity to lower self-similarity. The paper's method penalizes deviation of the SS rate from a target value, namely the post-training SS rate of an unregularized model, through the loss $L_{\\text{total}} = L_{\\text{task}} + \\alpha(SS_{\\text{rate}}(G_M^{(k)}) - \\gamma)^2$. This constraint consistently improves ResMLP, MLP-Mixer, and ViT small/base, with the largest gain being ViT-Base moving from 61.40 to 67.15 on CIFAR-100; it does not help convolutional networks or PoolFormer, and slightly hurts Vgg. The architecture-dependent effect is corroborated by measurements of statistical scale invariance and geometric invariance, which the paper presents as indicators of intrinsic self-similarity.","pith_inferences":["As an editorial extension, the fixed target $\\gamma$ is read from the very unregularized training run the method is meant to improve, so the paper's own caveat about fixed targets points toward a curriculum or self-tuning version that has not been tested.","As an editorial extension, the paper's global-aggregation explanation yields a testable prediction the authors did not run: a convolutional network with dilated global receptive fields should start showing S2-Con gains.","As an editorial extension, one could replace the SS-rate regularizer with another differentiable function of the same pairwise-distance matrix; if gains persist, the effect is a generic feature-distance regularizer rather than self-similarity per se."],"forward_implications":["The S2-Con regularizer can yield accuracy gains near the 6-point ceiling, for instance ViT-Base on CIFAR-100 from 61.40 to 67.15 and MLP-Mixer on CIFAR-10 from 86.81 to 88.27.","The same constraint leaves Vgg, ResNet, and PvT essentially flat or slightly lower, so the benefit is not generic regularization but depends on the architecture's intrinsic self-similarity.","Self-similarity of feature networks is not static: it degrades during training, which gives a new training-dynamics observable for how hidden representations specialize.","Sampling one hidden layer's $G_M$ per iteration keeps the SS-rate regularization computationally tractable for deep models.","The paper notes that the fixed target $\\gamma$ may constrain a model's autonomous optimization pathway, pointing toward time-variant self-similarity targets as future work."],"supporting_citations":[{"why":"Grounds the claim that complex networks can be self-similar and supplies the box-covering scaling law that the SS rate measures deviation from.","marker":"Song, Havlin, and Makse 2005"},{"why":"Supplies the box-covering algorithms the paper adapts; the need to make $N_\\theta$ differentiable motivates the proxy in Algorithm 2.","marker":"Kov́acs, Nagy, and Molontay 2021"},{"why":"Provides the notion of statistical scale invariance used to identify which architectures intrinsically possess self-similarity.","marker":"Mahoney and Martin 2019"},{"why":"Supplies the correlation-dimension method used to measure geometric invariance across feature layers.","marker":"Grassberger and Procaccia 1983"},{"why":"Defines the ViT architectures whose accuracy gains under S2-Con carry the main performance claim.","marker":"Dosovitskiy et al. 2020"},{"why":"Defines the MLP-Mixer architecture, another beneficiary of the regularizer that supports the architecture-dependent conclusion.","marker":"Tolstikhin et al. 2021"},{"why":"Provides the ResNet baselines that serve as the control case where S2-Con produces little change.","marker":"He et al. 2016"},{"why":"Supplies the CIFAR-10 and CIFAR-100 benchmarks where the reported accuracy gains are measured.","marker":"Krizhevsky, Hinton et al. 2009"}],"fun_headline_variants":["Self-similarity penalty lifts MLP and ViT up to 6 points","Regularize self-similarity to win +6 on MLP and ViT","Self-similarity loss gives 6 points only to MLP and ViT","Self-similarity penalty improves MLP and ViT, not CNNs","Why self-similarity loss works for MLP and ViT but not CNN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the paper's closed-form formula for the box count is a faithful stand-in for the true box-covering count of the feature graph. If that formula is wrong, the regularizer is just penalizing a clever nonlinear function of pairwise feature distances, and the whole self-similarity story does not carry.","fun_headline_variants_meta":{"raw":{"variants":["Self-similarity penalty lifts MLP and ViT up to 6 points","Regularize self-similarity to win +6 on MLP and ViT","Self-similarity loss gives 6 points only to MLP and ViT","Self-similarity penalty improves MLP and ViT, not CNNs","Why self-similarity loss works for MLP and ViT but not CNN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001297,"raw_usage":{"total_tokens":5281,"prompt_tokens":922,"completion_tokens":4359,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":4254}},"tokens_in":538,"tokens_out":4359,"duration_ms":28012,"temperature":1.0,"reasoning_tokens":4254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:50:05.152123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small trained MLP or ViT on CIFAR-10 and compute the SS rate both with the paper's proxy and with an exact minimum-box-count algorithm, such as burning or cluster-growing, on the same thresholded feature graph. If the proxy disagrees with the exact box count in the ordering of layers or models, then the reported accuracy gains cannot be attributed to self-similarity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounds the claim that complex networks can be self-similar and supplies the box-covering scaling law that the SS rate measures deviation from."},{"cited_title":"T.; Nagy, M.; and Molontay, R","cited_arxiv_id":null,"evidence_quote":"Supplies the box-covering algorithms the paper adapts; the need to make $N_\\theta$ differentiable motivates the proxy in Algorithm 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the notion of statistical scale invariance used to identify which architectures intrinsically possess self-similarity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the correlation-dimension method used to measure geometric invariance across feature layers."},{"cited_title":"O.; Houlsby, N.; Kolesnikov, A.; Beyer, L.; Zhai, X.; Unterthiner, T.; Yung, J.; Steiner, A.; Keysers, D.; Uszkoreit, J.; et al","cited_arxiv_id":null,"evidence_quote":"Defines the MLP-Mixer architecture, another beneficiary of the regularizer that supports the architecture-dependent conclusion."}],"review_version":1}