{"id":"28095415-23f7-4b7f-8c73-345c6cadcb73","arxiv_id":"2411.17182","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In CRATE-family transformers, the attention-like MSSA update with skip connection raises the coding rate it was designed to compress, yet layer-averaged SRR still correlates positively with the generalization gap (tau = 0.445) and slightly improves CIFAR-10/100 accuracy when used as a regularizer.","lead":"This paper investigates whether Sparse Rate Reduction (SRR), the information-theoretic objective behind the white-box CRATE transformer, is truly optimized in the forward pass and whether it predicts or improves generalization. It uncovers a derivation artifact in the attention-like compression step, reports that SRR correlates with the generalization gap better than 21 baseline complexity measures, and uses SRR as a regularizer with small accuracy gains on CIFAR-10/100.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline tau = 0.445 is computed on a forward pass with LayerNorm removed from models trained with LayerNorm; no evidence exists that this preserves the ranking of SRR.","rationale":"The paper's central empirical claim is the positive correlation of SRR with generalization and its superiority over 21 baselines. I read the derivation in Section 4.1 as sound: eqs. (5)-(10) are internally consistent, and the toy experiment cleanly shows the sign flip. The regularization results in Table 2 are also plausible, though single-seed. The most load-bearing vulnerability is not the math but the measurement protocol: the evaluation removes LayerNorm after training with it. Since the measure is data-dependent and scale-sensitive, and the baselines are computed under the same modified forward pass, the entire comparison could be an artifact. This is a correctable empirical check, not a mathematical refutation. The reader flagged the same issue; I see no additional concern that is more central. The secondary in-sample and width-selection issues are real but less decisive, because the width-768 table independently gives SRR the highest overall tau. Therefore the verdict stays CONDITIONAL pending the LayerNorm sensitivity analysis and ideally held-out hyperparameter validation.","tokens_in":15838,"tokens_out":8777,"duration_ms":84434,"concrete_test":"Recompute Table 1 at d = 384 with LayerNorm kept in the forward pass for all activation-dependent measures (SRR, 1/margin, PAC-Bayes flatness, and any other measure that uses activations), holding everything else fixed. Report Kendall tau overall and per-axis for SRR and the leading baselines. In the same 32 models, compute the Spearman rank correlation between SRR values with and without LayerNorm; if tau no longer ranks SRR first, or if the with/without SRR rank correlation is weak (e.g., rho < 0.8), the evaluation protocol is the source of the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2's central correlation result rests on an unvalidated evaluation-protocol choice. Appendix C states: 'We include Layer Normalization before each operator during training, but also remove it when evaluating the complexity measures.' All 64 models were trained with LayerNorm, yet the SRR measure (eq. 11) and every activation-dependent baseline are evaluated on a forward pass in which LayerNorm is absent. The SRR measure contains the log-determinant coding-rate terms R and Rc and the l0 count; these are highly sensitive to the scale and location of the activations, which LayerNorm directly controls. Removing LayerNorm changes the function computed by each trained model, so the numbers fed into Kendall's tau do not describe the models that were trained. The paper gives no sensitivity analysis or rank-preservation check, and the choice of width 384 is made after observing that correlations are 'more prominent' there. If the rank ordering of the 32 models by SRR changes when LayerNorm is included, the headline tau = 0.445 and the claimed superiority over path-norm and sharpness-based baselines are artifacts of the evaluation protocol rather than a property of the learned representations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates the Sparse Rate Reduction (SRR) objective in the white-box Transformer-like CRATE architecture. The authors first analyze the MSSA update with a skip connection and argue, via a Taylor-expansion argument and a toy experiment, that the standard CRATE-C implementation actually increases the coding rate Rc rather than decreasing it. They then introduce two variants, CRATE-N and CRATE-T, and study how SRR evolves across layers and training epochs. In the central empirical study, they train 64 CRATE-family models on CIFAR-10 and report Kendall rank correlations between a layer-averaged SRR complexity measure and the generalization gap, finding a positive overall correlation (tau = 0.445 at width 384 and 0.407 at width 768) that outperforms several baseline complexity measures. Finally, they show that adding an SRR-based regularizer to the cross-entropy loss yields small accuracy improvements on CIFAR-10/100.","tokens_in":15990,"tokens_out":3808,"duration_ms":38582,"significance":"If the correlation result is valid, the paper makes a useful contribution by proposing a representation-based complexity measure for an interpretable Transformer-like architecture and by connecting an information-theoretic training objective to generalization. The paper is honest about the scope of its conclusions: the Limitations section explicitly notes that the findings are restricted to the CRATE family and that the analysis is limited in scale. The derivation of the decompression artifact in Section 4.1 is a concrete and testable observation, and the toy experiment plus the Appendix A ablations give a clear demonstration that the MSSA update with softmax increases Rc. The introduction of CRATE-N and CRATE-T as design variants, and the comparison of their layer-wise SRR behavior, are also useful. The central limitation, however, is that the headline correlation is computed under an evaluation protocol that removes LayerNorm from models trained with LayerNorm, and no evidence is provided that this protocol preserves the ranking of models by SRR.","major_comments":[{"comment":"The correlation study computes all complexity measures on a forward pass with LayerNorm removed, although every model was trained with LayerNorm before each operator. Because the SRR measure in eq. (11) contains log-determinant coding-rate terms and an l0 count, both of which are sensitive to the scale and location of activations, removing LayerNorm changes the function computed by the trained model. The paper provides no sensitivity analysis or rank-preservation check for any of the measures, including SRR. Without evidence that model rankings by SRR are stable under this protocol change, the reported tau = 0.445 (and tau = 0.407 at width 768) could be an artifact of the evaluation protocol rather than a property of the learned representations. Please report correlations with LayerNorm included at evaluation time, or an ablation demonstrating that the rank ordering of the 64 models is preserved when LayerNorm is removed.","section":"Appendix C, Section 5.2"},{"comment":"The claim that SRR has 'predictive power' for generalization is supported only by rank correlations computed on the same 64 in-sample models whose generalization gaps are used for the correlation. No held-out hyperparameter settings are tested, and the width 384 is selected for presentation because the correlations are 'more prominent' there. At width 384, the model-type axis alone gives tau = 0.714 for SRR, which is much larger than the overall tau = 0.445, suggesting that model-type differences dominate the correlation. To support the predictive-power claim, the authors should report an out-of-sample evaluation, such as leave-one-hyperparameter-out analysis, or explicitly condition on the model-type axis and show that SRR remains predictive within each model family.","section":"Section 5.2, Tables 1 and 5"},{"comment":"The regularization experiments are presented as evidence that SRR causally improves generalization, but the reported gains are small (0.03 to 1.13 percentage points), the regularization coefficient eta = 0.001 is selected by a grid search, and no comparison is made against regularizing with other complexity measures (e.g., path-norm, sharpness, or a simple activation-norm penalty). Since the stop-gradient and last-layer-only implementation is a specific design choice, it is unclear whether the improvement comes from the SRR objective itself or from any layer-wise activation regularizer. Please provide ablations that compare SRR regularization with alternative complexity-measure regularizers under the same protocol, and report variance over multiple seeds.","section":"Section 6, Eq. (14)"}],"minor_comments":[{"comment":"The symbol lambda is used both for the sparsity weight inside the SRR measure, as in eq. (11), and for the overall regularization coefficient in eq. (14). These are different quantities and should be denoted by different symbols to avoid ambiguity.","section":"Eq. (14)"},{"comment":"The displayed Kendall tau formula does not account for ties in the complexity measure or the generalization gap. The standard Kendall tau-b correction, or a statement that no ties occur, should be included so the reported coefficients are reproducible.","section":"Eq. (13)"},{"comment":"The figure quality is inconsistent: Figure 1(b) has garbled axis labels, and Figures 2 and 3 contain many overlapping curves that are difficult to read in print. Please reformat these figures and, where possible, add markers or a small multiples layout.","section":"Figures 1(b), 2, and 3"},{"comment":"The sentence 'in the absence of operation (3)' is confusing because operation (3) is in the same layer as the MSSA operator; the intended meaning appears to be 'when the sparsification step is disabled,' which should be stated explicitly.","section":"Section 4.3"},{"comment":"The efficient-implementation comparison reports single accuracies without standard deviations or seed information, making it hard to judge whether differences such as 77.61 vs. 77.75 are meaningful. Please report multiple runs or at least state that results are from a single seed.","section":"Table 6"},{"comment":"The phrase 'correlation with the generation' in the experimental-results paragraph appears to be a typo for 'generalization' and should be corrected.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a machine-learning venue and the authors are transparent about limitations, which is commendable. My main concern is the evaluation-protocol issue with LayerNorm removal, which directly affects the headline correlation and the claimed superiority of SRR over baselines. I do not see this as an unfixable flaw; adding the requested rank-preservation checks and out-of-sample analysis would substantially strengthen the paper. I would not recommend rejection on the current evidence, but the revision needs to address the load-bearing concerns rather than simple presentation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honest paper about CRATE. The best part is Section 4.1: the skip-connected MSSA update derived in Yu et al. actually increases Rc, not decreases it, and the toy experiment plus eigenvalue argument make that case convincingly. The two new variants (CRATE-N and CRATE-T) are sensible, and the layer-wise SRR curves are a useful addition. The generalization-correlation headline is plausible but conditional: the reported tau = 0.445 is an in-sample correlation on 32 models, and the measure is computed on a forward pass with LayerNorm removed even though all models were trained with it.\n\nWhat's new: the decompression artifact, the two variants, the SRR complexity measure with its correlation table, and the SRR regularizer. Equations (5)-(8) check out, and the correlation study follows the standard Jiang et al. protocol, reports per-axis coefficients, and SRR beats 21 baselines at both widths. The regularization results are modest but consistent (0.03 to 1.13 points), and the paper honestly discloses its limitations.\n\nSoft spots, in order: the LayerNorm issue is the biggest. Appendix C says LayerNorm is removed when evaluating every complexity measure. SRR contains log-det terms and an l0 count, so its value depends heavily on activation magnitude; no sensitivity check shows the rank ordering survives with LayerNorm kept. Second, the headline tau is a point estimate on 32 models per width, and the width-384 table was chosen after seeing correlations were 'more prominent' there. Third, the batch-size axis shows essentially no correlation. These don't sink the paper, but they mean 'predictive power' should be framed as 'correlates on this model family, under this evaluation protocol.'\n\nWho it's for: people working on white-box transformers, algorithm unrolling, or empirical generalization measures. The artifact finding alone justifies reading it. The correlation claim needs a sensitivity analysis before being cited as predictive.\n\nRecommendation: send it to peer review. A good referee would ask for confidence intervals, a with-LayerNorm comparison, and ideally a held-out hyperparameter split. If those come back clean, this becomes a useful reference.","headline":"Solid, honest CRATE investigation with a real artifact finding and a plausible but protocol-sensitive generalization correlation that needs a sensitivity check before being called predictive.","tokens_in":16675,"tokens_out":2972,"would_cite":true,"duration_ms":26540,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse Rate Reduction, the objective behind the CRATE Transformer-like model, predicts generalization better than 21 baseline complexity measures and works as a regularizer.","keywords":["sparse rate reduction","CRATE","generalization gap","complexity measure","algorithm unrolling","Kendall rank correlation","coding rate","regularization"],"falsifier":"Compute the SRR measure on the same 64 trained models with the Layer Normalization modules left in the forward pass (e.g., folding their scale and shift parameters into the coding-rate computation), and recompute Kendall's $\\tau$ against the generalization gap; if the correlation falls below the sharpness-based baselines, the reported $\\tau = 0.445$ is an artifact of the LayerNorm-removal protocol. A complementary check is to train additional CRATE variants at hyperparameter settings outside the original 64 combinations (for example, learning rate $5\\times 10^{-5}$ or width 576) and test whether the in-sample correlation rank order persists.","tokens_in":15483,"feed_emoji":"📈","tokens_out":9806,"duration_ms":79761,"temperature":0.7,"pith_summary":"This paper sets out to show that Sparse Rate Reduction (SRR) — the information-theoretic objective that the white-box Transformer-like CRATE architecture is designed to optimize — is more than a design story: it is a usable predictor of generalization and a usable regularizer. On a collection of 64 CRATE variants obtained by varying four hyperparameters and four implementations, the layer-averaged SRR measure ranks models by generalization gap with Kendall's $\\tau = 0.445$ at width 384 (0.407 at width 768), ahead of all 21 norm-, margin-, and sharpness-based baselines tested. Adding a stop-gradient SRR penalty to the cross-entropy loss improves top-1 accuracy on CIFAR-10/100 by small, consistent margins. Along the way the paper identifies a flaw in the original CRATE derivation: the self-attention-style update with a skip connection that was supposed to compress representations actually increases the subspace coding rate, because the second-order Taylor expansion used to derive it drops the first-order term. The reason to care is that if SRR really tracks generalization, an architecturally interpretable objective gains a practical role in model selection and training.","feed_headline":"Sparse Rate Reduction tops 21 complexity measures on generalization","feed_subtitle":"Across 64 trained variants it predicts generalization gaps with τ = 0.445 and lifts CIFAR accuracy.","key_machinery":"The load-bearing object is the SRR complexity measure in equation (11), the layer average of $\\lambda \\|Z^{\\ell}\\|_0 + R_c(Z^{\\ell};U^{\\ell}) - R(Z^{\\ell})$, where $R(Z) = \\frac{1}{2} \\log \\det\\left(I + \\frac{d}{N\\epsilon^2}Z^{\\top}Z\\right)$ is the coding rate of the token matrix and $R_c(Z;U) = \\sum_{k} R(U_k^{\\top}Z)$ is the coding rate of tokens projected onto $K$ incoherent low-dimensional subspaces spanned by $U$. The second key object is the MSSA update with skip connection, equation (8), $Z \\leftarrow Z + \\alpha\\gamma^2 \\sum_k U_k U_k^{\\top}Z\\,\\mathrm{softmax}((U_k^{\\top}Z)^{\\top}(U_k^{\\top}Z))$, which the paper derives from a Taylor expansion of the gradient of $R_c$ and then shows acts as an ascent step on $R_c$. These two objects carry the argument: the paper measures SRR on the forward passes of four CRATE implementations, correlates it with the generalization gap using Kendall's $\\tau$, and uses the same measure, with gradients stopped, as a regularizer in equation (14).","core_discovery":"The paper's central claim is that SRR, evaluated as a layer-averaged complexity measure on the forward pass of CRATE-family models, has genuine predictive power for generalization and can be used to improve training. Concretely, the authors train 64 models spanning CRATE-C, CRATE-N, CRATE-T, and CRATE with varied batch size, learning rate, width, and dropout; they report that the measure $\\mu_{\\mathrm{SRR}}(w;Z) = \\frac{1}{L} \\sum_{\\ell} [\\lambda \\|Z^{\\ell}\\|_0 + R_c(Z^{\\ell};U^{\\ell}) - R(Z^{\\ell})]$ has Kendall's $\\tau = 0.445$ with the generalization gap at width 384 and $\\tau = 0.407$ at width 768, outperforming every baseline, and that its strongest per-axis signal comes from the model-type axis ($\\tau = 0.714$). They also claim that minimizing SRR with a stop-gradient at the last layer improves CIFAR-10/100 accuracy by 0.03 to 1.13 points. The paper's secondary discovery is that the canonical MSSA update with a skip connection, equation (8), secretly maximizes the compression term $R_c$ instead of minimizing it; this is traced to the Taylor approximation in equation (7), whose discarded first-order term is what would make descent actually descend. The authors treat these findings as evidence that SRR is a meaningful principle for designing and diagnosing unrolled Transformer-like models, while explicitly limiting the conclusion to the CRATE family.","pith_inferences":["Because the model-type axis contributes the largest per-axis correlation ($\\tau = 0.714$), the headline $\\tau$ may mostly separate architectural variants rather than predict generalization within a fixed architecture; testing SRR across many hyperparameter settings inside a single variant, including held-out settings, would clarify this.","If the correlation depends on the LayerNorm-removal protocol, then a simpler measure — such as the norm or coding rate of pre-normalization activations — might reproduce the ranking, which would be a cheaper substitute for full SRR; the paper does not test this.","SRR regularization with detached activations is a local, layer-wise objective; extending it to per-layer training or to the Forward-Forward style of credit assignment could make it work for very deep models, but that is an extrapolation beyond the paper's experiments.","The paper's own limitation that SRR is undefined for standard Transformers with untied query, key, and value matrices suggests a concrete next test: define an analogous coding-rate penalty on attention outputs with untied projections and check whether the positive correlation to generalization survives outside the CRATE family."],"forward_implications":["If SRR ranks generalization as reported, then the layer-averaged SRR value of a trained CRATE-family model is a cheap model-selection signal: pick the variant with lower SRR before committing to deployment.","Because adding a stop-gradient last-layer SRR penalty improves CIFAR-10/100 accuracy by 0.03–1.13 points, SRR regularization is a low-cost addition to the cross-entropy objective for unrolled models, requiring only one layer's SRR per step.","The negative correlation of path-norm with generalization implies that path-norm-based regularizers are unlikely to improve these Transformer-like models, contrary to their role in other settings.","The finding that the MSSA update in CRATE-C actually increases $R_c$ implies that the original CRATE's forward pass is not faithfully implementing its stated compress-then-sparsify optimization, and that the sign-flipped CRATE-N is a more faithful instantiation of the compression step."],"supporting_citations":[{"why":"Defines Sparse Rate Reduction and the CRATE architecture; supplies the MSSA and ISTA derivations that the paper inspects and modifies.","marker":"[45]"},{"why":"Defines the coding rate function $R(Z)$ and the maximal coding rate reduction principle that SRR extends.","marker":"[46]"},{"why":"Supplies the methodology for collecting trained models, the stopping criterion, Kendall correlation evaluation, and the baseline complexity measures.","marker":"[18]"},{"why":"Defines Kendall's rank correlation coefficient used for all reported $\\tau$ values.","marker":"[19]"},{"why":"Defines path-norm, the key baseline measure that SRR is claimed to outperform and that the paper finds negatively correlated.","marker":"[31]"},{"why":"Defines sharpness-based complexity measures used as baselines in the correlation comparison.","marker":"[20]"},{"why":"Layer Normalization is used in training but removed during complexity evaluation, a protocol choice that the correlation claim depends on.","marker":"[2]"},{"why":"Provides the algorithm-unrolling template (learned ISTA-like sparse coding layers) that CRATE and the layer-by-layer SRR analysis build on.","marker":"[15]"}],"fun_headline_variants":["Sparse rate reduction beats 21 complexity measures on generalization","SRR as regularizer improves CIFAR-10/100 accuracy by up to 1.13 pts","Sparse rate reduction tops 21 baselines and improves CIFAR accuracy","Generalization prediction: sparse rate reduction outperforms path-norm and sharpness","Sparse rate reduction: better complexity measure and useful regularizer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline correlation was measured on models with their normalization layers switched off during evaluation, and the paper gives no evidence that switching them off keeps the models in the same order; if it does not, the correlation is an artifact of how the measure was computed rather than a property of the trained models.","fun_headline_variants_meta":{"raw":{"variants":["Sparse rate reduction beats 21 complexity measures on generalization","SRR as regularizer improves CIFAR-10/100 accuracy by up to 1.13 pts","Sparse rate reduction tops 21 baselines and improves CIFAR accuracy","Generalization prediction: sparse rate reduction outperforms path-norm and sharpness","Sparse rate reduction: better complexity measure and useful regularizer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000895,"raw_usage":{"total_tokens":3925,"prompt_tokens":1082,"completion_tokens":2843,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":2742}},"tokens_in":698,"tokens_out":2843,"duration_ms":19606,"temperature":1.0,"reasoning_tokens":2742,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:27:40.946321+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the SRR measure on the same 64 trained models with the Layer Normalization modules left in the forward pass (e.g., folding their scale and shift parameters into the coding-rate computation), and recompute Kendall's $\\tau$ against the generalization gap; if the correlation falls below the sharpness-based baselines, the reported $\\tau = 0.445$ is an artifact of the LayerNorm-removal protocol. A complementary check is to train additional CRATE variants at hyperparameter settings outside the original 64 combinations (for example, learning rate $5\\times 10^{-5}$ or width 576) and test whether the in-sample correlation rank order persists.","supporting_citations":[{"cited_title":"On large-batch training for deep learning: Generalization gap and sharp minima","cited_arxiv_id":null,"evidence_quote":"Defines sharpness-based complexity measures used as baselines in the correlation comparison."},{"cited_title":"White-box transformers via sparse rate reduction","cited_arxiv_id":null,"evidence_quote":"Defines Sparse Rate Reduction and the CRATE architecture; supplies the MSSA and ISTA derivations that the paper inspects and modifies."},{"cited_title":"Learning diverse and discriminative representations via the principle of maximal coding rate reduction","cited_arxiv_id":null,"evidence_quote":"Defines the coding rate function $R(Z)$ and the maximal coding rate reduction principle that SRR extends."},{"cited_title":"Fantastic generalization measures and where to find them","cited_arxiv_id":null,"evidence_quote":"Supplies the methodology for collecting trained models, the stopping criterion, Kendall correlation evaluation, and the baseline complexity measures."},{"cited_title":"Learning fast approximations of sparse coding","cited_arxiv_id":null,"evidence_quote":"Provides the algorithm-unrolling template (learned ISTA-like sparse coding layers) that CRATE and the layer-by-layer SRR analysis build on."}],"review_version":1}