{"id":"e80080c6-a740-4856-a206-040e577c43c2","arxiv_id":"2504.19561","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Effective state-size, the rank of a causal operator submatrix, is proposed and empirically validated as a memory-utilization proxy that beats cache size at predicting recall-task accuracy and reveals how LLMs reset context at separator tokens.","lead":"This paper introduces effective state-size (ESS), a number computed from a sequence model's input-output operator that estimates how much of the model's memory capacity is actually being used for a given input. The authors show ESS correlates with accuracy on memory-heavy recall tasks, often better than raw cache size, and use it to guide model compression, initialization, and regularization.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"For LIVs, ESS is only a lower bound, and the causal gap is bridged solely by correlation evidence that depends on the kv-normalization and post-hoc filters; a direct test of whether ESS equals usable state size is missing.","rationale":"The paper has genuine strengths: Theorem 3.2 is correct for input-invariant systems, the LIV caveat is disclosed honestly, the synthetic task sweep is broad, and the applications (initialization, regularization, distillation, state modulation) are plausible and often supported by controlled interventions. My objection is not that ESS is useless; it is that the strongest headline claim about LIV memory utilization is currently supported only by correlation, and that correlation is conditional on the kv normalization, the accuracy filter, the epoch-0 drop, and unquantified uncertainty. A direct behavioral test of whether ESS equals the smallest causally usable state dimension would convert the proxy claim into a quantified claim and would reveal how tight the lower bound is for GLA, WLA, and LA. Until that test is run, the reader's CONDITIONAL verdict is the right one, so no change is needed.","tokens_in":79942,"tokens_out":11377,"duration_ms":140191,"concrete_test":"Take a trained GLA model from the MQAR sweep (e.g., TSS=256, kv=64) and compute entropy-ESS and tolerance-ESS from Hi. Then perform a causal state-truncation test: during a forward pass, replace the recurrent state at each step with its projection onto the r-dimensional subspace spanned by the top-r singular directions of the input-state map C_i defined in Section D.2.2, for r = 4, 8, 16, ..., 256, and measure MQAR accuracy. If the smallest r that preserves accuracy is consistently close to ESS, the metric measures usable causal memory; if the required r is much larger than ESS or tracks TSS instead, then the acausal lower bound is too loose and the paper's interpretation is unsupported. Repeat across checkpoints and featurizers (GLA, WLA, LA) to check whether the gap depends on architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that rank(Hi) measures how much of a model's state is actually used for memory. For input-invariant systems this is Theorem 3.2. For LIVs, however, Section 3 concedes that the minimal realization process may not preserve causality, so ESS is only a lower bound on any causal recurrence. Every model in the headline validation (GLA, WLA, LA, SA) is an LIV, so the entire interpretation rests on an unproven empirical bridge: that a single forward pass's Hi rank tracks usable causal memory. The bridge is tested only indirectly, via correlation with accuracy, and the correlations are sensitive to disclosed choices: Section E.3 states the kv normalization is 'critical' for observing the cross-task correlations, SA points with accuracy above 0.95 are filtered to create variation, epoch 0 is dropped for within-model correlations, and no confidence intervals are reported. The strongest claim (ESS/kv predicts performance better than TSS/kv) could therefore be driven by these choices rather than by ESS tracking true memory utilization. Moreover, because the minimal realization of the realized operator can be acausal, ESS may systematically understate the state size required by a causal LIV, making the metric a loose lower bound rather than a direct measurement of engaged memory.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces effective state-size (ESS), a per-sequence-index measure of memory utilization computed as the rank (or its tolerance- or entropy-smoothed variant) of the submatrix H_i = T_{i:,:i-1} of the materialized causal operator T = f_T(u). The authors prove (Theorem 3.2, Appendix D) that for input-invariant linear systems, rank(H_i) is the minimal state size of any recurrent realization, and they argue that for input-varying linear operators (LIVs) it lower-bounds the state size. They validate ESS by training GLA/LA/WLA/SA models on MQAR, selective copying, and compression, showing that ESS/kv correlates with accuracy better than TSS/kv (Section 4.1), that ESS tracks within-task performance over training (Section 4.2), and that failure modes (state saturation, state collapse) are visible in ESS/TSS. Applications include ESS-informed featurizer analysis of S6 at initialization, an A-to-I regularizer, model-order reduction via distillation, and the observation of 'state modulation' in LLM states at separator tokens.","tokens_in":80105,"tokens_out":15741,"duration_ms":155411,"significance":"The central theoretical contribution—equating rank(H_i) with the minimal recurrent state size for input-invariant operators—is a clean application of classical minimal realization theory, and the proof sketches in D.2.4 and D.3 are plausible. The entropy-ESS default is parameter-free, the PyTorch implementation is shipped in E.1.3, and the analysis choices that affect the headline correlations (kv normalization, the accuracy>0.95 filter for SA, the epoch-0 drop) are disclosed in E.3 rather than hidden; this transparency is a genuine strength. If the empirical bridge for LIVs holds, ESS would be a useful, cheap diagnostic: it exposes that S6's state initialization under-utilizes added capacity (Section A) and that GLA/WLA suffer state collapse that an A-to-I regularizer can fix (Section B)—both falsifiable claims. The main risk is that every model in the validation set is an LIV, for which ESS is only a lower bound on causal state size, so the memory-utilization interpretation rests on the correlation evidence alone.","major_comments":[{"comment":"Section 4.1 states that 'measured over the entire task-model space, ESS/kv exhibits a significantly higher correlation with accuracy than TSS/kv,' but Section E.3 discloses three choices that materially shape this result: SA configurations with accuracy above 0.95 are filtered from the cross-task correlation, epoch 0 is dropped from the within-task correlations, and the kv normalization is called 'critical for observing the cross task-model correlations' (unnormalized ESS correlations 'break down,' per Figure 10g). Since this is the paper's principal validation of ESS for LIVs, the Section 4.1 claim should be re-scoped to the non-saturated task-model configurations, and the authors should report the number of points entering each correlation plus robustness checks such as unfiltered SA, alternative normalizations, or a partial correlation controlling for kv. The epoch-0 exclusion likewise deserves reporting alongside Figure 3, since Figure 25 shows a phase in which ESS decreases while accuracy is flat, which contradicts the 'ESS tracks performance' reading at initialization.","section":"§4.1 and §E.3"},{"comment":"The paper concedes that for LIVs the minimal realization of the materialized operator may be acausal, so ESS is only a lower bound on the state size required by a causal recurrence; the 'proxy for memory utilization' interpretation for LIVs is then justified only by the correlation evidence of Section 4. A direct test is available but not performed: Section D.2.2 defines the 'causally determinable minimal state-size (causal ESS)' as the rank of the causal projection [A_{i-1;1}B_0 ... A_{i-1}B_{i-2} B_{i-1}], which is computable from the trained features of GLA/LA/WLA and is exactly the state dimension a causal recurrence actually engages. Computing causal ESS on the sweep and comparing it with ESS (e.g., a scatter plot or the ratio ESS/causal-ESS) would quantify the acausality gap and either support or bound the memory-utilization interpretation. Without this comparison, the central claim cannot be distinguished from the weaker claim that ESS predicts performance.","section":"§3 'Interpreting effective state-size'; §D.2.2"},{"comment":"The headline comparison 'ESS/kv exhibits a significantly higher correlation with accuracy than TSS/kv' is reported without confidence intervals, standard errors, or a significance test, despite the large scatter in Figure 2 and the non-independence of the points (they are bucketed by featurizer, TSS, kv, and sequence length, with ESS averaged over only 8 samples). A bootstrap over task-model configurations, or at least a report of the sample size and a correlation-difference test, is needed to support the word 'significantly'; as presented, part of the gap in Figure 4 may reflect the discreteness of TSS (a hyperparameter taking a handful of values) rather than the greater informativeness of ESS.","section":"§4.1, Figure 4"}],"minor_comments":[{"comment":"The complexity is stated as O((dℓ)^3) per layer, but the per-sequence-index definition requires an SVD of H_i at every i, giving O(d · Σ_i (ℓ−i)·i·min(ℓ−i,i)) = O(dℓ^4) per layer if all indices are computed; please specify whether the stated bound refers to a single index and describe the protocol used for the (total ESS)_i plots in Section 5.2.","section":"§3.1 'Computational complexity of ESS'"},{"comment":"The reference list contains stray page annotations such as '(pages 1, 19)' after URLs (e.g., Abnar & Zuidema, Akaike, and most other entries); these appear to be leftover bibliography artifacts and should be removed.","section":"References"},{"comment":"Figure 6 and the state-saturation discussion use tolerance-ESS at tolerance 1e−3 (per Section E.3), but the main-text caption and prose do not state this; please add the tolerance to the figure caption.","section":"Figure 6 / §4.2"},{"comment":"Equation (A.0.1) writes the GLA gate as diag(sigmoid(Wu)^{1/β}), which is inconsistent with Eq. (E.2.6) in Section E.2, diag(sigmoid(W_{A2}W_{A1}u_i)^{1/β}); the two formulations should be aligned.","section":"§A, Eq. (A.0.1)"},{"comment":"The 'preemptive state modulation' explanation attributes to SSMs a learned reliance on explicit EOS signals, but the evidence is descriptive ESS trajectories on a small fixed set of sentences; a control condition (e.g., non-semantic separator tokens) or a more cautious framing would strengthen the causal reading.","section":"§5.2 and §E.8"}],"recommendation":"major_revision","confidential_remarks":"For the editor: I read the paper in good faith, and the stress-test concern does land, though the authors' transparency softens it: the theory for input-invariant operators is standard and sound, and the disclosed analysis choices make the empirical claims checkable in principle. My main requests are the causal-ESS comparison that the paper's own Section D.2.2 makes cheap, proper uncertainty quantification for the headline correlation, and a re-scoped Section 4.1 claim. I regard this as a major revision rather than a rejection; the metric is likely useful even if its LIV interpretation is ultimately only a heuristic. For scope, the LLM 'state modulation' section is suggestive rather than conclusive, and I would not anchor acceptance on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives the community a genuinely useful measurement: ESS, the rank of the operator submatrix Hi, packaged with tolerance and entropy variants. For LTI systems the theory is classical minimal realization and they cite it properly. What's new is the packaging, the empirical validation on synthetic recall tasks, and the downstream applications—distillation, initialization diagnostics, regularization, and the state-modulation finding in LLMs. The code in the appendix is a real asset.\n\nThe empirical case is solid but needs a careful read. Every model in the validation (GLA, WLA, LA, SA) is an LIV, and for LIVs the paper honestly concedes that the minimal realization may not preserve causality, so ESS is only a lower bound on the state size of any causal recurrence. That means the whole interpretation—that ESS tracks usable memory—rests on correlation evidence. The correlations are genuine but fragile: they require the kv normalization the authors call critical, they drop epoch 0, they filter SA models with accuracy above 0.95, and they report no confidence intervals. The choice between entropy- and tolerance-ESS is made per analysis, which is a bit ad hoc. These are the soft spots, and they are fixable. The stress-test note is right: there is no direct test that ESS equals usable state size for an LIV.\n\nCredit where due: the paper is unusually transparent about its analysis choices and doesn't hide the limitations of the theory. The state modulation result is a real observation—attention models and gated SSMs respond differently to separator tokens, and that connects nicely to bigram recall. The initialization analysis of S6 is a nice diagnostic.\n\nWho should read it: anyone building or analyzing linear-attention, Mamba-style, or hybrid architectures. It will be cited.\n\nRecommendation: send to peer review. The referee process should push for uncertainty quantification, robustness of the correlations to the disclosed filters, and ideally a direct test of the causal realization claim. The paper deserves that effort.","headline":"A useful, well-packaged memory-utilization metric whose LTI theory is solid and whose LIV interpretation rests on strong but fragile correlations; needs a revision that hardens the empirical bridge.","tokens_in":80774,"tokens_out":2398,"would_cite":true,"duration_ms":25970,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A rank-based metric measures how much memory a sequence model actually uses, not just how much it could.","keywords":["effective state-size","memory utilization","sequence models","minimal realization","linear input-varying systems","associative recall","model distillation","state collapse"],"falsifier":"Find a realistic input-varying sequence model for which ESS decreases or stays flat over training while recall accuracy rises substantially, and where a competing capacity-based measure explains the accuracy differences better across a held-out task-model sweep. Concretely: train a family of gated linear attention models on MQAR, sweep kv and TSS as in the paper, and compute the split-half correlation of length-normalized ESS/kv against accuracy; if TSS/kv or a simple attention-focused spectral statistic wins or ties across the randomized halves, the claim that ESS outperforms capacity as a memory-utilization proxy would fail.","tokens_in":79646,"feed_emoji":"🧠","tokens_out":3595,"duration_ms":30624,"temperature":0.7,"pith_summary":"The paper introduces effective state-size (ESS), a single number computed from a model's realized linear operator, and argues that ESS measures how much of a sequence model's state is actually engaged for a given input. Motivated by classical minimal realization theory, the authors show that for linear systems the rank of a certain operator submatrix equals the minimal state size of an equivalent recurrence. For the larger class of input-varying linear systems (which include attention, gated recurrences, and convolutions), they argue the same quantity serves as a lower bound on the required state and, empirically, as a much stronger proxy for task performance than cache size. They demonstrate that across associative recall, selective copying, and compression, task-normalized ESS correlates with accuracy far better than the theoretically realizable state-size (TSS), and they use ESS to expose two training failure modes, guide distillation and regularization, and quantify how language models reset their memory at delimiter tokens. The paper's central claim is that what matters for memory is not how large the state is, but how much of it the model actually uses, and that ESS is the right way to measure that.","feed_headline":"A rank-based metric tells how much memory a model actually uses","feed_subtitle":"Effective state-size beats cache size as a predictor of recall performance in sequence models.","key_machinery":"The central object is the strictly lower-triangular operator submatrix $\\mathbf{H}_i = \\mathbf{T}_{i:,:i-1}$ of the causal operator $\\mathbf{T}$ realized by the model. The paper proves that $\\mathrm{rank}(\\mathbf{H}_i)$ is the minimal state size of any recurrent realization of an input-invariant linear system, and then computes two rank-smoothed versions of this quantity: tolerance-ESS, which counts singular values above a threshold $\\tau$, and entropy-ESS, which exponentiates the spectral entropy of $\\mathbf{H}_i$. The paper also defines theoretically realizable state-size (TSS) as the model-recurrent state dimension, which upper-bounds ESS and plays the role of a 'capacity' baseline throughout the comparisons.","core_discovery":"For a causal sequence model materialized as a linear operator $\\mathbf{T}$, the effective state-size at sequence index $i$ is the rank of the submatrix $\\mathbf{H}_i \\equiv \\mathbf{T}_{i:,:i-1}$, which collects the strictly causal part of the operator up to time $i$. By minimal realization theory, this rank exactly equals the minimal state size of any recurrence realizing the same input-invariant operator (Theorem 3.2); for input-varying linear systems it is a lower bound, requiring an empirical bridge. The paper's principal empirical discovery is that task-adjusted ESS ($\\mathrm{ESS}/kv$ for associative recall) correlates with post-training accuracy across a sweep of four featurizer families (gated, weighted, and plain linear attention, plus softmax attention) substantially better than the corresponding cache-size measure $\\mathrm{TSS}/kv$. It also identifies two failure modes: state saturation, where ESS is pinned near TSS because the state is too small; and state collapse, where a sufficiently large state is left largely unused. On language models, ESS exhibits dips when the model encounters end-of-speech tokens, a phenomenon the authors call state modulation, which they connect to recall performance on bigram tasks and to cross-architectural differences in handling context boundaries.","pith_inferences":["ESS could be adapted to probe the per-layer or per-head distribution of memory use in very large models, potentially identifying which layers are redundant before any training run.","The state modulation measurement suggests a testable design principle for hybrid architectures: place layers with unbounded state capacity after layers that need to reset, which is consistent with why attention-first hybrids fail in the paper's experiments.","The task-normalization step (dividing by the number of key-value pairs) is a place where a deeper theory could be built: if ESS were grounded in a task-dependent information requirement, the same metric could be extended to natural-language recall without hand-picking the normalizer.","Entropy-ESS and tolerance-ESS disagree on state collapse, which suggests that a spectrally graded version of ESS could sharpen failure-mode diagnosis beyond the paper's binary categorization."],"forward_implications":["ESS can serve as a diagnostic before and during training: measuring ESS at initialization can reveal whether an architecture is preconditioned to use all its states, as demonstrated by the poor ESS-TSS scaling of S6 at initialization.","ESS can guide model-order reduction: a teacher layer with high ESS leads to larger distillation loss, so ESS is a predictor of compressibility that can steer the choice of student state sizes.","ESS can be targeted by regularizers: pushing gated and weighted linear attention recurrences toward the identity, which increases ESS, cures state collapse and lets them outperform plain linear attention on recall tasks.","ESS is a quantitative tool for studying language models: the dip in ESS at separator tokens (state modulation) can be used to compare architectures, and the hierarchy SA>GLA>WLA>LA in state modulation mirrors their bigram recall performance.","ESS reconciles an apparent paradox in efficient sequence models: a model can have a large cache yet realize low memory utilization, which explains why capacity metrics alone mispredict recall behavior."],"supporting_citations":[{"why":"Provides the classical minimal realization theory that is the basis for the minimal-state-size interpretation of the operator submatrix rank.","marker":"[DeWilde & van der Veen, 1998]"},{"why":"Supplies the effective-rank construction that becomes entropy-ESS.","marker":"[Roy & Vetterli, 2007]"},{"why":"Defines the synthetic task space (MQAR, selective copying, compression) used to validate ESS across task-model configurations.","marker":"[Poli et al., 2024]"},{"why":"Defines the gated linear attention (GLA) architecture, one of the central featurizers in the empirical sweeps.","marker":"[Yang et al., 2024a]"},{"why":"Gives the linear attention construction that is reformulated here as a recurrence and used as the LA featurizer.","marker":"[Katharopoulos et al., 2020]"},{"why":"Introduces MQAR and bigram recall perplexity, the recall-oriented benchmarks whose task-adjusted ESS is the primary correlation target.","marker":"[Arora et al., 2023]"},{"why":"Provides the operator/activation-matching distillation procedure used to test ESS as a predictor of model-order reduction.","marker":"[Bick et al., 2024]"},{"why":"Defines the S6 featurizer whose poor ESS-TSS scaling at initialization is diagnosed in the initialization experiments.","marker":"[Gu & Dao, 2024]"}],"fun_headline_variants":["Effective state-size: a rank-based metric that beats cache size","Forget cache size: effective state-size reveals true memory use","Rank-based effective state-size predicts recall better","Effective state-size: how much memory a model truly uses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That the numerical rank of a single forward pass's operator submatrix $\\mathbf{H}_i$ tracks how much trainable memory a model actually engages when the model is input-varying, since the formal minimal-state-size theorem only covers input-invariant systems and LIVs are supported by an empirical correlation bridge.","fun_headline_variants_meta":{"raw":{"variants":["Effective state-size: a rank-based metric that beats cache size","Forget cache size: effective state-size reveals true memory use","Rank-based effective state-size predicts recall better","Effective state-size: how much memory a model truly uses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000834,"raw_usage":{"total_tokens":3692,"prompt_tokens":1048,"completion_tokens":2644,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":2578}},"tokens_in":664,"tokens_out":2644,"duration_ms":20569,"temperature":1.0,"reasoning_tokens":2578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:50:32.620396+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a realistic input-varying sequence model for which ESS decreases or stays flat over training while recall accuracy rises substantially, and where a competing capacity-based measure explains the accuracy differences better across a held-out task-model sweep. Concretely: train a family of gated linear attention models on MQAR, sweep kv and TSS as in the paper, and compute the split-half correlation of length-normalized ESS/kv against accuracy; if TSS/kv or a simple attention-focused spectral statistic wins or ties across the randomized halves, the claim that ESS outperforms capacity as a memory-utilization proxy would fail.","supporting_citations":[{"cited_title":"and van der Veen, A","cited_arxiv_id":null,"evidence_quote":"Provides the classical minimal realization theory that is the basis for the minimal-state-size interpretation of the operator submatrix rank."}],"review_version":1}