{"id":"e78b1ddb-29f6-4f1a-b46e-bad596e14123","arxiv_id":"2507.15776","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A pretrained transformer shows similar surprisal-decay patterns for linear and exponentially spaced context sampling, suggesting inference-time computations can be separated from model architecture.","lead":"Using a standard language model, this paper compares sequential accumulation of past context with sampling past tokens at exponentially growing intervals during next-word prediction. It argues that the computations a model performs at inference can be dissociated from the architecture it was trained with, a point relevant to how brains and machines build memory.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Methods definition ΔS_k = S_k − S_{k−1} = −ln p(w_t|w_{t−k}) is not valid for a transformer's conditional distribution; the correct expression is a conditional log-ratio. Since Fig.","rationale":"The paper's central claim is that structuring context access at inference time can dissociate architecture from inference computations, specifically that hierarchical access maintains predictive capacity while using fewer computations. For this claim to be supported, the quantitative results in Fig. 1 must measure the actual effect of added context on prediction. The Methods definition of ΔS is not a valid decomposition for a non-Markovian transformer: it asserts S_k − S_{k−1} = −ln p(w_t|w_{t−k}), which drops the intervening context w_{t−1}...w_{t−k+1}. The correct expression is a log-ratio of conditional probabilities. The same error appears in the hierarchical difference. Because every entry of the transition matrix M and all derived rank/efficiency curves use these incorrect values, the reported evidence for 'maintains predictive capacity' collapses. This is an internal inconsistency, not a disagreement with external consensus. I also note the paper provides no code, no formal proof, and no direct perplexity comparison; however, the invalid ΔS identity is the most load-bearing single flaw. The reader's weakest_assumption identified precisely this issue, and I agree. The recommended verdict is unchanged: REJECT, since the central claim lacks valid quantitative support as submitted.","tokens_in":3410,"tokens_out":8864,"duration_ms":89285,"concrete_test":"Using the same Gemma-2B model and tokenizer on the Sherlock Holmes text, recompute the sequential and hierarchical surprisal changes with the correct conditional log-ratio formulas: ΔS^seq_k = ln[p(w_t|w_{t−1}...w_{t−k+1}) / p(w_t|w_{t−1}...w_{t−k})] and ΔS^hier_k = ln[p(w_t|w_{t−1}...w_{t−2^{m−1}}) / p(w_t|w_{t−1}...w_{t−2^m})] with m=⌊log2 k⌋. Rebuild the transition matrix and Fig. 1B–D. If the sequential rank rises toward the linear bound or the hierarchical efficiency advantage reverses, the reported dissociation is an artifact of the invalid identity. If the curves are unchanged, the paper's quantitative conclusions are robust; a further perplexity comparison at matched token budgets would still be needed to establish predictive capacity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Methods defines ΔS^seq_k = S_k − S_{k−1} = −ln p(w_t|w_{t−k}) and ΔS^hier_k = S_{2⌊log2 k⌋} − S_{2⌊log2 k⌋−1} = −ln p(w_t|w_{t−⌈k/2⌉},...,w_{t−k}). These equalities are not valid for an autoregressive transformer: S_k − S_{k−1} equals ln[p(w_t|w_{t−1}...w_{t−k+1}) / p(w_t|w_{t−1}...w_{t−k})], a log-odds contribution of the k-th past token given the intervening words, not the marginal surprisal of w_t given only that token. Similarly, the hierarchical difference is a ratio of full-context probabilities, not a marginal over the oldest block. This is not a minor notational slip: every entry of the global transition matrix M, and hence all rank, amplitude, and efficiency curves in Fig. 1B–D, inherit the error. The claim that hierarchical access 'maintains predictive capacity' is therefore without valid quantitative support. Even after correcting the definition, the paper would still need a direct perplexity/accuracy comparison, because rank and surprisal-change amplitude do not measure predictive capacity. The concern is internal inconsistency, not a conflict with existing consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This short report claims that an unmodified autoregressive transformer can emulate the multi-timescale computations of deep temporal hierarchical models by imposing a hierarchical pattern of context access during iterative inference. The authors define a surprisal-change measure, build a global transition matrix of these changes across context lengths, and report rank, amplitude, and an efficiency ratio for sequential versus hierarchical access. They conclude that hierarchical access maintains predictive capacity while instantiating fewer computations, thereby dissociating model architecture from inference-time computations.","tokens_in":3704,"tokens_out":4933,"duration_ms":50427,"significance":"The conceptual ambition—separating what a model architecture can represent from the computations actually performed at inference—is timely and sensible. If substantiated, the claim that a pretrained transformer can mimic deep temporal computations through structured context access would be of interest to both cognitive neuroscience and machine learning. However, the manuscript's quantitative support is not sound. The central surprisal-change identity is mathematically invalid for a transformer's conditional distribution, the logarithmic rank bound for the hierarchical scheme is largely a consequence of the definition, and no direct predictive-capacity metric is reported. These are load-bearing issues, not presentation problems.","major_comments":[{"comment":"The equality ΔS^seq_k(w_t) = S_k(w_t) − S_{k−1}(w_t) = −ln p(w_t|w_{t−k}) is not valid for an autoregressive transformer. From the definition S_k(w_t) = −ln p(w_t|w_{t−1},...,w_{t−k}), the correct sequential difference is ln[p(w_t|w_{t−1},...,w_{t−k+1})/p(w_t|w_{t−1},...,w_{t−k})], a conditional log-ratio that measures the incremental contribution of w_{t−k} given the intervening context, not the marginal surprisal of w_t given only that token. The analogous hierarchical difference is likewise a ratio of full-context probabilities. Because every entry of the global transition matrix M and hence all rank, amplitude, and efficiency curves in Fig. 1B–D inherit this definition, the paper's central quantitative results are invalid. This is an internal inconsistency in the manuscript's own formalism, not a disagreement with prior work.","section":"Methods, Experimental details (definition of ΔS)"},{"comment":"The logarithmic upper bound on the rank of the hierarchical transition matrix is essentially guaranteed by construction. By definition, ΔS^hier_k depends on k only through ⌊log2 k⌋, so the matrix columns are constant on each dyadic interval and the column space has at most ⌊log2 L⌋ distinct vectors. Observing that the hierarchical rank follows this bound is therefore close to tautological and does not by itself demonstrate efficient computation. Separately, the claimed logarithmic rank for the sequential case is asserted from Fig. 1C without reporting numerical ranks, the tolerance criterion used, or a comparison against the linear upper bound; the 28% figure at L=250 appears without any rank-estimation details.","section":"Methods, rank bound; Results, Fig. 1C"},{"comment":"The phrase \"maintains predictive capacity\" is never supported by a direct predictive metric. Rank and surprisal-change amplitude are not measures of predictive capacity: a model could have many redundant columns (low rank) and still make accurate predictions, or high rank and poor predictions. The paper needs a direct comparison of predictive accuracy (e.g., perplexity or next-token accuracy) between the sequential and hierarchical access schemes at matched context lengths. Without such a comparison, the central claim that hierarchical access \"maintains predictive capacity while instantiating fewer computations\" is unsupported.","section":"Results, Fig. 1B–D; General discussion"},{"comment":"The claim that both schemes \"follow a power-law decay\" is made without any power-law fit, exponent, or goodness-of-fit statistic; the figure shows means and standard errors for a single text. The qualitative \"staircase\" profile of the hierarchical curve is a direct consequence of the dyadic binning in the definition of ΔS^hier, so it does not constitute independent evidence for a distinct computational regime.","section":"Results, Fig. 1B"}],"minor_comments":[{"comment":"The notation in the Methods (e.g., \"∆S seq.\n k (wt)\") has typographical artifacts: superscripts and subscripts are split and spaces appear mid-expression; these should be cleaned up.","section":"Methods, Experimental details"},{"comment":"The definition of \"predictive efficiency\" is inconsistent between the figure caption and the text: Fig. 1D caption says \"ratio of rank to the amplitude of change of surprisal,\" whereas the Results text says \"ratio of the amplitude of the surprisal contribution to the rank.\" The direction of the claimed efficiency difference should also be reconciled with the verbal conclusion that hierarchical access is more efficient.","section":"Figure 1D caption and Results text"},{"comment":"The manuscript does not state the rank-estimation procedure (e.g., singular-value threshold), the number of tokens W used, or what independent units the standard errors in Fig. 1 are computed over; these details are necessary to interpret Fig. 1C.","section":"Methods, Experimental details"},{"comment":"The reference to \"Parr et al. [2025]\" is given as \"Beyond markov: Transformers, memory, and attention, Cognitive Neuroscience, pages 1–19, 2025\"; the page range and volume information are incomplete.","section":"References"}],"recommendation":"reject","confidential_remarks":"For the editor: the manuscript's central figure is built on an invalid mathematical identity, and the hierarchical rank claim is largely a definitional consequence. The paper would need a complete reanalysis with a corrected surprisal-change measure and a direct predictive-capacity comparison before it could be considered further; this goes beyond a standard revision. I would not encourage a full re-review cycle on the current version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a real conceptual point and a broken quantitative core. The dissociation between architecture and inference-time computations is worth saying, and the idea that exponentially spaced context access can mimic deep temporal hierarchies is plausible. But as written, the central measurement does not measure what the authors claim.\n\nThe trouble is in Methods. They define ΔS_k^seq = S_k - S_{k-1} and then assert this equals -ln p(w_t|w_{t-k}). For an autoregressive transformer, S_k - S_{k-1} is log[p(w_t|w_{t-1}...w_{t-k+1}) / p(w_t|w_{t-1}...w_{t-k})]—the contribution of adding the k-th token given the intervening context, not the marginal surprisal of that token alone. The hierarchical version has the same problem: a difference between two full-context surprisals is not equal to a conditional probability over the oldest block. Every entry of the global matrix M inherits this, so the rank, amplitude, and efficiency curves in Fig. 1 don't support the claims.\n\nWhat is actually new? If I take the reported numbers at face value, the observation that the sequential surprisal matrix has rank well below the linear bound (about 28% at L=250) is curious. But that observation is ambiguous because the wrong definition may have been used to construct the matrix, and no code or data is provided to check. The hierarchical low rank is mostly guaranteed: the construction only varies O(log L) dyadic context windows, so a logarithmic rank upper bound is baked in. The paper also never reports perplexity or prediction accuracy, so 'maintains predictive capacity' is an assertion, not a result. There is no deep temporal baseline model, and no efficiency metric that connects rank to actual compute.\n\nCitations look appropriate for the q-bio/active-inference community, and the framing around Parr et al. is fair. But the load-bearing math is internally inconsistent, not just stylistically sloppy.\n\nI would not send this to peer review in its current form. The fix is straightforward: recompute ΔS as the correct log-ratio, report a direct predictive-capacity comparison (perplexity or accuracy), add code and data, and then the conceptual dissociation could be tested properly. As it stands, it is a position piece with an unsupported quantitative claim.","headline":"A conceptually attractive short note whose central Methods equality is wrong; the empirical curves should not be trusted until reanalyzed with the correct surprisal difference.","tokens_in":4204,"tokens_out":5344,"would_cite":false,"duration_ms":54147,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that inducing hierarchical context access during iterative inference lets an unmodified transformer emulate the multi-timescale computations of deep temporal models, without changing its architecture.","keywords":["deep temporal structures","transformers","language models","structured context access","iterative inference","surprisal","hierarchical temporal factorisation","non-Markovian sequence modelling"],"falsifier":"Recompute the rank and efficiency curves using a surprisal-change definition that keeps the intervening context words, for example $\\Delta S_k=-\\ln p(w_t\\mid w_{t-k},\\dots,w_{t-1})+\\ln p(w_t\\mid w_{t-k+1},\\dots,w_{t-1})$; if the hierarchical staircase and logarithmic rank profile disappear, the dissociation claim is an artifact of the ignored-context approximation.","tokens_in":3180,"feed_emoji":"🧠","tokens_out":10343,"duration_ms":102378,"temperature":0.7,"pith_summary":"This paper tries to establish that model architecture and inference-time computation can be dissociated: an unmodified transformer, trained only for next-token prediction, can emulate the multi-timescale computations of deep temporal models by changing how it accesses context during iterative inference. The authors show that querying past tokens at exponentially increasing intervals produces a stepwise surprisal profile, matching the hierarchical temporal factorisation of deep temporal models, while sequential context access produces a smooth decay and uses a smaller fraction of the available rank. If this is right, explicit hierarchical structure is not required for hierarchical computation, and the way predictions are constructed and refined can be designed separately from the architecture that learns the predictive distribution. This matters because it redirects attention from model structure to inference procedure in both machine learning and cognitive modelling.","feed_headline":"Transformer mimics deep temporal models without rewiring","feed_subtitle":"A study changes which past tokens a model queries at inference, matching hierarchical predictive capacity with fewer computations","key_machinery":"The central mechanism is hierarchical context access during iterative inference: instead of adding context tokens one by one, the model accumulates token sets whose length equals the current context length, reproducing the inductive bias of temporal hierarchies in which each stage doubles the integration window. The quantitative evidence is carried by the context-surprisal transition matrix $M\\in\\mathbb{R}^{W\\times L}$ with entries $M_{t,k}=\\Delta S_k(w_t)$, where $\\Delta S_k^{\\mathrm{seq}}(w_t)=-\\ln p(w_t\\mid w_{t-k})$ for sequential access and $\\Delta S_k^{\\mathrm{hier}}(w_t)=S_{2^{\\lfloor\\log_2 k\\rfloor}}(w_t)-S_{2^{\\lfloor\\log_2 k\\rfloor}-1}(w_t)$ for hierarchical access. The rank of this matrix, together with the ratio of surprisal amplitude to rank, is what links the query schedule to the claim of fewer, sufficient computations.","core_discovery":"The paper claims that inducing hierarchical temporal factorisation during iterative inference lets an autoregressive transformer mimic deep temporal computations. Using surprisals of next-token predictions under sequential and hierarchical context accumulation, the authors find that hierarchical access follows a stepwise 'staircase' profile and that the rank of the context-surprisal transition matrix reaches its logarithmic upper bound, while sequential access decays smoothly and uses only 70 of 250 rank units at the longest context tested. They take this as evidence that the computations that construct and refine predictions are not bound to the underlying model architecture, and that deep temporal computations are not unique to models with explicit temporal hierarchies.","pith_inferences":["A direct test of the numerical claim would be to repeat the rank and efficiency analysis with a surprisal-change measure that does not drop the intervening context words; if the staircase and logarithmic-rank pattern vanish, the dissociation result depends on the Markov-like approximation in the Methods.","The same hierarchical query schedule could be applied to other autoregressive model families; observing the same effect there would show that the mechanism is a property of iterative inference rather than of attention.","A cognitive reading the authors gesture at but do not develop is that neural circuits could implement multi-timescale inference by changing which past states they sample during inference, without rewiring their architecture."],"forward_implications":["Hierarchical context access reproduces the stepwise multi-timescale surprisal profile of deep temporal models in an unmodified transformer, so explicit hierarchical architecture is not required for that profile.","Sequential context access is numerically inefficient for iterative refinement: at context length 250 it uses only 70 of the 250 possible rank units, while hierarchical access follows its logarithmic upper bound.","Repeatedly doubling the queried context interval means each added level of the hierarchy expands the effective context exponentially for the same number of computational steps.","If the dissociation holds, inference procedures and model architecture can be designed separately; how a predictive distribution factorises need not dictate the computations performed at prediction time."],"supporting_citations":[{"why":"Supplies the contrast between autoregressive and deep temporal models for non-Markovian sequence modelling that the paper builds on.","marker":"Parr et al. [2025]"},{"why":"Provides the hierarchy-of-timescales result that stacking layers with doubling integration windows gives an exponentially expanding receptive field.","marker":"[Kiebel et al., 2008]"},{"why":"Supplies the dilated-convolution inductive bias that motivates reproducing temporal and hierarchical depth through hierarchical accumulation.","marker":"[Yu and Koltun, 2015]"},{"why":"Defines deep temporal models as hierarchical latent-variable models with Bayesian inference, the computations the transformer is claimed to emulate.","marker":"[Friston et al., 2018]"},{"why":"Provides the text corpus on which the surprisal and rank comparisons are computed.","marker":"[Doyle, 1992]"}],"fun_headline_variants":["Inference-time context access mimics deep temporal models","Autoregressive transformer mimics deep temporal via iterative context","Model architecture does not dictate inference computations","Context structuring at inference replicates deep temporal factorisation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on treating the change in surprisal when the context grows by one token as if it isolated that token's contribution, which assumes the words in between do not matter; transformer predictions do not satisfy that independence, and the paper does not test it.","fun_headline_variants_meta":{"raw":{"variants":["Inference-time context access mimics deep temporal models","Autoregressive transformer mimics deep temporal via iterative context","Model architecture does not dictate inference computations","Context structuring at inference replicates deep temporal factorisation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000378,"raw_usage":{"total_tokens":1920,"prompt_tokens":765,"completion_tokens":1155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":381,"completion_tokens_details":{"reasoning_tokens":1097}},"tokens_in":381,"tokens_out":1155,"duration_ms":9245,"temperature":1.0,"reasoning_tokens":1097,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:23:50.395170+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the rank and efficiency curves using a surprisal-change definition that keeps the intervening context words, for example $\\Delta S_k=-\\ln p(w_t\\mid w_{t-k},\\dots,w_{t-1})+\\ln p(w_t\\mid w_{t-k+1},\\dots,w_{t-1})$; if the hierarchical staircase and logarithmic rank profile disappear, the dissociation claim is an artifact of the ignored-context approximation.","supporting_citations":[],"review_version":1}