{"id":"2be6328c-e023-40f5-9f4f-f5614dadb140","arxiv_id":"2607.22646","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Pre-trained LLMs on HMM next-token prediction appear to use finite-window Soft n-gram-like learned predictors rather than Bayes-optimal inference, as shown by a new activation-probing and causal-patching pipeline.","lead":"Probing the internal activations of pretrained language models while they predict Hidden Markov Model sequences, this paper argues the models are not running the statistically optimal Bayesian filter; instead they implement short-window, gradient-learned n-gram-like prediction whose internal representation shifts with the task. A new layer-wise probing-and-intervention method, PAP, shows that what is decodable in a model's activations is not always what causally drives its ou","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PAP's Soft n-gram target is not finite-window: W_t is refit on the full prefix, so causal patching of W_t z_t cannot support the paper's finite-window conclusion.","rationale":"I read the paper in good faith: it is a serious, well-structured attempt to connect behavior, theory, and internal activations. The behavioral narrowing, the constructive Transformer implementations, the small-Transformer finite-window result, and the OLMo negative control are genuine contributions. The weakest point is exactly where the reader placed it: PAP's causal test for Soft n-gram does not establish the finite-window part of the central claim because the Soft n-gram target W_t z_t uses a weight W_t refit on the entire prefix. This makes the probed representation full-context and output-like, so the high IIA could reflect output writability rather than a finite-window algorithm. The proposed test — fixing W or comparing against the model's own output distribution — would settle the question. Since this is addressable without changing the paper's architecture, the existing CONDITIONAL verdict remains appropriate; I do not move it.","tokens_in":72660,"tokens_out":8883,"duration_ms":96709,"concrete_test":"Repeat the PAP probe-inverse experiment (Section 4.3, Figure 12) on Qwen3-1.7B in the belief-dominated regime, but replace the Soft n-gram target r_t = W_t^{ridge} z_t with r_t = W^{fixed} z_t, where W^{fixed} is fit once on held-out sequences (same window n=4) and held constant across positions. If IIA remains comparable, the finite-window feature z_t is causal; if IIA drops substantially (e.g., below the Oracle-belief IIA), the original effect was carried by the full-context W_t and the finite-window conclusion fails. As an additional control, patch the model's own unperturbed next-token distribution p_model(·|o_{1:t}^{tgt}); if this achieves IIA comparable to Soft n-gram, the intervention is largely output injection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that pre-trained LLMs use finite-window, gradient-descent-like Soft n-gram inference rather than Bayesian/spectral tracking — rests on PAP causal interventions showing that patching the Soft n-gram belief steers predictions. But the probed Soft n-gram belief (Table 2, App. B.3) is r_t = W_t ψ_t^{(n)}, where W_t is the ridge-regression solution fit on the entire prefix o_{1:t} at each position t. Therefore r_t depends on the full context through W_t; it is not a finite-window quantity. Lemma 3.1's matrix G_{t,n} likewise depends on the belief b_{t-n}, so the 'n-gram' label applies only to the feature, not to the algorithm. Injecting r_t^tgt with probe-inverse patching is thus closer to writing a full-context, output-space next-token summary into the residual stream than to demonstrating a finite-window internal state. The Bigram control does not resolve this: Bigram is far from the model's output in the belief-dominated regime, whereas the ridge-refit Soft n-gram is, by construction, fit to match next-token predictions from the full prefix. Hence the observed causal effectiveness is consistent with the model's output being linearly writable from the residual stream and does not identify the algorithm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage approach to identify the algorithm implemented by pretrained LLMs when predicting next observations from HMM sequences. Stage 1 benchmarks 12 LLMs against classical and learning-based HMM predictors across 75 configurations and finds that no single baseline matches LLM behavior everywhere, narrowing the candidate space to Linear n-gram, Non-linear n-gram, and Spectral classes. Stage 2 gives theoretical Transformer constructions for all three classes, unified through a nonlinear-feature representation, and reports that a small Transformer trained on a single HMM converges to a finite-window linear n-gram predictor. Stage 3 introduces the Principal Activations Probe (PAP), a PCA+ridge probing method with causal activation patching, and applies it to Qwen3-1.7B. The paper concludes that pre-trained LLMs do not implement statistically optimal iterative inference, but instead use finite-window, gradient-descent-like Soft n-gram approximations, and that PAP causal interventions establish this. The manuscript is extensive and contains a large amount of empirical and theoretical material, but the central conclusion depends on claims about the Soft n-gram representation that are not fully supported by the evidence presented.","tokens_in":72982,"tokens_out":6047,"duration_ms":73068,"significance":"If the central claim were fully supported, the paper would be a significant step toward mechanistic identification of in-context learning algorithms in LLMs, offering a template (behavioral comparison + theoretical constructions + probing + causal intervention) that could transfer to other algorithmic families. The paper has genuine strengths: a broad and systematic empirical comparison; explicit Transformer constructions for several algorithmic classes; a carefully validated small-Transformer experiment; and a probing method with null tests, multiple patch types, and cross-model-size checks. These components are valuable and likely reusable. However, the headline conclusion about finite-window Soft n-gram inference in pre-trained LLMs is currently underdetermined by the PAP results, for the reasons detailed in the major comments. The disconnect between the strong conclusion and the fragility of the causal identification is the main gap between the paper's potential and its current form.","major_comments":[{"comment":"The 'Soft n-gram belief' r_t = W_t z_t is not a finite-window quantity. Appendix B.3 states that for the Soft n-gram entry, Ridge MSE weights are used: W_t is the closed-form ridge solution fit on the full prefix o_{1:t-1}. Thus r_t depends on the entire context through W_t, and only the feature z_t is finite-window. Patching r_t^{tgt} via Eq. (4) therefore transfers full-context information from the target sequence. The observed causal efficacy is consistent with the model using long-range information encoded in W_t, and does not identify a finite-window mechanism. The Bigram control does not control for this because Bigram has no full-context adaptive operator. This directly undermines the central claim in Section 6 that LLMs use 'finite-window, gradient-descent-like' inference.","section":"§4.2 / Table 2 / App. B.3"},{"comment":"Lemma 3.1/Lemma C.1 does not support the statement that the prediction problem 'reduces to learning a linear operator.' In Eq. (11), the matrix G_{t,n} depends on t through the unobserved belief b_{t-n} and the normalizer α_{t,n}. A fixed linear matrix W such that q_{t+1} ≈ W φ_t is an approximation, not an exact reduction. Theorem 3.2 shows that a Transformer can emulate gradient descent on a fixed logistic-regression parameter, but does not show that the resulting W approximates the time-varying Bayes-optimal G_{t,n}. The manuscript should state this approximation explicitly, characterize its error, and identify regimes where it is accurate; otherwise the theoretical grounding for the 'Soft n-gram' class as a candidate algorithm is incomplete.","section":"§3.1 / Lemma C.1"},{"comment":"The probe-inverse patch for Soft n-gram writes an N-dimensional next-token distribution (W_t z_t) into the residual stream, whereas the Oracle patch targets an M-dimensional hidden-state belief. Because the Soft n-gram target is, by construction, a predictor of the next token, high IIA may merely reflect that the model's output is linearly writable from the residual stream, not that the algorithm's internal state is being used. A discriminating control should patch a fixed finite-window n-gram distribution with no full-context operator, or compare IIA against the Oracle belief with matched dimensionality. Without such a control, the causal identification of Soft n-gram as the operative algorithm is fragile.","section":"§4.1, Eq. (4)"},{"comment":"The conclusion that LLMs use 'gradient-descent-like' approximations is not supported by the pretrained-LLM experiments. The PAP target for Soft n-gram uses Ridge MSE (closed-form ridge regression), not online gradient descent; the only evidence of gradient-descent-like behavior comes from the small-Transformer experiment in Section 3.2, which trains on a single fixed HMM and does not address pre-trained models. The conclusion should distinguish between the small-Transformer finding and the pretrained-LLM finding, and either provide direct evidence for online GD in pretrained LLMs or soften the claim.","section":"§6 / §3.2"}],"minor_comments":[{"comment":"The abstract says PAP 'reveals low-dimensional linear representations that causally drive model predictions'; for Soft n-gram the target r_t is N-dimensional, so 'low-dimensional' should be qualified or the PCA dimension should be made explicit.","section":"Abstract / §1"},{"comment":"The empirical comparison reports no error bars or confidence intervals. Given the checklist claims closed-form linear regression for crucial experiments, adding error bars for the behavioral comparisons would strengthen the claims.","section":"§2.3"},{"comment":"Ridge regularization λ is said to be selected on a held-out validation split, but no details are given on the sweep or the criterion. Please specify.","section":"Appendix B.3"},{"comment":"The Neumann-series approximation for f_bi requires conditions on the spectral radius of (I - U^T P_2,1); these conditions are not stated. Since this is part of a capability construction rather than the main empirical claim, this is a clarity issue, not a blocking one.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong candidate in scope and ambition, but the causal-identification concerns raised in the major comments are load-bearing. The central conclusion about finite-window, gradient-descent-like inference in pretrained LLMs is currently not established, because the Soft n-gram belief used in PAP is full-context through W_t and the patched object is an output-space distribution. The authors should be encouraged to add discriminating controls (e.g., fixed finite-window n-gram, matched-dimensionality belief targets, or patching W_t and z_t separately) and to explicitly acknowledge the time-dependence of G_{t,n} in Lemma 3.1. With those changes, the paper could become a valuable contribution. I would not recommend rejection, as the issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is worth a referee's time, but the headline claim—that pre-trained LLMs use finite-window, gradient-descent-like Soft n-gram inference—outruns the evidence. The probing pipeline (PAP) and the small-transformer validation are genuine contributions; the finite-window conclusion is not supported by the Soft n-gram probe as implemented.\n\nWhat's genuinely new: a 12-model/75-configuration behavior benchmark that narrows candidate HMM-ICL algorithms to three classes; a PAP layer-wise probing/intervention method that cleanly separates decodability from causal use (the OLMo contrast, high R2 but failed patching, is a nice control); and a clear demonstration that a small transformer trained on a fixed HMM converges to a finite-window linear n-gram predictor, with learned weights matching the theory. Lemma 3.1 is a clean re-derivation. These pieces are real.\n\nThe soft spot is the central one for the paper's main claim. The Soft n-gram belief is r_t = W_t ψ_t^(n), where W_t is the ridge-regression solution fit on the entire prefix o_{1:t}. So W_t carries full-context information. The 'n-gram' is only in the feature, not in the algorithm. Injecting W_t z_t into the residual stream is therefore closer to writing a full-context next-token distribution into the readout than to demonstrating a finite-window internal state. The Bigram contrast doesn't fix this: Bigram is far from the model's output in the belief-dominated regime, while the ridge-refit Soft n-gram is by construction close to the model's next-token predictions. The observed causal effectiveness of the Soft n-gram patch is then consistent with a linearly writable output, without identifying the algorithm. The paper's own Lemma 3.1 has the same issue: G_{t,n} depends on the belief at t−n, so the matrix is not finite-window either.\n\nMinor but relevant: intervention metrics have no error bars or significance tests, and the probe-inverse patch is evaluated against the same target algorithm's belief used to construct it. The theory sections rely on idealized attention/hard-attention constructions; that's fine for capability results, less so for claims about actual use.\n\nNet reading: the paper successfully shows that the Oracle/Bayesian belief is decodable but not always causally used, and that the small trained transformer picks a finite-window predictor. The pre-trained LLM's actual algorithm is not pinned down by the current Soft n-gram probe. That's an addressable flaw, not a fatal one: make W_t depend only on a bounded window (or at least run a control with fixed weights) and report variance. The paper deserves serious peer review and will likely improve with revision. I'd take it to reading group; I'd cite the PAP pipeline and the small-transformer result, not the finite-window conclusion.","headline":"Good empirical pipeline and a clean small-transformer result, but the headline finite-window claim is not supported by the Soft n-gram probe, which refits weights on the full context.","tokens_in":73494,"tokens_out":2441,"would_cite":true,"duration_ms":28551,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-trained LLMs solving HMM next-token prediction do not run Bayesian inference; they learn finite-window, gradient-descent-like approximations described by Soft n-gram statistics.","keywords":["hidden Markov models","in-context learning","mechanistic interpretability","linear probing","activation patching","n-gram statistics","principal activations probe","transformers"],"falsifier":"Construct source/target histories that share the same last n observations but differ in earlier context, so the Soft n-gram belief W_t z_t changes only through the full-context-refit weight W_t; if patching this belief still shifts the LLM output, the causal effect is coming from long-context weights, not the finite-window feature itself.","tokens_in":55,"feed_emoji":"🔬","tokens_out":3426,"duration_ms":98968,"temperature":0.7,"pith_summary":"The paper tries to pin down the algorithm that pre-trained LLMs actually use when they predict the next observation from a Hidden Markov Model sequence. It narrows candidate algorithms to three classes, proves Transformers can implement all of them in-context, and then introduces a probing-and-intervention method called PAP to test which representation causally drives the model's outputs. The central conclusion is that LLMs do not implement statistically optimal iterative inference; instead they appear to learn finite-window, gradient-descent-like approximations well-described by Soft n-gram statistics, even in regimes where longer-range integration would yield strictly lower cross-entropy loss. A sympathetic reader would care because this distinguishes behavioral mimicry from mechanistic explanation: the LLM can approach Bayes-optimal predictions while using a fundamentally local, approximate algorithm.","feed_headline":"LLMs use n-gram shortcuts, not Bayesian filtering, on HMM tasks","feed_subtitle":"Probing internal activations shows finite-window learning drives next-token predictions even where longer-range tracking would win.","key_machinery":"The Principal Activations Probe (PAP) is the central tool: it projects layer-wise residual activations onto their top principal components, fits a ridge regression from those coordinates to candidate algorithmic representations (Oracle belief, Soft n-gram belief/operator, Spectral belief/operator), and then tests causal use with three escalating patches—full residual, PCA-subspace, and probe-inverse. The underlying theoretical identity is Lemma 3.1: the Bayes-optimal next-token distribution is linear in the Kronecker product of the last n one-hot observations, which unifies the n-gram, nonlinear-feature, and spectral constructions and justifies why a finite-window linear predictor can approx","core_discovery":"On HMM next-token prediction, pre-trained LLM predictions converge toward the Bayes-optimal oracle as context grows, but the internal algorithm responsible is not belief tracking or spectral filtering. Using PAP, the paper shows low-dimensional linear representations in the residual stream causally drive predictions, and that the Soft n-gram representation—a linearly learned predictor over a finite window of recent observations—is the most causally effective algorithmic state across different HMM regimes. The same probing shows that algorithmic information can be decodable without being causally used: early layers may encode beliefs that downstream computation bypasses. In a small Transforme","pith_inferences":["A testable extension of the paper's claim is that corrupting the finite-window features or the Soft n-gram operator should shift LLM predictions predictably, whereas intervening on a full-context belief state should have little effect in belief-dominated regimes.","The result suggests that in natural-language tasks with latent structure, models may rely on local statistical shortcuts even when they appear to perform long-range reasoning; PAP-style causal probing could reveal whether apparent world models are actually causally inert.","One subtlety the paper leaves implicit is that the Soft n-gram operator W_t is refit by ridge regression over the entire observation prefix; if the causal effect flows through W_t's long-context summary rather than the finite-window feature z_t, the 'finite-window' characterization would need qualification."],"forward_implications":["LLMs can match Bayes-optimal HMM predictions while internally implementing a finite-window, n-gram-like algorithm, so behavioral convergence alone does not reveal the learned mechanism.","Decodability and causal use must be tested separately: a representation can be present in activations yet not drive predictions.","The internally recruited representation shifts with HMM regime: richer Soft n-gram tracking is used when belief integration matters, while bigram-like statistics suffice when emissions are nearly deterministic.","Distinct computational stages localize to different layers, with early layers often encoding information that later layers bypass.","A small Transformer trained on a fixed HMM learns a finite-window predictor almost exactly, suggesting this algorithm class is a natural inductive outcome of the architecture."],"fun_headline_variants":["LLMs choose n-gram shortcuts over optimal HMM filters","On HMMs, LLMs run on soft n-gram states, not beliefs","Probing LLMs: HMM logic is a linear finite-window state","LLMs' HMM predictions hinge on simple n-gram vectors"],"cache_read_input_tokens":74752,"weakest_assumption_plain":"The causal conclusion rests on the assumption that patching the probe's readout directions replaces the algorithm the model uses, rather than overwriting its next-token distribution near the output; since the Soft n-gram belief is itself a next-token distribution, the two can be hard to distinguish.","fun_headline_variants_meta":{"raw":{"variants":["LLMs choose n-gram shortcuts over optimal HMM filters","On HMMs, LLMs run on soft n-gram states, not beliefs","Probing LLMs: HMM logic is a linear finite-window state","LLMs' HMM predictions hinge on simple n-gram vectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1282,"prompt_tokens":753,"completion_tokens":529,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":461}},"tokens_in":497,"tokens_out":529,"duration_ms":5970,"temperature":1.0,"reasoning_tokens":461,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:05:55.635003+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct source/target histories that share the same last n observations but differ in earlier context, so the Soft n-gram belief W_t z_t changes only through the full-context-refit weight W_t; if patching this belief still shifts the LLM output, the causal effect is coming from long-context weights, not the finite-window feature itself.","supporting_citations":[],"review_version":1}