{"id":"6d83a14c-71c8-4101-9819-bd5f2b600738","arxiv_id":"2607.22575","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Long-context LLMs solve temporal order judgments by reinstating a learned one-dimensional position code through a single attention head, matching human distance effects.","lead":"Human-like order memory in large language models uses a single attention head that stamps each part of a text with a hidden time code and re-reads that code when asked what came first. The work links a concrete circuit in LLMs to a leading theory of how the brain remembers order.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal readout fitted on random words is never validated on the intact novel; causal drop may reflect perturbation of a critical head rather than removal of the code.","rationale":"The reader's weakest assumption is indeed the most load-bearing point: the entire mechanistic pipeline—TRS localization, head selection, and the causal readout direction—is anchored to a readout fitted on random iid nouns, while the key behavioral and causal evidence comes from the intact novel. The paper deserves credit for the cross-document generalization (shuffled book, House of Commons) and for the causal effect on the original book, which makes the identity assumption more than pure speculation. However, the causal effect alone cannot distinguish 'the model uses ŵ as its temporal code' from 'the model's critical retrieval head is somewhat sensitive to any strong projection', and the fairly moderate effect size (10-15%) plus incomplete 70B head coverage keep this from being a clean confirmation. Because the concern is about the precision of the causal attribution rather than a demonstrated contradiction, a CONDITIONAL verdict remains appropriate. If the proposed book-specific readout test is added and passes, the identity concern would be resolved and the abstract's 'relies on' would be on firmer ground. The paper is otherwise careful: it includes control directions for amplification, validates with linear probing, tests shuffled texts, and reports cross-model replications, all of which independently support the core phenomenon.","tokens_in":24002,"tokens_out":12824,"duration_ms":148901,"concrete_test":"On a held-out subset of the 296 TMORA pairs, fit a book-specific readout w_book as PC1 of retrieval representations for L8H11/L14H21 and compute cosine(w_book, ŵ). Then apply the Eq. 13 ablation on the remaining pairs using w_book vs. ŵ (and a random direction in the same head as a control). If cosine > 0.9 and the w_book and ŵ ablations produce statistically indistinguishable drops while random-direction ablation does not, the identity assumption survives. If w_book ablation drops accuracy substantially more (e.g., >25% vs. 10-15%), the readout is distribution-dependent and the central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim requires that ŵ ≡ w_train,pc1, fit on a 60k iid random-word sequence (Sec. 5.2), is the same one-dimensional code the model uses when judging order on The Murder of Roger Ackroyd. The paper validates ŵ on sentence-shuffled book variants and a House of Commons transcript, but the intact novel—the only stimulus used for both the human comparison and the causal interventions—is not part of the readout validation set (Table 1). If the model's temporal code is context-dependent or non-linear on naturalistic narrative, the TRS localization and the Eq. 13/14 intervention could be probing a partially-aligned but not identical axis. The NoPE cosine alignment of 0.69 (App. A.1) is moderate and computed on random tokens, not on the book. The significant 10-15% accuracy drop when ablating ŵ on the original book mitigates this, but it does not fully establish identity: no ablation of a randomly chosen direction within the same head is reported, so one cannot rule out that the drop reflects non-specific perturbation of a head that is critical for other reasons. If the book-specific code differs, the 'one-dimensional temporal code' may be an artifact of the training distribution, and the causal conclusion would need re-scoping.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SORT, a temporal order memory task built from 50-word segments of a full-length novel, and collects a new human behavioral dataset on The Murder of Roger Ackroyd. It reports that Llama-3.1-8B and Llama-3.1-70B reproduce the human distance effect, and that this behavior is largely preserved under sentence-block shuffling. Using PCA-based readout directions fit on a 60k random-word document, the authors localize temporal reinstatement to single heads (L8H11 in 8B, L14H21 in 70B) and show that projecting out the readout direction at retrieval spans reduces accuracy by 10-15%, while amplifying it can improve accuracy. The paper concludes that temporal order memory in these models relies on a one-dimensional temporal code reinstated by a single attention head.","tokens_in":24350,"tokens_out":7925,"duration_ms":79833,"significance":"If established, the result is significant: it provides a mechanistically transparent model of temporal context reinstatement in a long-context setting and connects transformer interpretability to episodic-memory theory. The paper also contributes a new human dataset and a toolkit. The behavioral distance effect is well supported by permutation tests with FDR correction, and the representational analyses include cross-document generalization and a NoPE control. However, the causal-specificity claim is not yet fully established: the readout direction is not validated on the intact novel, the projection ablation lacks a within-head random-direction control, and the 70B causal sweep covers only a fraction of heads. These gaps are fixable with additional analyses or careful re-scoping.","major_comments":[{"comment":"The readout direction ŵ is fit on a 60k i.i.d. random-word sequence and validated only on shuffled TMORA variants and a House of Commons transcript; the intact novel, which is the sole stimulus for the human comparison and for the causal interventions in Eqs. (13)-(14), is absent from the validation set (Table 1). If the model uses a context-dependent or nonlinear temporal code on naturalistic narrative, the TRS localization and the later intervention could probe a partially aligned but non-identical axis. The NoPE cosine alignment of 0.69 (App. A.1) is computed on random token sequences, not on the book, and does not close this gap. Please report per-head temporal alignment on the intact novel itself and, ideally, run the Eq. (13) intervention with a book-specific readout direction as a control.","section":"Section 5.2 / Table 1"},{"comment":"The projection-out intervention removes only the component along ŵ during retrieval spans, but no control ablation of a random direction within the same head is reported. The random-direction control in Fig. 7 is attached to the scaling manipulation (Eq. 14), not to the projection. Without such a control, a 10-15% drop could reflect nonspecific disruption of a causally important head rather than removal of the temporal code. To support the one-dimensional-code claim, ablate random unit vectors in the output subspace of L8H11/L14H21 and show that accuracy is unaffected.","section":"Section 5.3, Eq. (13)"},{"comment":"The claim that a single head is causally necessary is asserted for both models, but for 70B only 180 of 5120 heads are ablated (top-20 TRS + 160 random). A head outside this subset could be equally important, and the 'single head' conclusion is therefore not supported for the larger model at the stated level of confidence. Either provide a full-head causal sweep or explicitly scope the conclusion to the tested subset and adjust the abstract accordingly.","section":"Section 5.3 / Figure 6"},{"comment":"The heading 'reduces average performance to near-chance' is contradicted by the immediately following sentence reporting a 10-15% drop. If baseline accuracy in the far-distance bins is roughly 0.8-0.9, post-ablation accuracy remains well above chance; 'near-chance' and the abstract's 'relies on' overstate the causal contribution. Please report baseline and ablated accuracy by distance bin and align the wording with the measured effect size.","section":"Section 5.3, 'Directional removal'"}],"minor_comments":[{"comment":"The number of unique segment pairs is given as 296 in Table 1, 540 in §A.10, and 990 samples in §3.2. Reconcile these counts and clarify how the 296-pair set relates to the full human dataset.","section":"Table 1 vs. §A.10"},{"comment":"The cross-model ablation is described as 'removing the top reinstatement head's contribution to the residual stream,' while §5.3 removes only the PC1 component. Specify whether the cross-model experiment uses the same projection intervention or a stronger head-removal.","section":"§A.8"},{"comment":"The four-segment experiment uses The Picture of Dorian Gray but does not state how the readout direction was fitted for this text. State whether the random-word PC1 was used and whether prompts match §A.11.","section":"§A.9"},{"comment":"Color bars have different scales and maxima across panels; add a brief caption note so readers do not visually overinterpret cross-panel differences.","section":"Figures 3-4 and 10-11"}],"recommendation":"major_revision","confidential_remarks":"The central mechanism claim is plausible and well-triangulated, but the causal-specificity evidence needs one additional control (random-direction ablation within the same head) and validation of the readout direction on the intact novel; both are feasible. The 70B head coverage should be disclosed upfront. I would not reject: the paper's main contribution, localized temporal reinstatement with cross-document generalization, is likely to stand with these additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful mechanistic paper, and the reader's conditional verdict is about right. The headline result — a single attention head (L8H11 in 8B, L14H21 in 70B) stores and reinstates a low-dimensional temporal code, and ablating that code's direction hurts order judgment accuracy — is new and well-supported enough to take seriously. The behavioral distance effect matching humans on a full novel is a nice contribution on its own. The paper also does the right controls: readout fit on random words, validated on shuffled book variants and a parliamentary transcript, a RoPE-free analysis showing the code isn't positional-embedding trivia, and the shuffling result ruling out narrative coherence as the main driver.\n\nWhere it's soft: the abstract's \"relies on\" is stronger than the evidence. The causal effect is a 10-15% accuracy drop, not to chance. That's real but leaves room for other mechanisms. The stress-test worry about the readout direction is legitimate but partial: the direction is never validated on the intact novel, only on shuffled versions of it and other documents, and the NoPE cosine alignment of 0.69 is moderate. Still, the fact that ablating w on the intact novel drops accuracy, while ablating other heads' temporal directions does not, does most of the work. A random-direction ablation within the same head would have sealed it. The 70B causal sweep covers only 180 of 5120 heads; that's a coverage gap, not a flaw in what was tested. The human data isn't public yet, and the code/toolkit release is a plus.\n\nI think the stress test overstates the risk of \"artifact of training distribution\" — the cross-head specificity plus generalization to shuffled book text makes the axis clearly relevant. The weaker claim (a single head causally contributes by reinstating a temporal code) holds. The stronger version (this is the mechanism) needs the gap filled.\n\nWho it's for: people in mechanistic interpretability of memory, and cognitive scientists treating LLMs as model organisms. It deserves a serious referee. I'd send it to review, asking for the intact-novel readout validation (or at least explicit discussion), a random-direction control within the same head, and softer abstract language.","headline":"Solid mechanistic work with a real finding; the \"relies on\" claim runs ahead of a 10-15% ablation effect, but the core result survives its soft spots.","tokens_in":24818,"tokens_out":2266,"would_cite":true,"duration_ms":24110,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-context language models answer 'which came first?' by reinstating a one-dimensional time code stored in a single attention head, and causal tests show this code is necessary for their order memory.","keywords":["temporal order memory","episodic memory","long-context language models","mechanistic interpretability","attention heads","temporal context reinstatement","distance effect","causal intervention"],"falsifier":"Compute the PC1 direction of L8H11's retrieval representations directly on The Murder of Roger Ackroyd and measure its cosine similarity to ŵ (fit on random words). If that alignment is low (e.g., below about 0.5), the readout used in the interventions does not describe the code actually active during the book task, and the causal conclusion is unsupported. Alternatively, run the full-head ablation sweep on all 5120 heads of Llama-3.1-70B: if several heads show accuracy drops comparable to L14H21's, the 'single head' claim fails.","tokens_in":23930,"feed_emoji":"🧠","tokens_out":4728,"duration_ms":48917,"temperature":0.7,"pith_summary":"The paper asks whether large language models recall temporal order the way humans do, and if so, what internal mechanism carries that ability. Using a new dataset in which 97 people judged which of two passages from a full novel came first, the authors show that Llama-3.1 models reproduce the human 'distance effect': accuracy rises with the number of words separating the passages. Mechanistic analysis then traces the behavior to a single attention head that stores a one-dimensional temporal code while reading and reinstates that code when the model is asked to compare passages. Causal interventions confirm the link: removing that head's time axis drops accuracy by 10-15%, while amplifying it can improve accuracy by up to 9%. The authors conclude that temporal context reinstatement—retrieval by replaying an encoded position signal—is a real, causally important mechanism in these models, and a plausible computational principle for episodic memory more broadly.","feed_headline":"Single attention head stores the time code behind LLM order memory","feed_subtitle":"Removing that head's code drops 'which came first?' accuracy 10-15%; amplifying it can push accuracy up by 9%.","key_machinery":"The load-bearing object is the temporal reinstatement score (TRS), which is computed from head-specific value-cache and retrieval representations using a readout direction ŵ fit as the first principal component of a 60,000-word random-word sequence. A head counts as a time-reinstatement head only if the same ŵ orders both its encoding-phase and retrieval-phase segment representations (Eq. 10). The causal tests then intervene on the top-TRS head's retrieval output at the query segments, either projecting out the component along ŵ or scaling it by a factor α (Eqs. 13-14), leaving the rest of the forward pass untouched.","core_discovery":"The central discovery is that in both Llama-3.1-8B and Llama-3.1-70B, temporal order judgments are carried by a single attention head: L8H11 in the 8B model, L14H21 in the 70B model. During encoding, that head's value-cache is organized along one principal direction ŵ, which orders segments by their position in the text; at retrieval, the head re-emits this code onto the query segments. The authors define a temporal reinstatement score (TRS) that requires the same readout direction to order both encoding-time and retrieval-time segment representations, and find one outlier head per model. Projecting ŵ out of the head's retrieval output reduces SORT accuracy by 10-15%, while scaling it up (α=","pith_inferences":["If the temporal code is as content-invariant as the cross-document generalization suggests, then the same readout direction could be used as a probe in other long-context systems; a system that orders segments correctly after its top-TRS head is ablated would be announcing that it uses a different, probably semantic, strategy.","The 70B causal test covered only 180 of 5120 attention heads; a full-head sweep might reveal that the larger model relies on several heads, not one, which would weaken the 'single head' story and sharpen when the localization claim holds.","A direct test of the shared-code assumption would compare ŵ to the PC1 direction computed from the original novel's own retrieval representations; if those align poorly, the causal intervention is probing the wrong axis for the actual book task.","The behavioral distance-effect match between humans and models suggests a usable behavioral fingerprint: future long-context systems could be screened for reinstatement-like mechanisms by checking whether their order accuracy still rises with distance on shuffled text."],"forward_implications":["Temporal order judgments in these models do not primarily depend on narrative or causal reasoning: performance is preserved under sentence-block shuffling, so an explicit time trace plus reinstatement is sufficient.","The mechanism is highly localized, not distributed: per model, one attention head dominates both the TRS ranking and the causal ablation effect, with the 8B model showing that no other head's PC1 direction significantly matters.","The temporal code is learned and independent of explicit positional encoding: it persists and even strengthens when RoPE is removed, and the NoPE readout direction aligns with ŵ at cosine 0.69.","The mechanism is not specific to one model family or task format: the paper reports preliminary single-head reinstatement effects in Mistral-7B and Qwen2.5-7B, and an extension from binary to four-segment ordering in Llama-3.1-8B.","Human and model distance-effect curves align on the same naturalistic book, supporting long-context LLMs as a mechanistically accessible model system for episodic-memory hypotheses."],"fun_headline_variants":["One head encodes the time code for LLM order memory","Single attention head reinstates time for order memory in LLMs","LLM episodic order memory tied to one temporal reinstatement head","A lone head drives LLM time-order recall via context reinstatement","LLM order memory: one head stores and replays the temporal code"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The readout direction ŵ is fit once on a 60,000-word random sequence, and the authors assume this same axis is the one the models use when making order judgments on the naturalistic novel; if the book task engages a different, context-dependent code, the localization and the causal intervention would be probing an irrelevant direction.","fun_headline_variants_meta":{"raw":{"variants":["One head encodes the time code for LLM order memory","Single attention head reinstates time for order memory in LLMs","LLM episodic order memory tied to one temporal reinstatement head","A lone head drives LLM time-order recall via context reinstatement","LLM order memory: one head stores and replays the temporal code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1233,"prompt_tokens":747,"completion_tokens":486,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":398}},"tokens_in":491,"tokens_out":486,"duration_ms":4957,"temperature":1.0,"reasoning_tokens":398,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:10:25.255754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the PC1 direction of L8H11's retrieval representations directly on The Murder of Roger Ackroyd and measure its cosine similarity to ŵ (fit on random words). If that alignment is low (e.g., below about 0.5), the readout used in the interventions does not describe the code actually active during the book task, and the causal conclusion is unsupported. Alternatively, run the full-head ablation sweep on all 5120 heads of Llama-3.1-70B: if several heads show accuracy drops comparable to L14H21's, the 'single head' claim fails.","supporting_citations":[],"review_version":1}