Pith. sign in

REVIEW 4 major objections 4 minor 16 references

Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Pith's one-line read Long-context language models answer 'which came first?' by reinstating a one-dimensional time code stored in a single attention head, and causal tests show this code is necessary for their order memory.

desk verdict Solid mechanistic work with a real finding; the "relies on" claim runs ahead of a 10-15% ablation effect, but the core result survives its soft spots. read the letter →

arxiv 2607.22575 v1 pith:7V5NN427 submitted 2026-06-05 cs.AI

classification cs.AI
keywords temporalordermemoryepisodiclong-contextlanguagemodelsmechanisticinterpretabilityattentionheadscontextreinstatementdistanceeffectcausalintervention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models recall temporal order the way humans do, and if so, what internal mechanism carries that ability. Using a new dataset in which 97 people judged which of two passages from a full novel came first, the authors show that Llama-3.1 models reproduce the human 'distance effect': accuracy rises with the number of words separating the passages. Mechanistic analysis then traces the behavior to a single attention head that stores a one-dimensional temporal code while reading and reinstates that code when the model is asked to compare passages. Causal interventions confirm the link: removing that head's time axis drops accuracy by 10-15%, while amplifying it can improve accuracy by up to 9%. The authors conclude that temporal context reinstatement—retrieval by replaying an encoded position signal—is a real, causally important mechanism in these models, and a plausible computational principle for episodic memory more broadly.

What carries the argument

The load-bearing object is the temporal reinstatement score (TRS), which is computed from head-specific value-cache and retrieval representations using a readout direction ŵ fit as the first principal component of a 60,000-word random-word sequence. A head counts as a time-reinstatement head only if the same ŵ orders both its encoding-phase and retrieval-phase segment representations (Eq. 10). The causal tests then intervene on the top-TRS head's retrieval output at the query segments, either projecting out the component along ŵ or scaling it by a factor α (Eqs. 13-14), leaving the rest of the forward pass untouched.

What would settle it

Compute the PC1 direction of L8H11's retrieval representations directly on The Murder of Roger Ackroyd and measure its cosine similarity to ŵ (fit on random words). If that alignment is low (e.g., below about 0.5), the readout used in the interventions does not describe the code actually active during the book task, and the causal conclusion is unsupported. Alternatively, run the full-head ablation sweep on all 5120 heads of Llama-3.1-70B: if several heads show accuracy drops comparable to L14H21's, the 'single head' claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that in both Llama-3.1-8B and Llama-3.1-70B, temporal order judgments are carried by a single attention head: L8H11 in the 8B model, L14H21 in the 70B model. During encoding, that head's value-cache is organized along one principal direction ŵ, which orders segments by their position in the text; at retrieval, the head re-emits this code onto the query segments. The authors define a temporal reinstatement score (TRS) that requires the same readout direction to order both encoding-time and retrieval-time segment representations, and find one outlier head per model. Projecting ŵ out of the head's retrieval output reduces SORT accuracy by 10-15%, while scaling it up (α=

Load-bearing premise

The readout direction ŵ is fit once on a 60,000-word random sequence, and the authors assume this same axis is the one the models use when making order judgments on the naturalistic novel; if the book task engages a different, context-dependent code, the localization and the causal intervention would be probing an irrelevant direction.

Editorial extensions

If this is right

  • Temporal order judgments in these models do not primarily depend on narrative or causal reasoning: performance is preserved under sentence-block shuffling, so an explicit time trace plus reinstatement is sufficient.
  • The mechanism is highly localized, not distributed: per model, one attention head dominates both the TRS ranking and the causal ablation effect, with the 8B model showing that no other head's PC1 direction significantly matters.
  • The temporal code is learned and independent of explicit positional encoding: it persists and even strengthens when RoPE is removed, and the NoPE readout direction aligns with ŵ at cosine 0.69.
  • The mechanism is not specific to one model family or task format: the paper reports preliminary single-head reinstatement effects in Mistral-7B and Qwen2.5-7B, and an extension from binary to four-segment ordering in Llama-3.1-8B.
  • Human and model distance-effect curves align on the same naturalistic book, supporting long-context LLMs as a mechanistically accessible model system for episodic-memory hypotheses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the temporal code is as content-invariant as the cross-document generalization suggests, then the same readout direction could be used as a probe in other long-context systems; a system that orders segments correctly after its top-TRS head is ablated would be announcing that it uses a different, probably semantic, strategy.
  • The 70B causal test covered only 180 of 5120 attention heads; a full-head sweep might reveal that the larger model relies on several heads, not one, which would weaken the 'single head' story and sharpen when the localization claim holds.
  • A direct test of the shared-code assumption would compare ŵ to the PC1 direction computed from the original novel's own retrieval representations; if those align poorly, the causal intervention is probing the wrong axis for the actual book task.
  • The behavioral distance-effect match between humans and models suggests a usable behavioral fingerprint: future long-context systems could be screened for reinstatement-like mechanisms by checking whether their order accuracy still rises with distance on shuffled text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces SORT, a temporal order memory task built from 50-word segments of a full-length novel, and collects a new human behavioral dataset on The Murder of Roger Ackroyd. It reports that Llama-3.1-8B and Llama-3.1-70B reproduce the human distance effect, and that this behavior is largely preserved under sentence-block shuffling. Using PCA-based readout directions fit on a 60k random-word document, the authors localize temporal reinstatement to single heads (L8H11 in 8B, L14H21 in 70B) and show that projecting out the readout direction at retrieval spans reduces accuracy by 10-15%, while amplifying it can improve accuracy. The paper concludes that temporal order memory in these models relies on a one-dimensional temporal code reinstated by a single attention head.

Significance. If established, the result is significant: it provides a mechanistically transparent model of temporal context reinstatement in a long-context setting and connects transformer interpretability to episodic-memory theory. The paper also contributes a new human dataset and a toolkit. The behavioral distance effect is well supported by permutation tests with FDR correction, and the representational analyses include cross-document generalization and a NoPE control. However, the causal-specificity claim is not yet fully established: the readout direction is not validated on the intact novel, the projection ablation lacks a within-head random-direction control, and the 70B causal sweep covers only a fraction of heads. These gaps are fixable with additional analyses or careful re-scoping.

major comments (4)
  1. [Section 5.2 / Table 1] The readout direction ŵ is fit on a 60k i.i.d. random-word sequence and validated only on shuffled TMORA variants and a House of Commons transcript; the intact novel, which is the sole stimulus for the human comparison and for the causal interventions in Eqs. (13)-(14), is absent from the validation set (Table 1). If the model uses a context-dependent or nonlinear temporal code on naturalistic narrative, the TRS localization and the later intervention could probe a partially aligned but non-identical axis. The NoPE cosine alignment of 0.69 (App. A.1) is computed on random token sequences, not on the book, and does not close this gap. Please report per-head temporal alignment on the intact novel itself and, ideally, run the Eq. (13) intervention with a book-specific readout direction as a control.
  2. [Section 5.3, Eq. (13)] The projection-out intervention removes only the component along ŵ during retrieval spans, but no control ablation of a random direction within the same head is reported. The random-direction control in Fig. 7 is attached to the scaling manipulation (Eq. 14), not to the projection. Without such a control, a 10-15% drop could reflect nonspecific disruption of a causally important head rather than removal of the temporal code. To support the one-dimensional-code claim, ablate random unit vectors in the output subspace of L8H11/L14H21 and show that accuracy is unaffected.
  3. [Section 5.3 / Figure 6] The claim that a single head is causally necessary is asserted for both models, but for 70B only 180 of 5120 heads are ablated (top-20 TRS + 160 random). A head outside this subset could be equally important, and the 'single head' conclusion is therefore not supported for the larger model at the stated level of confidence. Either provide a full-head causal sweep or explicitly scope the conclusion to the tested subset and adjust the abstract accordingly.
  4. [Section 5.3, 'Directional removal'] The heading 'reduces average performance to near-chance' is contradicted by the immediately following sentence reporting a 10-15% drop. If baseline accuracy in the far-distance bins is roughly 0.8-0.9, post-ablation accuracy remains well above chance; 'near-chance' and the abstract's 'relies on' overstate the causal contribution. Please report baseline and ablated accuracy by distance bin and align the wording with the measured effect size.
minor comments (4)
  1. [Table 1 vs. §A.10] The number of unique segment pairs is given as 296 in Table 1, 540 in §A.10, and 990 samples in §3.2. Reconcile these counts and clarify how the 296-pair set relates to the full human dataset.
  2. [§A.8] The cross-model ablation is described as 'removing the top reinstatement head's contribution to the residual stream,' while §5.3 removes only the PC1 component. Specify whether the cross-model experiment uses the same projection intervention or a stronger head-removal.
  3. [§A.9] The four-segment experiment uses The Picture of Dorian Gray but does not state how the readout direction was fitted for this text. State whether the random-word PC1 was used and whether prompts match §A.11.
  4. [Figures 3-4 and 10-11] Color bars have different scales and maxima across panels; add a brief caption note so readers do not visually overinterpret cross-panel differences.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: readout fitted on random-word sequences, evaluated on disjoint documents, and causally probed on the intact novel.

full rationale

The paper's central chain is: (1) fit head-specific PC1 temporal readouts w on a 60k random-word sequence (Sec. 5.2, App. A.5); (2) evaluate those fixed readouts on disjoint documents (shuffled TMORA, House of Commons) via Eq. 9; (3) form TRS in Eq. 10 from held-out encoding/retrieval rank correlations; (4) localize L8H11/L14H21; and (5) causally test the fitted direction on the intact novel via Eqs. 13-14. At no point is the target quantity (task accuracy on TMORA, or the distance effect) used to fit the readout; the causal test is performed on a stimulus that was not used for readout fitting or head selection. The behavioral distance effect (Sec. 4) is measured independently of the representational analysis. The appendix's NoPE comparison uses a separately estimated direction and reports a moderate cosine similarity (0.69), which is a check, not a fit. Self-citations (Toneva/Vo et al. in related work) are background context and are not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The limitation that the intact novel is not in the readout-validation set (Table 1) is a generalization/construct-validity caveat, not a circular step: the causal effect on the novel could in principle refute the readout's relevance. Thus the derivation is self-contained and non-circular.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No free parameters are used for the behavioral comparison itself. The readout directions are fit on the random-word training set (a fitted axis, not a theory parameter), and the best amplification factor is selected post hoc. The main axiomatic load is the linear representation hypothesis, segment averaging, and the shuffling control assumption.

free parameters (2)
  • per-head temporal readout directions w_enc, w_ret = PC1 directions fit on 60k random-word sequence (vector values not enumerated)
    Fit via PCA on the random-word training set; the central localization and intervention analyses depend on these directions.
  • amplification factor α = 1.75 (post-hoc best)
    Chosen post hoc after scanning α∈[0.25,2.0]; the reported 9% accuracy improvement is at the best value, though significance is BH-corrected across the scan.
assumptions (5)
  • domain assumption Temporal order information is encoded as a linear direction in attention head representations (linear representation hypothesis).
    Invoked in Section 5.2 'Following the linear representation hypothesis (Park et al., 2024)' to justify PCA readouts.
  • domain assumption Segment-averaged value/output vectors preserve the temporal code; token-level averaging does not destroy the order signal.
    Equations 3-4 define segment averages; the entire head-localization pipeline relies on these averages retaining temporal structure.
  • ad hoc to paper Four-sentence block shuffling disrupts the narrative/causal structure relevant to order judgments while preserving local lexical content.
    Section 4 uses shuffled variants to rule out semantic reconstruction; if local within-block content still carries usable order information, the conclusion that narrative is not used is weakened.
  • standard math The model's expected probability comparison over A/B label assignments is an appropriate measure of its order judgment.
    Section 3.4 defines the response rule; standard for LLM evaluation.
  • domain assumption Human participants' judgments on 50-word segments reflect episodic memory for the book's temporal order.
    Section 3.2 assumes the SORT task probes episodic memory; validated only by plot comprehension checks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models." pith.science (2026). https://pith.science/paper/7V5NN427

@misc{pith2026260722575,
  author       = {Pith},
  title        = {Pith review of: Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7V5NN427}},
  note         = {Machine review of arXiv:2607.22575}
}
read the original abstract

Human episodic memory supports the retrieval of experiences that unfold over extended timescales, yet the computational mechanisms underlying this ability remain debated due to the limited mechanistic accessibility in long-term memory experiments in humans. Long-context LLMs may offer promising ways to reveal plausible computational mechanisms that drive this type of retrieval. Here, we investigate whether and how LLMs capture the core behavioral signatures of episodic memory via a temporal order memory task. Using a new dataset of human behavior based on memory of a full-length novel, we show that models exhibit the same characteristic distance effect observed in humans on this task. We next apply long-context mechanistic interpretability analyses to uncover how models solve this task, and find that model performance relies on a one-dimensional temporal code that is reinstated during retrieval by a single time-reinstatement attention head. These findings support temporal context reinstatement as an important mechanism for episodic-like temporal-order memory in LLMs, offering new insights into how temporal aspects of long-term episodic memory may be instantiated in both artificial and biological systems.

Figures

Figures reproduced from arXiv: 2607.22575 by the authors.

Figure 1
Figure 1. Task schematic. For each trial, two 50-word segments are sampled from a full-length text with known order and distance in words. At test time they are presented as A/B in random order, and the subject/model answers which segment appeared earlier. Document Purpose # Pairs Document length TMORA behavior causal interventions 296 93k 60k random words fit readout 600 100k House of Commons (transcript, truncated) test rea… view at source ↗
Figure 2
Figure 2. Temporal order judgment accuracy vs. distance on The Murder of Roger Ackroyd (original vs. sentence-block-shuffled). Distance-binned accuracy (mean ± 95% CI). Left: Humans, Llama 3.1-8B-Instruct, and Llama 3.1-70B-Instruct evaluated on the same 296 segment pairs from the original book. Right: Llama 3.1-8B-Instruct and Llama 3.1-70B-Instruct evaluated on four-sentence-block shuffled variants of The Murder of Roger Ac… view at source ↗
Figure 3
Figure 3. Encoding-phase temporal information in the value cache of each attention head. We use readout-directions fitted to the training-set to evaluate the rank-correlation τ of positional information between the random word training set and the five held-out test datasets. Data are averaged across all test data. Left: Llama 3.1-8B-Instruct, Right: Llama 3.1-70B-Instruct. Head in￾dices are shown in query-head coordinates (w… view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: Distribution of temporal reinstatement scores across all attention heads in Llama 3.1-8B-Instruct and Llama 3.1-70B￾Instruct respectively. See [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Reduction in accuracy after projecting out attention heads’ principal temporal direction, showing (in order) the heads with the 20 highest reductions in accuracy. The top corresponds to the 8b model (all heads), bottom shows the 70b model (top 20 TRS heads and 160 rand…
Figure 7
Figure 7. Figure 7: Effect of increasing/decreasing gain on PC1 in the top time-reinstatement head (solid line) compared with randomly cho￾sen directions. Shaded areas correspond to bootstrapped 95% confidence intervals. Asterisks show statistical significance of p < 0.05 in paired one-si…
Figure 8
Figure 8. Figure 8: Llama 3.1-8B-Instruct with RoPE enabled (left) and RoPE disabled (right), evaluated on 10 random token sequences of 94k tokens each. Shown is the temporal order information (Kendall’s τ ) along PC1 trained on the first random sequence and evaluated on the remaining 9 s…
Figure 9
Figure 9. Figure 9: Same setup as in [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 11
Figure 11. Figure 11: Retrieval-phase temporal information.Equivalent figure to 4 (temporal information in retrieval-phase representations) but instead of PC1, we use ridge-regression linear probing of segment position. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Distribution of temporal reinstatement scores across the two model, but instead of PC1, we use ridge-regression linear probing of segment position. These results suggest that PC1 can be used in place of a ridge regression with a high regularization strength to find th…
Figure 13
Figure 13. Figure 13: Llama 3.1-8B-Instruct: Generalization of temporal order readout directions in the value cache (encoding-phase) from random-word sequence fitted PC1 to that direction readout across the five test documents. 0.0 0.2 0.4 0.6 0.8 1.0 Train τret, PC1 0.0 0.2 0.4 0.6 0.8 1.…
Figure 14
Figure 14. Figure 14: Llama 3.1-8B-Instruct: Generalization of temporal order readout directions in the retrieval-phase representations from random-word sequence fitted PC1 to that direction readout across the five test documents. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Llama 3.1-70B-Instruct: Generalization of temporal order readout directions in the value cache (encoding-phase) from random-word sequence fitted PC1 to that direction readout across the five test documents. 0.0 0.2 0.4 0.6 0.8 1.0 Train τret, PC1 0.0 0.2 0.4 0.6 0.8 1…
Figure 16
Figure 16. Figure 16: Llama 3.1-70B-Instruct: Generalization of temporal order readout directions in the retrieval-phase representations from random-word sequence fitted PC1 to that direction readout across the five test documents. A.5. Dataset details The Murder of Roger Ackroyd (original…
Figure 17
Figure 17. Figure 17: Retrieval-phase attention mass fraction falling within the span of the segments, averaged over test datasets. Confirming that L8H11 in Llama 3.1-8B-Instruct and L14H21 in Llama 3.1- 70B-Instruct are among the top retrieval heads. 16 [PITH_FULL_IMAGE:figures/full_fig_…
Figure 18
Figure 18. Figure 18: Distribution of temporal reinstatement scores across the two model. A.8. Cross-model temporal reinstatement and ablation To test whether single-head temporal reinstatement is spe￾cific to the Llama-3 family, we repeat the localization and parts of the causal analyses …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 1 canonical work pages

  1. [1]

    I have finished the book The Murder of Roger Ackroyd [Options: True/False]

  2. [2]

    On what date did you finish the book? [Calendar ques- tion type]

  3. [3]

    Did you read or listen to the book? [Options: Read/Listen]

  4. [4]

    Was this your first time reading / listening to the book? [Options: Yes / No]

  5. [5]

    What is your age? [Options: 18-25, 25-35, 35-45, 45-55, 55-65, 65+]

  6. [6]

    ISBN 979-8-89176-256-5

    Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl

  7. [7]

    What is your experience with the English language? [Options: Native / Fluent / Advanced / Intermediate / Beginner]

  8. [8]

    training

    URL https://openreview.net/forum? id=pXlmOmlHJZ. Park, K., Choe, Y . J., and Veitch, V . The linear represen- tation hypothesis and the geometry of large language models. InForty-first International Conference on Ma- chine Learning, 2024. URL https://openreview. net/forum?id=UGpGkLzwpP. Polyn, S. M., Norman, K. A., and Kahana, M. J. A con- text maintenanc...

Show all 16 references
  1. [14]

    What gender do you identify with? [Options: Fe- male/Male/Other]

  2. [16]

    How many books did you read or listen to in the past year? [Options: 1-2 / 3-5 / 6-10 / 10+] We use the responses above to determine the number of days that have passed since finishing the book, and make this information available in the human dataset together with the respons...

  3. [30]

    findings-acl.30/

    URL https://aclanthology.org/2025. findings-acl.30/. Ji-An, L., Zhou, C. Y ., Benna, M. K., and Mattar, M. G. Linking in-context learning in transformers to human episodic memory. In Globerson, A., Mackey, L., Bel- grave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (ed...

  4. [99]

    findings-emnlp.99/

    URL https://aclanthology.org/2022. findings-emnlp.99/. Howard, M. W. and Kahana, M. J. A Distributed Repre- sentation of Temporal Context.Journal of Mathematical Psychology, 46(3):269–299, June 2002. ISSN 0022-2496. doi: 10.1006/jmps.2001.1388. Irie, K. Why are positional enco...

  5. [102]

    acl-short.102/

    URL https://aclanthology.org/2023. acl-short.102/. Christensen, W. F. and Zabriskie, B. N. When Your Per- mutation Test is Doomed to Fail.The American Statisti- cian, 76(1):53–63, January 2022. ISSN 0003-1305. doi: 10.1080/00031305.2021.1902856. Dai, Q., Heinzerling, B., and I...

  6. [967]

    emnlp-main.967/

    URL https://aclanthology.org/2024. emnlp-main.967/. Fogel, F., d’Aspremont, A., and V ojnovic, M. Spectral ranking using seriation.Journal of Machine Learning Research, 17(88):1–45, 2016. URL http://jmlr. org/papers/v17/16-035.html. Fountas, Z., Benfeghoul, M., Oomerjee, A., C...

  7. [1995]

    doi: 10.1111/j.2517-6161.1995

    ISSN 1467-9868. doi: 10.1111/j.2517-6161.1995. tb02031.x. URL http://dx.doi.org/10.1111/ j.2517-6161.1995.tb02031.x. Benjamini, Y . and Yekutieli, D. The control of the false dis- 9 Temporal Context Reinstatement Drives Episodic-Like Order Memory in Long-Context Language Model...

  8. [2025]

    doi: 10.1016/j.neuron.2025

    ISSN 0896-6273. doi: 10.1016/j.neuron.2025. 02.029. URL http://dx.doi.org/10.1016/j. neuron.2025.02.029. Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., et al. Shared computational principles for lan- ...

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.