{"id":"8b518340-a09c-4725-a191-174a27df80c6","arxiv_id":"2505.13844","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Simulated associative memory, added as augmented text or taught through instruction tuning, appears to improve how well language model activations predict fMRI brain responses in story listening.","lead":"The authors test whether giving language models extra 'associative memory' text, either by appending related words or by fine-tuning the model to generate such text, makes their internal activations match human brain activity more closely when people listen to stories. The measured gains are small and the paper lacks statistical tests for part of its evidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SFT result is confounded by training on the same Narratives stories used for brain-score evaluation, so the reported 2-7% MTL gain may reflect in-domain exposure rather than associative memory.","rationale":"I read the paper's central claim as having two pillars: augmentation with simulated associative memory and instruction tuning on associative content. The reader's weakest assumption (temporal alignment in Sec 3.3) is a legitimate modeling concern, but my stronger and more concrete concern is that the instruction-tuning pillar has a train/evaluation overlap: Appendix A.3 says Association is composed of sentences from filtered Narratives stories, and Section 4.1 shows the same filtered stories are the evaluation stimuli. This makes the SFT improvement ambiguous: the model may simply have been fine-tuned on the test stories. The temporal-alignment issue can be probed by shifting augmentations in time, but it does not explain why the second pillar's gain should be attributed to association rather than memorization. The proposed control (re-train on ROCStories only) would settle whether associative instruction tuning transfers to unseen story text. For the paper as written, the SFT claim is not supported without that control, so I would move the verdict to REJECT pending the check; the augmentation half could remain as a conditional finding.","tokens_in":14296,"tokens_out":8174,"duration_ms":81313,"concrete_test":"Rebuild Association using only ROCStories, or hold out a disjoint set of Narratives stories for evaluation, so that no sentence from the evaluation stimuli appears in SFT; retrain LLaMA-2 under the same LoRA and frozen-layer settings and recompute the instruction tuning scores in MTL. If the 2-7% gain disappears or becomes statistically indistinguishable from the random/no-association control, the SFT claim is explained by data leakage.","verdict_should_be":"REJECT","load_bearing_attack":"Section 4.1 selects 15 Narratives stories for all reported experiments. Appendix A.3 states that the Association SFT dataset is composed of sentences from the filtered Narratives stories plus ROCStories sentences. LLaMA-2 is therefore fine-tuned on the very story text used later to compute brain scores. The 2-7% medial temporal lobe gain in Section 5.3 could come from the model being adapted to the evaluation stories, not from learning to generate associative content; the comparison with base LLaMA-2 does not control for exposure to the stimulus text. This does not invalidate the augmentation result in Section 5.2, but it directly undermines the paper's second central claim and the corresponding contribution. The limitations section acknowledges annotator noise but never mentions this train/evaluation overlap, and no code is provided to check the actual splits.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether adding associative memory content to text stimuli improves the linear encoding-model alignment between language models (GPT-2 and LLaMA-2) and fMRI responses from the Narratives dataset. It first computes baseline brain scores by mapping per-layer LM activations to fMRI frames with a finite impulse response model and ridge regression. It then augments the text with human- or GPT-4-generated word- or sentence-level associations, defining the associative memory score as the difference between augmented and original brain scores. Finally, it builds the Association instruction-tuning dataset and fine-tunes LLaMA-2 with LoRA or frozen-layer training, reporting instruction tuning scores. The paper's two central claims are that simulated associative memory improves LM-brain alignment and that instructing LLMs to generate associative content through SFT also improves alignment.","tokens_in":14464,"tokens_out":6361,"duration_ms":60687,"significance":"If the reported effects were statistically robust and the training/evaluation confound were removed, the paper would make a useful contribution to the growing LM-brain alignment literature: it proposes a concrete augmentation mechanism for a cognitive process that is comparatively underexplored, releases a new instruction-tuning dataset, and includes a random-augmentation control. The paper also uses held-out fMRI data for the encoding model and provides detailed implementation information. However, the two central claims are currently under-supported: the associative memory score is reported without inferential statistics, the SFT result appears to train on the same Narratives stories used for evaluation, and the augmented content is placed in the same fMRI time bins under an untested simultaneity assumption. The significance of the findings is therefore conditional on addressing these load-bearing issues.","major_comments":[{"comment":"The instruction-tuning result is confounded by a training/evaluation overlap. Appendix A.3 states that the Association dataset is composed of sentences from the filtered Narratives stories, and §4.1 states that after filtering short articles, 15 Narratives stories are used for the reported experiments. As written, the same story text used to compute the brain scores in §5.3 appears in the SFT training data, so the 2–7% MTL gain in Figure 6 could reflect in-domain exposure to the evaluation stimuli rather than learning to generate associative content. The comparison against base LLaMA-2 does not control for this exposure. Please re-evaluate on Narratives stories (or other naturalistic stimuli) that are held out from Association, or explicitly document a split and demonstrate non-overlap; the manuscript currently provides no split information and no code to verify it.","section":"§5.3, Appendix A.3, §4.1"},{"comment":"The associative memory score is the paper's first central result, but it is reported without error bars, confidence intervals, significance tests, or multiple-comparison correction. The raw differences are 0.0014–0.02 in Figure 4 and 0–0.05 in Figure 5, against baseline brain scores around 0.126 (Figure 3), so the gains are small relative to the score scale. The qualitative statement that random augmentation leads to 'none and even negative growth' is not a substitute for a statistical comparison. Please provide subject-level or story-level bootstrap/permutation intervals for the difference scores, and account for the number of ROIs and layers examined.","section":"§5.2, Figures 4 and 5, Eq. (10)"},{"comment":"The temporal alignment of the augmented content is a load-bearing modeling choice. All expanded content is placed at the end of the triggering sentence and assigned the last word's offset, 'as if the associative memory forms simultaneously when subject receives specific text stimuli.' This means the augmented tokens are pooled into the same fMRI frames as the trigger words. If associative retrieval in the brain is delayed or distributed over a longer time window, the reported improvement could be an artifact of injecting extra context into the same time bin. No behavioral or neural evidence is offered for this simultaneity, and no robustness analysis is given. I request an exploration of alternative alignments, for example shifting the augmented content by one or more TRs relative to the trigger or extending the FIR window, to show the gain is not an artifact of the alignment.","section":"§3.3, Eqs. (9)–(10)"}],"minor_comments":[{"comment":"The text says 'RidgeClassifierCV' regressor from scikit-learn is used to predict continuous features, but RidgeClassifierCV is a classification estimator; the ridge regression in Eq. (6) and the continuous fMRI targets imply the regressor variant (e.g., RidgeCV) is intended. Please correct this inconsistency.","section":"Appendix A.2"},{"comment":"There is a typo in the sentence-level human augmentation example: 'creatin' should be 'creating'.","section":"Table 1"},{"comment":"The caption says the dot color corresponds to the specific ROI, but the figure as printed does not provide a legend mapping colors to the nine ROIs. A legend would make the figure interpretable.","section":"Figure 5"},{"comment":"The instruction tuning scores in Figure 6 are reported as percentages without error bars; the subject-level confidence intervals in Figure 7 are informative and should also be used for the whole-brain percentage scores in Figure 6.","section":"§5.3, Figure 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. The first result—adding simulated associative memory to story stimuli and showing a brain-score bump over random augmentation—is genuinely new and worth engaging with. The random-augmentation control is exactly the right design. The second result, however, is not cleanly interpretable as stated because the fine-tuning data overlap the evaluation stories. That needs to be fixed before I would trust the 2-7% MTL gain as evidence for associative memory.\\n\\nWhat is new: they try human and GPT-4 annotations, compare word-level versus sentence-level augmentation, show word-level helps more, and release Association, a 1000-sample instruction-tuning dataset. The brain-score ceiling test is a nice addition. The writing is clear and the related work is adequate. Credit where it is due.\\n\\nSoft spots, in rough order of seriousness. First, the SFT claim is confounded. Section 4.1 selects 15 Narratives stories for evaluation, and Appendix A.3 says Association is built from sentences from those filtered stories plus ROCStories. Fine-tuning LLaMA-2 on the very texts you later score creates an exposure confound. The comparison to base LLaMA-2 does not control for in-domain exposure, and the limitations section never mentions it. This does not kill the augmentation result, but it undermines the paper's second central claim.\\n\\nSecond, the associative memory score figures (4 and 5) report no error bars or significance tests over subjects, and the raw gains (0.0014 to 0.05) are small relative to baseline scores around 0.1. The random-augmentation baseline helps, but without subject-level variance estimates, I cannot tell whether the effect is robust. Third, the temporal alignment assumption in Section 3.3—putting all expanded content at the end of the triggering sentence with onsets set to the last word's offset—is a modeling choice. If associative retrieval is delayed or spread over time, the augmented input might simply be leaking extra context into the same fMRI time bin. That deserves a sensitivity check. Fourth, the layer for each model is selected from the same data used for the reported scores; that is selection on the test set and likely inflates the numbers.\\n\\nBottom line: the paper is a solid draft with a good control and a useful dataset, but the second claim is confounded and the first is under-supported statistically. It deserves a serious referee—the questions are worth asking—but it needs major revision. I would probably bring it to reading group as a case study in confounds more than as a result I would build on.","headline":"Augmentation result is promising but statistically thin; the SFT result is confounded by train/eval story overlap and needs a control before the 2-7% claim is credible.","tokens_in":14962,"tokens_out":2316,"would_cite":false,"duration_ms":21281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that appending simulated associative memory to story stimuli improves how well language model activations predict fMRI brain responses, and that fine-tuning a large language model on associative content adds further gains…","keywords":["associative memory","brain alignment","fMRI encoding models","language models","instruction tuning","medial temporal lobe","data augmentation","LLaMA-2"],"falsifier":"Shift the onsets of the appended association content by one, two, or three fMRI TRs after the triggering word's offset and recompute the associative memory score; if the gain vanishes or becomes negative, the reported improvement depends on the simultaneous-onset assumption. A second check: replace the associated words with equally frequent but content-bearing unrelated words; if the gain persists, the effect is not specific to association.","tokens_in":14075,"feed_emoji":"🧠","tokens_out":7455,"duration_ms":65109,"temperature":0.7,"pith_summary":"The paper asks whether language models align better with human brain activity when they are given access to associative memory, the process by which listeners link new speech to related knowledge. It answers yes in two ways: appending human- or GPT-4-written associated words or sentences to the story text raises the correlation between model activations and fMRI signals in regions tied to associative memory, working memory, and speech processing; and supervised fine-tuning of LLaMA-2 on a new 1,000-sample dataset, where prompts ask the model to output associated content, raises brain scores in the medial temporal lobe by 2–7%. The work matters because it offers a concrete, testable mechanism for improving brain-language alignment, and because random augmentation does not reproduce the gain, the effect is linked to associative content rather than simply longer input.","feed_headline":"Associative memory boosts language-model brain alignment","feed_subtitle":"Adding related words to story stimuli, or training a model to produce them, improves fMRI prediction in memory regions.","key_machinery":"The machinery is a linear encoding framework: ridge regression maps mean-pooled activations from a chosen Transformer layer to fMRI frames with a finite impulse response model, and the Pearson correlation between predicted and actual fMRI is the brain score. The associative memory score is the difference between the brain score with augmented text and the brain score on original text, and the instruction tuning score is the percentage gain after fine-tuning. The load-bearing maneuver is to place augmented association content at the end of the sentence that triggered it, with the same onset time as the last word, so the association signal lands in the same fMRI frames as the trigger. Low-rank adaptation (LoRA) and frozen-layer fine-tuning are the two training schemes used to adapt LLaMA-2 on the Association dataset.","core_discovery":"The central discovery is that aligning language models to the brain can be improved by injecting associative content into the model's input, without changing the brain data. For the first research question, each story sentence that triggers association is augmented with associative words or sentences; the language model is then run on the augmented sequence, and its activations are mean-pooled into fMRI frames through a finite-impulse-response model and ridge regression. The resulting correlation, the brain score, rises in nine regions of interest including inferior and middle temporal regions, frontal and parietal regions, and the medial temporal lobe, with word-level human-annotated augmentation giving the strongest and most consistent gains and random augmentation giving none. For the second question, LLaMA-2 fine-tuned on the Association dataset using LoRA or frozen-layer training shows a 2% to 7% gain in medial temporal lobe score over the base model, and a 50% to 60% gain in the superior parietal lobule at the subject level.","pith_inferences":["The authors leave the timing mechanism implicit: the paper collapses associative retrieval into the trigger word's fMRI frame; a natural extension is to let association content enter several TRs later or with a learned temporal kernel and test whether alignment improves further.","Because word-level augmentation outperforms sentence-level, the inference is that content-word associations, not grammatical expansions, carry the brain-alignment signal; this could be tested by comparing augmentation with high-frequency synonyms versus low-frequency associated words.","The paper does not fully separate associative memory from general semantic relatedness; one could test specificity by using related-but-nonassociative paraphrases or word-association norms, a distinction that would change the interpretation.","The authors note the limitation that their data and annotators share a specific language and cultural context, so a direct extension is to test whether the same gains appear in other languages and participant populations."],"forward_implications":["Word-level human-annotated association is the strongest augmentation for improving brain scores; sentence-level and GPT-4-generated content also help but less consistently, so future datasets should favor multi-aspect lexical associations.","Supervised fine-tuning on the Association dataset transfers to better brain prediction in memory-related regions: the medial temporal lobe gains 2–7% and the superior parietal lobule gains 50–60% at the subject level.","Random word and sentence augmentation does not improve and can hurt brain scores, implying the gains are not just from longer input.","For LLaMA-2 the best-fitting layer is shallower (14 of 32) than for GPT-2 (9 of 12), so the linear mapping should be evaluated per model rather than at a fixed layer."],"supporting_citations":[{"why":"Supplies the fMRI dataset of hundreds of subjects listening to natural stories, the central data for all brain score computations.","marker":"Nastase et al. (2021)"},{"why":"Establishes the FIR temporal alignment and ridge-regression mapping from word embeddings to fMRI that the paper adapts.","marker":"Huth et al. (2016)"},{"why":"Provides the baseline framework for linearly mapping language model activations to brain activity and justifies using GPT-2.","marker":"Caucheteux and King (2020)"},{"why":"Shows that predictive, future-oriented processing improves brain alignment, a parallel effect this paper extends to associative content.","marker":"Caucheteux et al. (2023)"},{"why":"Supplies the baseline result that ordinary instruction tuning does not improve brain scores, making the associative SFT gains meaningful.","marker":"Gao et al. (2023)"},{"why":"Provides the LoRA method used for supervised fine-tuning of LLaMA-2.","marker":"Hu et al. (2021)"},{"why":"Contributes story sentences used to build the Association instruction-tuning dataset.","marker":"Mostafazadeh et al. (2016)"}],"fun_headline_variants":["Associative memory improves LM-brain alignment","Add related words to boost brain alignment","Fine-tuning on associations aids fMRI prediction","Injecting associative content sharpens brain scores","Memory-driven input boosts LM-brain fit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is temporal: augmented association content is treated as if it enters the brain at the same instant as the last word of the sentence that triggered it, so the model's added tokens are pooled into those same fMRI frames.","fun_headline_variants_meta":{"raw":{"variants":["Associative memory improves LM-brain alignment","Add related words to boost brain alignment","Fine-tuning on associations aids fMRI prediction","Injecting associative content sharpens brain scores","Memory-driven input boosts LM-brain fit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2758,"prompt_tokens":855,"completion_tokens":1903,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1841}},"tokens_in":471,"tokens_out":1903,"duration_ms":14447,"temperature":1.0,"reasoning_tokens":1841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:08:54.030297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shift the onsets of the appended association content by one, two, or three fMRI TRs after the triggering word's offset and recompute the associative memory score; if the gain vanishes or becomes negative, the reported improvement depends on the simultaneous-onset assumption. A second check: replace the associated words with equally frequent but content-bearing unrelated words; if the gain persists, the effect is not specific to association.","supporting_citations":[{"cited_title":"narratives","cited_arxiv_id":null,"evidence_quote":"Supplies the fMRI dataset of hundreds of subjects listening to natural stories, the central data for all brain score computations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the baseline framework for linearly mapping language model activations to brain activity and justifies using GPT-2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that predictive, future-oriented processing improves brain alignment, a parallel effect this paper extends to associative content."},{"cited_title":"Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention","cited_arxiv_id":"2310.19084","evidence_quote":"Supplies the baseline result that ordinary instruction tuning does not improve brain scores, making the associative SFT gains meaningful."}],"review_version":1}