Pith. sign in

REVIEW 4 major objections 5 minor 39 references

LLMs can memorize facts but fail to assign scenario roles.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LLMs fine-tuned on paraphrases of fictional facts can recall the paraphrases but cannot answer questions about who did what in those facts, suggesting memorization without robust scenario-level understanding.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A useful small dataset and a plausible negative result, but the probing pillar is unvalidated and the 'do NOT have scenario cognition' conclusion is much stronger than the evidence. the 4 major comments →

arxiv 2509.04866 v1 pith:W34WDGTH submitted 2025-09-05 cs.CL

Memorization $\neq$ Understanding: Do Large Language Models Have the Ability of Scenario Cognition?

Boxiang Ma , Ru Li , Yuanlong Wang , Hongye Tan , Xiaoli Li This is my paper
classification cs.CL
keywords scenario cognitionmemorization vs understandinglarge language modelsprobingsupervised fine-tuningsemantic rolesframe semanticsgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models truly understand the semantic scenarios in the text they process or merely memorize surface forms. To test this, the authors construct a dataset of fictional facts with multiple paraphrased descriptions, fine-tune ten open-source LLMs on these descriptions, and then ask the models to answer questions that rephrase the same facts from a different role perspective. The models perform well on the descriptions they were trained on but near chance on the rephrased questions, and linear probes of their internal representations fail to recover which entities fill which roles. The paper concludes that current LLMs lack 'scenario cognition'—the ability to bind semantic scene elements to their arguments—and that their impressive output performance reflects surface-level memorization rather than deep semantic understanding.

Core claim

After supervised fine-tuning on 5,000 paraphrased descriptions of 500 fictional facts, ten open-source LLMs from the Qwen, LLaMA, and Gemma families score high on the training distribution (Memory Set) but collapse on rephrased scenario questions (Understanding Set), with exact-match scores dropping from roughly 0.76–0.87 to 0.10–0.24. Adding 30% of the Understanding Set to fine-tuning raises Understanding performance only modestly, far below Memory Set levels. From the internal-representation perspective, a linear probe trained to classify scenario element–argument pairs never exceeds 0.5 F1 at any layer, and recall stays especially low, indicating that the bindings are not linearly decodab

What carries the argument

The key mechanism is a bi-perspective evaluation framework built on a scenario-based dataset. The dataset consists of Atomic Knowledge (fictional facts with at least three roles), a Memory Set of ten paraphrased descriptions per fact used for supervised fine-tuning, and an Understanding Set of questions that target scenario elements in a completion format (e.g., 'The director who presented... is ___'). The second perspective uses a linear probe on concatenated hidden-state vectors of scenario elements and candidate arguments, trained as a binary classifier to detect matching element–argument pairs. This framework operationalizes 'scenario cognition' as the ability to link semantic scene elem

Load-bearing premise

The Understanding Set fairly tests the same knowledge as the Memory Set, so the performance gap must be attributed to missing scenario cognition rather than to output format or phrasing differences.

What would settle it

Fine-tune a model on the same Memory Set and evaluate it on Understanding Set questions paraphrased by humans (not by the same generator) in a held-out design. If performance approaches Memory Set levels, the gap is surface variation. Alternatively, if any existing open LLM probed at multiple layers with a small nonlinear probe yields F1 above 0.8 on the released dataset, the claim that role-argument bindings are not internally encoded would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Merely scaling model size or extending fine-tuning epochs is unlikely to yield scenario cognition, since output-level gains with scale do not translate into probe performance or Understanding Set success.
  • Standard supervised fine-tuning on paraphrases produces surface-level 'data memory'; evaluating models only on the training distribution substantially overstates their semantic competence.
  • The format-adaptation experiment shows that the memorization–understanding gap is not an artifact of question phrasing: exposing models to the target output style leaves performance far below Memory Set levels.
  • If scenario cognition is absent from current architectures, tasks requiring role binding in novel contexts—such as multi-step instruction following or complex semantic parsing—will remain unreliable.
  • The paper's account ties at least some hallucination types to missing scenario cognition: models generate fluent text because of formal competence but lack the internal bindings needed to verify semantic correctness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would be to compare human performance on the same Understanding Set under similar time constraints; if humans also drop sharply on rephrased questions, part of the gap may reflect task difficulty rather than a model-specific deficit.
  • The probing result suggests that role-argument bindings may be encoded in non-linear or distributed form, or may be context-dependent; probing with interventions (e.g., causal tracing of role-entity pairs) could reveal whether bindings are ever formed even if not linearly decodable.
  • The scenario-cognition framing could be applied to evaluate multilingual models: if the memorization–understanding gap persists across languages, it would strengthen the claim that the deficit is architectural rather than corpus-driven.
  • One implicit implication is that benchmark gains achieved by training on paraphrases may be conflating memorization with comprehension; the field may need 'understanding sets' as a standard control in fine-tuning evaluations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a bi-perspective framework to evaluate whether LLMs possess "scenario cognition," defined as the ability to link semantic scenario elements (e.g., director, producer) with their arguments (e.g., Paxton, Helen) in context. The authors construct a synthetic dataset of 500 fictional atomic facts, each expanded into 10 paraphrased descriptions (Memory Set, 5,000 samples) and 1,581 scenario-question/answer pairs (Understanding Set). After full-parameter SFT on the Memory Set, they evaluate ten open-source models from Qwen2.5, LLaMA3.x, and Gemma2 families, reporting high memorization scores on the Memory Set but low scores on the Understanding Set. They complement this with linear probing of hidden states to test whether element-argument associations are linearly decodable, and they report F1 scores below 0.5. The paper concludes that current LLMs do NOT have scenario cognition and rely primarily on surface-level memorization. The manuscript includes format-adaptation experiments, a case study, and appendices with alternative probing designs.

Significance. If the central claim were established, the paper would make a useful contribution to the ongoing debate about whether LLM generalization reflects semantic understanding or shallow pattern matching. The dataset is released, the evaluation spans multiple model families and scales, and the bi-perspective design (output-level and representation-level) is a reasonable template for future studies. The format-adaptation experiment in Table 2 is a commendable attempt to address one obvious confound. However, the strong negative conclusion currently rests on an unvalidated probing null result and a remaining output-format confound, so the paper's significance is conditional on additional controls and more carefully scoped claims.

major comments (4)
  1. [§2.2, §3.2.2, Fig. 4] The probing pillar is not validated as evidence of absence. The authors claim that because probe F1 is below 0.5, models have not effectively encoded element-argument associations. But with a 47%/53% positive/negative split, an all-positive classifier achieves F1 ≈ 0.64, so all reported F1 values are below a trivial majority baseline. No positive control is shown: there is no demonstration that the same linear probe can detect element-argument associations in a model or representation known to contain them. Moreover, Appendix B.4 reports that average attention for target pairs is "notably higher" than for non-target pairs, which is direct evidence that some scenario-relevant information is accessible internally. The strong negative conclusion from the probe is therefore not supported. The authors should add (a) a positive control, e.g., a synthetic representation or a model fine-tuned wi
  2. [§3.2.1, Tables 1–2] The Memory vs Understanding gap is confounded with output format. Memory Set evaluation uses the same completion format as SFT (first segment as prompt, second segment as target), while the Understanding Set requires completing a newly phrased template with a short answer. The format-adaptation experiment trains on only 30% of the Understanding Set and evaluates on the remaining 70%; the residual gap may reflect generalization to unseen question templates rather than absence of scenario cognition. To make the central claim load-bearing, the authors should run a stronger format control: for example, train on 100% of the Understanding Set (or all template types) and measure whether the gap disappears; or evaluate Memory Set knowledge using Understanding-style prompts; or use answer-extraction scoring instead of generation metrics. Without such a control, the observed gap is consistent with
  3. [§5, Abstract] The claim that "current LLMs do NOT have the ability of scenario cognition" is overgeneralized. The experiments cover ten open-source models after 5-epoch SFT on 5,000 synthetic sentences. This supports a more limited conclusion: under this training/data configuration, the evaluated models show poor generalization to the constructed Understanding Set. It does not warrant a universal claim about all current LLMs, nor about models with different training protocols (e.g., instruction tuning, larger-scale pretraining, or in-context learning without SFT). The authors should rephrase the conclusion to match the scope of the evidence.
  4. [§2.1.3, §2.1.4] Dataset validity is not quantified. The Understanding Set is generated and annotated by Qwen2.5-Max, and the paper reports manual validation only qualitatively ("manual correction of low-quality annotations," "manual validation to guarantee data quality"). No inter-annotator agreement, error-rate statistics, or number of corrected samples are given. If the scenario questions are unnatural, ambiguous, or contain annotation errors, the performance gap could be an artifact of test design. Reporting human-evaluation statistics and making the annotation guidelines public would substantially strengthen the claim that the dataset measures scenario cognition.
minor comments (5)
  1. [General] The paper switches between "scene cognition" and "scenario cognition" (e.g., Section 3.3 uses "situational cognition" and "scene-level understanding"); unify the terminology.
  2. [§3.1] Results are averaged over five runs but no standard deviations or confidence intervals are reported. Given the temperature=1 sampling, variance may be nontrivial; please include error bars or tables with variance.
  3. [Appendix D, Table 5] Some example questions are grammatically awkward (e.g., "The subject that Captain James Morrison was rescued..."); if these are representative, it strengthens the need for human quality metrics on the Understanding Set.
  4. [§2.1.1] The semantic-similarity threshold d(vx, vy) > 0.5 is described but not justified; a sentence or two on how this threshold was chosen would help.
  5. [Appendix B.4] The MLP-based probes report near-zero performance for some larger models; it would be useful to state whether the probes converged or whether this indicates an optimization failure, since that affects interpretation.

Circularity Check

0 steps flagged

No significant circularity: the central negative result is an empirical measurement, not a consequence of the paper's definitions or fitted parameters.

full rationale

The paper's central claim — that current LLMs lack scenario cognition — is an empirical negative result obtained by fine-tuning on a Memory Set, evaluating on a held-out Understanding Set, and probing internal representations. None of these conclusions is derived by construction from the definition of scenario cognition; rather, the definition operationalizes the construct and the experiments measure it. The 30%-Understanding-Set format-adaptation experiment is a held-out control, not a fitted parameter renamed as a prediction. The linear probe is a supervised classifier whose failure is measured; while one may question its validity (no positive control, a trivial majority baseline of about 0.64 F1), that is a correctness or validity concern, not circularity, because the conclusion does not reduce to the probe's training labels by any equation. Self-citations (e.g., Li et al. 2024 for frame-semantic terminology, Zhao et al. 2025 in related work) are contextual and not load-bearing: no uniqueness theorem, ansatz, or known result is imported from the authors' prior work to force the conclusion. The use of LLMs (Qwen2.5-Max, DeepSeek-V3) to generate the synthetic dataset is an instrumentation choice and does not make the measured failure equivalent to the generation procedure. The paper's own Appendix B.4 attention result partially weakens the strong negative conclusion, but that is an internal-evidence tension rather than a circular step. Overall, the derivation chain is self-contained and empirically grounded, with at most minor, non-load-bearing self-citations, so the circularity score is low.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 1 invented entities

The central claim rests on the construct validity of the synthetic dataset and probing setup rather than on any mathematical derivation. There are no fitted physical constants: the load-bearing assumptions are that fictional facts are uncontaminated, that the Understanding Set fairly tests the same knowledge, and that probe failure implies absent information.

free parameters (2)
  • Semantic similarity retention threshold = 0.5 (L2 distance)
    Chosen in Equation 2, Section 2.1.1 to filter generated Atomic Knowledge; controls dataset diversity and size, and therefore which facts and paraphrases are used for training and evaluation.
  • Number of expanded descriptions per Atomic Knowledge = 10
    Fixed in Section 2.1.2; this count determines the Memory Set size (5,000) and the amount of paraphrase diversity the models see.
axioms (5)
  • domain assumption Frame Semantics' semantic scenes and scenario elements are the right units for testing understanding
    Section 1 grounds the definition of scenario cognition in Fillmore's frame semantics; the entire evaluation presupposes that element-argument association is the relevant ability.
  • domain assumption A linear probe's success or failure reveals whether information is encoded in representations
    Section 2.2 interprets probe accuracy as evidence about internal encoding; low probe scores could reflect probe insensitivity or representation geometry rather than absent information.
  • domain assumption Fictional facts are free of pretraining contamination
    Section 2.1.1 generates fictional facts to prevent reliance on memorized real-world knowledge; this assumes the models have not seen these facts or close variants in pretraining.
  • domain assumption LLM-generated and manually filtered annotations are correct ground truth
    Atomic Knowledge, paraphrases, scenario element annotations, and questions are produced by LLMs with manual validation (Sections 2.1.1 to 2.1.4); errors in these labels would directly affect measured performance.
  • domain assumption The Understanding Set questions are semantically equivalent to the Memory Set descriptions
    Section 2.1.4 assumes that completing 'The director who presented... is ___' tests the same knowledge as the training sentence; if not, the gap is a format mismatch rather than a cognition deficit.
invented entities (1)
  • Scenario cognition no independent evidence
    purpose: The ability being measured: linking scenario elements to their arguments in context
    Defined and operationalized in Sections 1 and 2 through the paper's own dataset and probes; there is no external benchmark or human construct validation establishing that this measurement reflects a stable cognitive faculty.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Memorization $\neq$ Understanding: Do Large Language Models Have the Ability of Scenario Cognition?." pith.science (2026). https://pith.science/paper/W34WDGTH

@misc{pith2026250904866,
  author       = {Pith},
  title        = {Pith review of: Memorization $\neq$ Understanding: Do Large Language Models Have the Ability of Scenario Cognition?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W34WDGTH}},
  note         = {Machine review of arXiv:2509.04866}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Driven by vast and diverse textual data, large language models (LLMs) have demonstrated impressive performance across numerous natural language processing (NLP) tasks. Yet, a critical question persists: does their generalization arise from mere memorization of training data or from deep semantic understanding? To investigate this, we propose a bi-perspective evaluation framework to assess LLMs' scenario cognition - the ability to link semantic scenario elements with their arguments in context. Specifically, we introduce a novel scenario-based dataset comprising diverse textual descriptions of fictional facts, annotated with scenario elements. LLMs are evaluated through their capacity to answer scenario-related questions (model output perspective) and via probing their internal representations for encoded scenario elements-argument associations (internal representation perspective). Our experiments reveal that current LLMs predominantly rely on superficial memorization, failing to achieve robust semantic scenario cognition, even in simple cases. These findings expose critical limitations in LLMs' semantic understanding and offer cognitive insights for advancing their capabilities.

Figures

Figures reproduced from arXiv: 2509.04866 by Boxiang Ma, Hongye Tan, Ru Li, Xiaoli Li, Yuanlong Wang.

Figure 1
Figure 1. Figure 1: Illustration of the contrast between human [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Diagram of the bi-perspective evaluation framework for assessing LLMs’ scenario cognition, depicting [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The trend of each metric as the training epoch increases from the perspective of model outputs, with solid [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of linear probing results for LLMs’ internal representations across training epochs, with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: , most errors do not display obvious gram￾matical or pragmatic flaws, yet they deviate sub￾stantially from factual correctness. Drawing on these observations, we argue that hallucination in LLMs is, at least in part, a reflection of their in￾sufficient situational cognition. Specifically, the model’s “data” memory provides strong compe￾tence in formal linguistic patterns, allowing it to generate grammatica… view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of probing results with SimilarityMLP for LLMs’ internal representations across training epochs, with subfigures for Precision, Recall, and F1 scores at different Transformer layers (Head, Mid, Tail), where the red dashed line at 0.5 indicates the baseline for binary classification of scenario element-argument correspondences [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of probing results with EnhancedSimilarityMLP for LLMs’ internal representations across training epochs, with subfigures for Precision, Recall, and F1 scores at different Transformer layers (Head, Mid, Tail), where the red dashed line at 0.5 indicates the baseline for binary classification of scenario element-argument correspondences [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of probing results using Attention-based Probing for LLMs’ internal representations at the final epoch. Each subfigure shows the Attention scores of a specific LLM at different Transformer layers (Head, Mid, Tail), comparing target pairs (i.e., scenario elements and their corresponding arguments) with non-target pairs (i.e., scenario elements and unrelated tokens). Each bar represents the ave… view at source ↗
Figure 9
Figure 9. Figure 9: Prompt template for Atomic Knowledge Gen [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 21 canonical work pages · 2 internal anchors

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644

    Pith/arXiv arXiv 2016

  4. [4]

    Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. 2024. Generalization vs. memorization: Tracing language models' capabilities back to pretraining data. In ICML 2024 Workshop on Foundation Models in the Wild

    work page 2024

  5. [5]

    a is b” fail to learn “b is a

    Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2024. The reversal curse: Llms trained on “a is b” fail to learn “b is a”

    work page 2024

  6. [6]

    Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120

    2023

  7. [7]

    Idan A Blank. 2023. What are large language models supposed to model? Trends in Cognitive Sciences, 27(11):987--989

    work page 2023

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  9. [9]

    Bowen Chen, Namgi Han, and Yusuke Miyao. 2024 a . A multi-perspective analysis of memorization in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11190--11209

    work page 2024

  10. [10]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024 b . Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216

    Pith/arXiv arXiv 2024

  11. [11]

    Alexis Conneau, Germ \'a n Kruszewski, Guillaume Lample, Lo \" c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126--2136

    work page 2018

  12. [12]

    Jessica Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. 2024. Cognitive bias in decision-making with llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12640--12653

    work page 2024

  13. [13]

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Sch \"u tze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012--1031

    work page 2021

  14. [14]

    Charles J Fillmore. 1967. The case for case

    work page 1967

  15. [15]

    Charles J Fillmore. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences, 280(1):20--32

    work page 1976

  16. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  17. [17]

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, and 1 others. 2023. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296

    Pith/arXiv arXiv 2023

  18. [18]

    Ray S Jackendoff. 1972. Semantic interpretation in generative grammar

    work page 1972

  19. [19]

    Romuald A Janik. 2023. Aspects of human memory and large language models. arXiv preprint arXiv:2311.03839

    Pith/arXiv arXiv 2023

  20. [20]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611--626

    2023

  21. [21]

    Sotiris Lamprinidis. 2023. Llm cognitive judgements differ from human. pages 17--23

    work page 2023

  22. [22]

    Ru Li, Yunxiao Zhao, Zhiqiang Wang, Xuefeng Su, Shaoru Guo, Yong Guan, Xiaoqi Han, and Hongyan Zhao. 2024. A comprehensive overview of cfn from a commonsense perspective. Machine Intelligence Research, 21(2):239--256

    work page 2024

  23. [23]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    Pith/arXiv arXiv 2024

  24. [24]

    Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuan-Jing Huang, and Xipeng Qiu. 2024. Scaling laws for fact memorization of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11263--11282

    work page 2024

  25. [25]

    Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in cognitive sciences

    work page 2024

  26. [26]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359--17372

    2022

  27. [27]

    Qian Niu, Junyu Liu, Ziqian Bi, Pohsun Feng, Benji Peng, Keyu Chen, Ming Li, Lawrence KQ Yan, Yichao Zhang, Caitlyn Heqi Yin, and 1 others. 2024. Large language models and cognitive science: A comprehensive review of similarities, differences, and challenges. arXiv preprint arXiv:2409.02387

    arXiv 2024

  28. [28]

    Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463--2473

    work page 2019

  29. [29]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE

    2020

  30. [30]

    Ali Satvaty, Suzan Verberne, and Fatih Turkmen. 2024. Undesirable memorization in large language models: A survey. arXiv preprint arXiv:2410.02650

    arXiv 2024

  31. [31]

    Xuefeng Su, Ru Li, Xiaoli Li, and Zhichao Yan. 2025. https://doi.org/10.26599/TST.2024.9010036 Efsp-te: End-to-end frame-semantic parsing with table encoder . Tsinghua Science and Technology, 30(4):1474--1495

    work page arXiv 2025

  32. [32]

    Siddharth Suresh, Kushin Mukherjee, Xizheng Yu, Wei-Chun Huang, Lisa Padua, and Timothy T Rogers. 2023. Conceptual structure coheres in human cognition but not in large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  33. [33]

    Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. olmpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743--758

    work page 2020

  34. [34]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    Pith/arXiv arXiv 2024

  35. [35]

    Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399

    Pith/arXiv arXiv 2023

  36. [36]

    Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526--1541

    work page 2023

  37. [37]

    Zhichao Yan, Jiapu Wang, Jiaoyan Chen, Xiaoli Li, Ru Li, and Jeff Z. Pan. 2024. https://arxiv.org/abs/2410.16708 Atomic fact decomposition helps attributed question answering . Preprint, arXiv:2410.16708

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [38]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  39. [39]

    Yunxiao Zhao, Hao Xu, Zhiqiang Wang, Xiaoli Li, Jiye Liang, and Ru Li. 2025. Explaining black-box language models with knowledge probing systems: A post-hoc explanation perspective. arXiv preprint arXiv:2508.16969

    work page internal anchor Pith review Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.