Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

The paper argues that locating knowledge in language-model training data is best done by entity identifier, not by text string, and builds a full suite — an entity-annotated Wikipedia corpus, a QID-based retrieval index, and 12 models with

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LMEnt is an open suite of entity-annotated Wikipedia data, 12 pretrained LMs, and an entity-QID retrieval index that substantially outperforms string-based search and enables per-step knowledge attribution.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A carefully engineered entity-annotated corpus and 12 models are a real contribution, but the 80.4% retrieval headline rests on approximated baselines and should be softened. the 2 major comments →

arxiv 2509.03405 v1 pith:O34I3AHS submitted 2025-09-03 cs.CL

LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations

Daniela Gottesman , Alon Gilae-Dotan , Ido Cohen , Yoav Gur-Arieh , Marius Mosbach , Ori Yoran , Mor Geva This is my paper
classification cs.CL
keywords entity linkingcoreference resolutionWikidata QIDspretraining data retrievalknowledge acquisitionfact frequencylanguage model knowledgecheckpoint analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LMEnt is a suite of three linked resources built to answer one question: exactly where and when does a language model encounter a given piece of knowledge during pretraining? The paper's central claim is that this question is best answered by entity, not by text: every chunk of a Wikipedia-based training corpus is annotated with the Wikidata identifiers of the entities it mentions, so retrieval becomes a lookup by entity ID instead of a string match. On a stratified sample of 1,000 entities, this entity-based retrieval wins against case-sensitive and case-insensitive string search on 66.3–80.4% of entities, and holds precision above 97% even when thousands of chunks are retrieved per entity, where string methods collapse to as low as 27%. The same transparency supports 12 pretrained models with checkpoints every 1,000 steps, letting the authors watch facts be learned and forgotten across training, with the finding that fact frequency is key but does not fully explain learning trends. If the suite holds together, it gives researchers a controlled environment for studying knowledge acquisition, editing, attribution, plasticity, and learning dynamics in language models.

Core claim

On its own terms, the paper establishes that entity-grounded bookkeeping of pretraining data changes what can be said about knowledge in language models. It builds a corpus of 10.5M chunks annotated with 400M mentions of 7.3M entities, where each mention carries up to three confidence scores (from hyperlinks, an entity linker, and coreference resolution) pointing to a Wikidata QID, and it shows that retrieving chunks by QID outperforms the string-matching approach of prior tools on 66.3–80.4% of a stratified sample of 1,000 entities, with precision at depth staying above 97% where string methods drop to 84% (case-sensitive) and 27% (case-insensitive). It further shows that models trained on

What carries the argument

The load-bearing object is the entity-mention annotation: every token span in every chunk is mapped to a ranked list of candidate Wikidata QIDs, each backed by up to three scores — H from Wikipedia hyperlinks (binary and most trusted), EL from the ReFinED entity linker (model confidence), and C/CC from Maverick coreference clusters, where a longest-common-substring similarity weights which entity a cluster supports. Retrieval then queries an Elasticsearch index by QID with score thresholds (H=1, EL≥0.6, C≥0.6), and each chunk carries its training-step ID, so every retrieved chunk maps to the exact step at which a model saw it. The three-source design is what lets an indirect mention like 'th

Load-bearing premise

The headline win-rate claim rests on the paper's own simplified stand-ins for the string-matching tools WIMBD and Infini-gram, which are never actually run; if those stand-ins overstate how noisy the real tools are, the 66–80% advantage is not established.

What would settle it

Run the actual WIMBD and Infini-gram code on the same 1,000-entity stratified test set with the same LLM judge and sampling protocol, and compare per-entity win rates and precision-at-depth curves; if the genuine tools match or beat LMEnt on most entities, the central retrieval claim fails. A cheaper check: have human annotators re-judge a random sample of 200 chunks scored by the LLM judge, to verify the reported above-97% precision is not an artifact of the judge.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any question of the form 'did the model see entity X, and at which step?' becomes answerable for 7.3M entities and 10.5M chunks, at training-step granularity.
  • Entity-based retrieval holds precision above 97% even when up to 100K chunks are retrieved per entity, while case-insensitive string search falls to 27%; deeper retrieval is only trustworthy for the entity-based index.
  • Models trained on a 3.6B-token Wikipedia-only corpus reach knowledge-recall accuracy on popular PopQA entities comparable to Pythia-1.4B and OLMo-1B, indicating that knowledge-dense data can substitute for a large share of scale.
  • Fact frequency correlates with net knowledge gain, but both learning and forgetting rates increase with frequency, so frequency alone does not explain when knowledge is acquired; the released checkpoints put that question within reach.
  • The entity-to-step mapping enables controlled causal interventions in pretraining data — reordering, upweighting, or editing mentions — whose effects on downstream knowledge can be measured directly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test that the Subject+Answer co-occurrence count, shown to track accuracy better than pageview popularity, is itself a training signal; reordering or upweighting chunks where subject and answer co-occur could become a cheap fact-recall intervention, testable directly on the released corpus.
  • The finding that forgetting as well as learning rises with fact frequency suggests spaced-repetition-style deduplication of repeated entity mentions might reduce net forgetting; the released 1,000-step checkpoints make this measurable without any new training runs.
  • Because every chunk maps to the exact training step that introduced it, the suite doubles as a contamination audit for knowledge benchmarks: one could check whether a PopQA answer is 'learned' only after the model saw the answer chunk, an application the paper does not discuss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces LMEnt, a suite for studying knowledge acquisition in LMs. It comprises (1) a Wikipedia-based pretraining corpus annotated with fine-grained entity mentions derived from hyperlinks, entity linking, and coreference resolution; (2) an Elasticsearch index for retrieving chunks by Wikidata QID; and (3) twelve pretrained OLMo-2-style models (170M, 600M, 1B) with checkpoints. The main empirical claims are that entity-based retrieval outperforms string-based search methods similar to WIMBD and Infini-gram on 66.3%–80.4% of entities at high precision, that the models achieve knowledge-recall performance comparable to larger open models, and that a learning-dynamics analysis shows fact frequency correlates with learning and forgetting. The authors release the corpus, annotations, models, and code.

Significance. If the retrieval and resource claims hold, LMEnt would be a valuable community asset for studying knowledge acquisition, editing, attribution, and interpretability. The engineering is careful in several respects: the annotation pipeline includes an error analysis on 112 sampled mentions, the LLM-as-a-judge is validated with the alternative annotator test, and retrieval thresholds are tuned on a dev set before evaluation. The release of pretraining data with entity-level traceability and 4K checkpoints is a substantive contribution. However, the headline quantitative claim about outperforming previous retrieval tools is not yet established because the comparison class consists of hand-rolled approximations, and the win-rate protocol measures a fixed-sample precision contest rather than total relevant-chunk counts. The resource itself is promising, but the central retrieval claim needs to be either re-run against the actual tools or properly qualified.

major comments (2)
  1. [§5.2, Fig. 5, Abstract] The strongest quantitative claim, 'outperforms previous approaches by as much as 80.4%,' is not established by the experiments as presented. Section 5.2 states that the CS-SS Canonical baseline 'approximates' Infini-gram and CI-SS Canonical 'approximates' WIMBD, but the actual tools are never run. The approximations are not faithful in ways that matter: Infini-gram is an arbitrary n-gram matcher, so a query for 'Buffalo Bills' also surfaces unigrams such as 'Buffalo'—exactly the noise source motivating LMEnt—whereas CS-SS Canonical only matches the full canonical name. Similarly, CI-SS Canonical is a single exact-string Elasticsearch query, not WIMBD's own index and query semantics. No evidence is provided that these stand-ins reproduce the retrieval sets of the real tools. The abstract and conclusion should either compare against the actual WIMBD/Infini-gram implementations or be reword
  2. [§5.2, Fig. 5, Fig. 7] The win-rate protocol does not measure what the prose claims. For each method and entity, the authors 'randomly sample 100 chunks from the set retrieved per method and entity' and then measure the absolute number of chunks judged as relevant. When both methods retrieve more than 100 chunks, this is a comparison of precision on a fixed 100-chunk sample, not a comparison of the total number of relevant chunks retrieved. A method that retrieves 1,000 relevant chunks at 90% precision can appear worse than a method that retrieves 100 relevant chunks at 100% precision. The claim that 'LMEnt returns more relevant document chunks' for 80% of entities is therefore not directly supported. The precision@k curves in Fig. 7 mitigate this only for the top-k ranking, not for the total-relevant-chunk claim. The protocol should be revised to compare total relevant counts (e.g., by estimating recall via s
minor comments (5)
  1. [§4, Table 2] The mention-count statistics are internally inconsistent. Section 4 says the 400M mentions 'are composed of' 115M hyperlink mentions, 203M entity-linking mentions, and 310M coreference mentions, which sum to 628M. Table 2 also lists 151M coreference mentions and 310M coreference cluster mentions. Please clarify whether these categories overlap and what the unique mention count is.
  2. [Abstract, §2] The corpus is described as 'fully annotated with entity mentions,' but the annotation pipeline has known error rates (2.7% errors in a 112-mention sample) and the retrieval thresholds (H=1, EL≥0.6, C≥0.6) intentionally exclude low-confidence mentions. Consider using 'fine-grained' or 'densely annotated' to avoid overstating coverage.
  3. [§B.3] The error analysis evaluates precision on retrieved mentions, but does not measure recall of the annotation pipeline. Since the suite's value depends on locating mentions, a statement about annotation recall (or a discussion of why recall is hard to measure) would strengthen the resource description.
  4. [§6] The term 'fact' is used for a (Subject, Answer) entity-pair co-occurrence, not for a relation-specific statement. This is acknowledged in the text, but the finding 'fact frequency is key' in the abstract and conclusion should be phrased as 'entity co-occurrence frequency' to avoid implying that the relation itself is measured.
  5. [§5.2, Baselines paragraph] The sentence 'we show in §5.2 that shorter references ... introduce noise' is self-referential because it appears within §5.2. Presumably it refers to experiments described later in the section; please rephrase.

Circularity Check

0 steps flagged

No circularity found: the retrieval and knowledge-acquisition claims are empirical comparisons against external benchmarks and an external LLM judge; the only self-citation is motivational and not load-bearing.

full rationale

LMEnt is primarily a resource paper: its central claims are that (1) entity-based chunk retrieval outperforms string-based retrieval, (2) the released models reach competitive knowledge-recall on PopQA/PAQ, and (3) co-occurrence frequency correlates with, but does not fully explain, knowledge acquisition. None of these claims is derived by construction from its own inputs. The retrieval win-rates are computed by comparing LMEnt's retrieved chunks against chunks retrieved by string-matching baselines, with relevance judged by an external LLM (Gemini 2.5 Flash) whose agreement with human annotators is separately tested. The retrieval thresholds were tuned on a 60-entity dev set and then evaluated on a disjoint 1K-entity test set, so the headline 80.4% win rate is not a re-statement of the tuning criterion. The learning-curve analysis correlates independently measured PopQA accuracy with corpus co-occurrence counts; although both quantities come from the same Wikipedia-derived corpus, the accuracy values are not algebraically or definitionally determined by the co-occurrence counts. The single self-citation (Gottesman & Geva, 2024) appears in the introduction as motivation for focusing on entity names and is not used to justify any quantitative result. The paper's main weakness is external-validity, not circularity: the WIMBD and Infini-gram comparisons use hand-rolled approximations ('CS-SS Canonical baseline approximates the Infinigram tool', 'CI-SS Canonical baseline approximates WIMBD'), so the 'outperforms previous approaches' claim depends on whether those approximations are faithful. That is a legitimate correctness risk, but it is not a case of a prediction being equivalent to a fitted input or of a self-citation chain forcing the conclusion. Consequently, no circular step is present and the score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No new theoretical entities (particles, forces, dimensions) are postulated; LMEnt introduces artifacts such as a corpus, an index, and models, but these are not new physical or formal objects.

free parameters (2)
  • Retrieval score thresholds (H=1, EL>=0.6, C>=0.6) = H=1; EL=0.6; C=0.6
    Chosen empirically on a dev set of 60 entities to maximize retrieval precision (Section 5.2, A.5); affects all retrieval win-rate and precision numbers.
  • Model hyperparameters = batch size 32,768; rank batch 8,192; peak LR 5e-3; weight decay 0.05; 1,000 warmup steps
    Selected via hyperparameter search to minimize final training perplexity (Section B.4); central to the models' knowledge performance but not to a derivation.
axioms (5)
  • domain assumption Wikipedia is a knowledge base structured around entities, providing a consistent snapshot and a good proxy for factual pretraining data.
    Motivates the entire resource; invoked in Section 2 without independent evidence that Wikipedia-only pretraining is representative of larger corpora.
  • domain assumption Off-the-shelf ReFinED and Maverick annotations are accurate enough at scale to support entity-level tracing.
    The annotation pipeline relies on these tools; only 112 mentions were manually audited (Section B.3), so corpus-wide accuracy is assumed.
  • ad hoc to paper The hand-rolled string-matching baselines faithfully approximate WIMBD and Infini-gram.
    The retrieval comparison substitutes approximations for the actual tools (Section 5.2); this assumption is load-bearing for the headline win rate.
  • domain assumption Entity names, rather than individual tokens, are central to knowledge representations.
    Assumed in Section 1 based on prior work; motivates annotating entities, but is not required for the resource to be useful.
  • ad hoc to paper Co-occurrence of subject and answer entities in a chunk defines fact exposure.
    In Section 6, facts are defined as (Subject, Answer) entity pairs; relations in text are not parsed, so this simplification underlies the learning/forgetting analysis.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations." pith.science (2026). https://pith.science/paper/O34I3AHS

@misc{pith2026250903405,
  author       = {Pith},
  title        = {Pith review of: LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O34I3AHS}},
  note         = {Machine review of arXiv:2509.03405}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Language models (LMs) increasingly drive real-world applications that require world knowledge. However, the internal processes through which models turn data into representations of knowledge and beliefs about the world, are poorly understood. Insights into these processes could pave the way for developing LMs with knowledge representations that are more consistent, robust, and complete. To facilitate studying these questions, we present LMEnt, a suite for analyzing knowledge acquisition in LMs during pretraining. LMEnt introduces: (1) a knowledge-rich pretraining corpus, fully annotated with entity mentions, based on Wikipedia, (2) an entity-based retrieval method over pretraining data that outperforms previous approaches by as much as 80.4%, and (3) 12 pretrained models with up to 1B parameters and 4K intermediate checkpoints, with comparable performance to popular open-sourced models on knowledge benchmarks. Together, these resources provide a controlled environment for analyzing connections between entity mentions in pretraining and downstream performance, and the effects of causal interventions in pretraining data. We show the utility of LMEnt by studying knowledge acquisition across checkpoints, finding that fact frequency is key, but does not fully explain learning trends. We release LMEnt to support studies of knowledge in LMs, including knowledge representations, plasticity, editing, attribution, and learning dynamics.

Figures

Figures reproduced from arXiv: 2509.03405 by Alon Gilae-Dotan, Daniela Gottesman, Ido Cohen, Marius Mosbach, Mor Geva, Ori Yoran, Yoav Gur-Arieh.

Figure 1
Figure 1. Figure 1: The LMEnt suite is composed of three components (left) fine-grained entity mentions for every document in the pretraining corpus, (middle) an index that retrieves by the entity QID and outperforms string-based retrieval methods, (right) 12 models trained on 1, 2, 4, and 6 epochs where each step can be mapped to the entities it mentions, and each entity can be traced to the steps that introduce it. fine-gra… view at source ↗
Figure 2
Figure 2. Figure 2: The document for the entity “Josh Allen” is split into two chunks, which are processed independently during pretraining. In the first chunk, the “Buffalo Bills” (Q221626) is mentioned explicitly—identified through hyperlinks and entity linking—and implicitly, through coreference resolution. Although both Q221626 (Buffalo Bills) and Q40435 (the city of Buffalo) share the surface form “Buffalo”, LMEnt disamb… view at source ↗
Figure 4
Figure 4. Figure 4: Mean accuracy on PopQA questions binned according to the number of chunks that the subject and answer entities co-occur in. Increasing model size helps learning associations between entities that appear more frequently in the same chunk. OLMo-1-20K-84B—an intermediate checkpoint of OLMo-1B trained for 20K steps on 84B unique tokens—because it is the most comparable base￾line to LMEnt models in terms of tra… view at source ↗
Figure 5
Figure 5. Figure 5: Pairwise wins rates for LMEnt with multiple string-based methods and ablated LMEnt variations. LMEnt outperforms string-based methods by 66.7%– 80.4%. Ablations (bottom three rows) show that hy￾perlinks and entity linking are the most crucial compo￾nents of LMEnt. judge are provided in §B.5. Since some entities are mentioned very frequently ( [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of the number of chunks retrieved by LMEnt, and both Canonical and Expanded variants of CI-SS and CS-SS. Retrieval was performed for 1K PopQA entities, selected via stratified sampling based on hyperlink counts. LMEnt retrieves more chunks than the Canonical variants for torso and tail entities, which together account for 99.7% of all entities in Wikipedia. The higher num￾ber of chunks returne… view at source ↗
Figure 8
Figure 8. Figure 8: Accuracy for LMEnt-1B-6E model on PopQA. Subj+Answer Chunks counts chunks that mention both the subject and answer entities of a question. Answer Chunks and Subject Chunks count chunks that mention the answer and subject entities in￾dividually. Subj Pop and Answer Pop are pageview popularities from Mallen et al. (2023). Subj+Answer Chunks correlates best with model behavior. (Mallen et al., 2023). In [PIT… view at source ↗
Figure 10
Figure 10. Figure 10: Accuracy on PopQA as a function of compute budget: (left) all entities and (right) questions for which [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Accuracy on PAQ as a function of compute budget: (left) all entities and (right) questions for which the [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Accuracy of OLMo-2-1B (OLMo et al., 2025), Pythia-1.4B (Biderman et al., 2023), and SmolLM-2- 1.7B (Allal et al., 2025) on PopQA, sliced by various indicators of popularity and fact frequency on PopQA. entity, we retrieve their scores using a fixed hyperlink threshold of H = 1, while varying EL, C, and CC across the set {0.4, 0.5, 0.6, 0.7, 0.8}, and evaluate precision over chunks retrieved at different d… view at source ↗
Figure 13
Figure 13. Figure 13: Pairwise win rates between CS-SS Canonical and other string-based methods (left). Pairwise win rates between CI-SS Canonical and CI-SS Expanded (right) [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: shows “Yes” chunk count differences between methods and the cumulative percentage of entities where one method outperforms the other, across entities of different frequencies. For tail entities (right), LMEnt outperforms by a few chunks, but this is meaningful given the scarcity of mentions. For torso entities middle), LMEnt wins by a substantial margin (≥ 20 additional correct chunks) in 40% of cases. Fo… view at source ↗
Figure 15
Figure 15. Figure 15: Percentage of net gains in facts learned between intermediate checkpoints of [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Error analysis of LMEnt entity mentions. B.4 Pretrained Models Details of the LMEnt models are described here and support the overview in §4. The 170M, 600M, and 1B models use (layers, hidden dimension) configurations of (10, 768), (16, 1344), and (16, 2048), respectively. A hyperparameter search was conducted over the following ranges: global batch size [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Prompt given to Gemini to automatically judge whether a chunk mentions an entity directly. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Instructions given to annotators for evaluating LLM-as-a-judge. [PITH_FULL_IMAGE:figures/full_fig_p023_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Prompt given to Gemini 2.5 Flash Preview 6-17 to convert PAQ questions to cloze-style prompts. [PITH_FULL_IMAGE:figures/full_fig_p023_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LLM generation novelty through the lens of semantic similarity

    cs.LG 2025-10 conditional novelty 6.0

    An output is called novel when no semantically similar pretraining context can be retrieved; using this embedding-based measure, SmolLM models show longer-range reuse and instruction tuning increases novelty.

Reference graph

Works this paper leans on

31 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Einstein

    The Text must contain the exact name of the Entity or a clearly identifiable and standard part of the Entity's name (e.g., "Einstein" for "Albert Einstein") that contextually refers directly to the Entity

    work page

  2. [2]

    the Wahhabi school of Saudi Arabia

    The mention must refer to the Entity itself. Merely using the Entity's name as a descriptive modifier for something else associated with the Entity (e.g., "the Wahhabi school of Saudi Arabia" referring to the school, not the country) does not count as directly mentioning or discussing the Entity, unless the descriptive phrase is the Entity (e.g., "Univers...

    work page

  3. [3]

    Exclusions (Do NOT count as direct mention/discussion): Mentions of individuals or things related to the Entity but not the Entity itself (e.g., family members, associates, parts or products of the entity unless the mention clearly refers back to the entity)

    work page

  4. [4]

    Inferences based on context, related events, locations, or concepts if the Entity's name is not used to refer to the Entity itself according to Rule 3

    work page

  5. [5]

    In Proceedings of the 62nd An- nual Meeting of the Association for Compu- tational Linguistics (V olume 1: Long Papers) , pages 1280–1297, Bangkok, Thailand

    DeepSeekMoE: Towards ultimate ex- pert specialization in mixture-of-experts lan- guage models. In Proceedings of the 62nd An- nual Meeting of the Association for Compu- tational Linguistics (V olume 1: Long Papers) , pages 1280–1297, Bangkok, Thailand. Associ- ation for Computational Linguistics. Jacob Devlin, Ming-Wei Chang, Kenton Lee et al

    work page

  6. [6]

    Ennis’s teacher

    Possessives, actions, or attributes must be logically consistent with the entity’s type. Implausible associations, like assigning a teacher to Ennis, a town in County Clare, Ireland (e.g., “Ennis’s teacher”), are invalid. Consider each rule individually and assign an intermediate judgment of Pass or Fail. If any rule receives a Fail, the final decision mu...

    work page

  7. [9]

    In Proceedings of the 63rd An- nual Meeting of the Association for Compu- tational Linguistics (V olume 1: Long Papers) , pages 9691–9709, Vienna, Austria

    Between circuits and Chomsky: Pre- pretraining on formal languages imparts lin- guistic biases. In Proceedings of the 63rd An- nual Meeting of the Association for Compu- tational Linguistics (V olume 1: Long Papers) , pages 9691–9709, Vienna, Austria. Association for Computational Linguistics. Binyuan Hui, Jian Yang, Zeyu Cui et al. 2024. Qwen2.5-coder te...

    work page 2024

  8. [11]

    In The Thirteenth Interna- tional Conference on Learning Representations

    Knowledge entropy decay during lan- guage model pretraining hinders new knowl- edge acquisition. In The Thirteenth Interna- tional Conference on Learning Representations. Knowledgator. 2025. Flashdeberta. https: //github.com/Knowledgator/ FlashDeBERTa. Accessed: 2025-08-01. Oleksii Kuchaiev, Jason Li, Daniil Shleifer et al

    work page 2025

  9. [12]

    arXiv preprint arXiv:1909.09577

    Nemo: a toolkit for building ai appli- cations using neural modules. arXiv preprint arXiv:1909.09577. Komal Kumar, Tajamul Ashraf, Omkar Thawakar et al. 2025. Llm post-training: A deep dive into reasoning large language models. Patrick Lewis, Yuxiang Wu, Linqing Liu et al

    Pith/arXiv arXiv 1909

  10. [13]

    Transac- tions of the Association for Computational Lin- guistics, 9:1098–1115

    PAQ: 65 million probably-asked ques- tions and what you can do with them. Transac- tions of the Association for Computational Lin- guistics, 9:1098–1115. Belinda Z. Li, Maxwell Nye and Jacob Andreas

    work page

  11. [14]

    Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inter- national Joint Conference on Natural Language Processing (V olume 1: Long Papers) , pages 1813–1827, Online. Association for Computa- tional Linguistics. D Li, B Jiang, L Huang et al....

    work page 2024

  12. [15]

    In Findings of the Association for Com- putational Linguistics: ACL 2022 , pages 1720– 1732, Dublin, Ireland

    How pre-trained language models cap- ture factual knowledge? a causal-inspired anal- ysis. In Findings of the Association for Com- putational Linguistics: ACL 2022 , pages 1720– 1732, Dublin, Ireland. Association for Compu- tational Linguistics. Jiacheng Liu, Taylor Blanton, Yanai Elazar et al

    work page 2022

  13. [16]

    In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V ol- ume 3: System Demonstrations) , pages 178– 188, Vienna, Austria

    OLMoTrace: Tracing language model outputs back to trillions of training tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V ol- ume 3: System Demonstrations) , pages 178– 188, Vienna, Austria. Association for Compu- tational Linguistics. Jiacheng Liu, Sewon Min, Luke Zettlemoyer et al. 2024a. Infini-gram: ...

    Pith/arXiv arXiv 2019

  14. [17]

    In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Pa- pers), pages 9802–9822, Toronto, Canada

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Pa- pers), pages 9802–9822, Toronto, Canada. As- sociation for Computational Linguistics. Giuliano Martinelli, Edoardo Barba and Roberto Navigl...

    work page 2024

  15. [18]

    In Advances in Neural Information Processing Systems

    Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems. Niklas Muennighoff, Luca Soldaini, Dirk Groen- eveld et al. 2025. OLMoe: Open mixture-of- experts language models. In The Thirteenth In- ternational Conference on Learning Represen- tations. NVIDIA. 2025. Nvidia nemotron nano 2: An ac- curate and effici...

    Pith/arXiv arXiv 2025

  16. [19]

    In Proceedings of NAACL- HLT 2019: Demonstrations

    fairseq: A fast, extensible toolkit for se- quence modeling. In Proceedings of NAACL- HLT 2019: Demonstrations. Matthew E. Peters, Mark Neumann, Robert Lo- gan et al. 2019. Knowledge enhanced contex- tual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Confer...

    work page 2019

  17. [20]

    Nicholas Roberts, Niladri S

    How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Nicholas Roberts, Niladri S. Chatterji, Sharan Narang et al. 2025. Compute optimal scaling of skills: Knowledge vs reasoning. In Findings of the Association f...

    Pith/arXiv arXiv 2020

  18. [26]

    Elisabeth

    Distinguishing Similar Names: If the Text contains a name that is similar to the Entity's name but refers to a different person, place, or thing (e.g., a different historical figure, a different organization with a similar name, or differing spellings like "Elisabeth" vs. "Elizabeth" referring to different individuals), it does not count as a direct menti...

    work page

  19. [28]

    The cloze statement must preserve the full meaning and specificity of the original question

    Do not lose any context from the question. The cloze statement must preserve the full meaning and specificity of the original question. For example, Question: What TV network does Funny or Die Presents air on? Answer: HBO Cloze Statement: Funny or Die Presents airs on the TV network

    work page

  20. [29]

    If the question is incomplete or unclear, you may add minimal language to clarify the intended answer

    Rephrase the question into a natural, grammatically correct sentence. If the question is incomplete or unclear, you may add minimal language to clarify the intended answer. For example, Question: stevie cameron worked as a confidential informant for the rcmp during which controversial Cloze Statement: Stevie Cameron worked as a confidential informant for ...

    work page

  21. [30]

    End the statement right before the answer

    work page

  22. [31]

    capital city

    Use any specific labels or terms from the question. For example, if the question specifies "capital city" or "TV network," include that exact phrase in the cloze. Output ONLY the Cloze Statement at the end of thinking. Question: {question} Answer: {answer} Cloze Statement: Figure 19: Prompt given to Gemini 2.5 Flash Preview 6-17 to convert PAQ questions t...

    work page

  23. [2016]

    Badr AlKhamissi, Millicent Li, Asli Celikyilmaz et al

    A Neural Knowledge Language Model. Badr AlKhamissi, Millicent Li, Asli Celikyilmaz et al. 2022. A review on language models as knowledge bases. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch et al. 2025. Smollm2: When smol goes big – data-centric training of a small language model. Zeyuan Allen-Zhu and Yuanzhi Li. 2024. Physics of language models: part 3.1...

    work page 2022

  24. [2017]

    the hospital

    Reference-aware language models. In Proceedings of the 2017 Conference on Empir- ical Methods in Natural Language Processing , pages 1850–1859, Copenhagen, Denmark. As- sociation for Computational Linguistics. Linxi Zhao, Sofian Zalouk, Christian K. Belardi et al. 2025. Pre-training large memory language models with internal and external knowledge. Vilém ...

    work page 2017

  25. [2019]

    BERT: Pre-training of deep bidirec- tional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Lin- guistics. Y...

    work page 2019

  26. [2020]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min et al

    How can we know what language mod- els know? Transactions of the Association for Computational Linguistics, 8:423–438. Vladimir Karpukhin, Barlas Oguz, Sewon Min et al. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onlin...

    work page 2020

  27. [2021]

    Transactions of the Association for Computational Linguistics , 9:1012–1031

    Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics , 9:1012–1031. Ronen Eldan and Yuanzhi Li. 2023. Tinysto- ries: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759. Mor Geva, Jasmijn Bastings, Katja Filippova et al

    Pith/arXiv arXiv 2023

  28. [2022]

    ReFinED: An efficient zero-shot-capable approach to end-to-end entity linking. In Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Tech- nologies: Industry Track , pages 209–220, Hy- brid: Seattle, Washington + Online. Association for Computational Linguistics. Elie Bakou...

    work page 2022

  29. [2023]

    In Pro- ceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing , pages 12216–12235, Singapore

    Dissecting recall of factual associations in auto-regressive language models. In Pro- ceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing , pages 12216–12235, Singapore. Association for Computational Linguistics. Abbas Ghaddar and Phillippe Langlais. 2016. Wi- kiCoref: An English coreference-annotated cor- pus of Wikipedi...

    Pith/arXiv arXiv 2023

  30. [2024]

    Curran Associates, Inc

    How do large language models acquire factual knowledge during pretraining? In Ad- vances in Neural Information Processing Sys- tems, volume 37, pages 60626–60668. Curran Associates, Inc. Roi Cohen, Eden Biran, Ori Yoran et al. 2024. Evaluating the ripple effects of knowledge edit- ing in language models. Transactions of the Association for Computational L...

    work page 2024

  31. [2025]

    In Proceed- ings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (V ol- ume 1: Long Papers) , pages 16051–16081, Vi- enna, Austria

    The alternative annotator test for LLM- as-a-judge: How to statistically justify replac- ing human annotators with LLMs. In Proceed- ings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (V ol- ume 1: Long Papers) , pages 16051–16081, Vi- enna, Austria. Association for Computational Linguistics. Hoyeon Chang, Jinho Park, Seong...

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.