Pith. sign in

REVIEW 5 major objections 3 minor 97 references

The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora

T0 review · 5 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The announcement is the cue: deleting a structural heading makes following prose harder to predict at every model scale tested, while swapping the markup notation changes nothing.

desk verdict A transparent, pre-registered paper that adds a missing variable to pre-training data curation; the format recommendation rests on a disclosed but untested training-time bet. read the letter →

arxiv 2608.09093 v1 pith:CQVMQLIL submitted 2026-08-10 cs.CL cs.AI

E. M. Freeburg This is my paper · ORCID
classification cs.CLcs.AI
keywords pre-trainingcorporastructuralnotationannouncementcueclean-windowsurvivalboundaryinferencelong-contexttrainingpureframeshortcutlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pre-training corpora are usually described by what documents contain, not by how their arrangement is written down. This paper argues that this notation is an unrecorded training variable, and that for long-range reading the cue a model actually uses is the announcement, a short standalone line saying a boundary is here, not the markup sigil that writes it. Across thirteen corpora the paper measures clean-window survival, the share of fixed-length windows with no structural markup, and finds the scarce resource is long unmarked text, which has largely left modern mixtures. Across five base models and two pipelines, deleting an announcement makes following prose measurably harder to predict, while swapping the notation changes nothing; base writers do not put deleted announcements back. The paper ships a format, the pure frame, that deletes every announcement into a reversible sidecar, mixed against marked copies over announcement presence, and argues this should be aimed at the long-context training stage.

What carries the argument

The central instrument is clean-window survival, S(W), the deterministic fraction of non-overlapping W-token windows in a tokenized stream that contain no structural markup; a companion statistic is the distribution of clean runs, the longest markup-free spans. The paper pairs this with a conceptual distinction between the sigil, the markup character such as ##, and the announcement, the short line set off by whitespace that says a boundary is here. The hinge of the argument is the announcement's reliability: in narrative a structural sigil marks an authored boundary at precision 0.994, or 0.967 after documented typographic apparatus is removed, so the cue is near-perfectly predictive and therefore a shortcut, while the sigil itself is interchangeable. The pure frame is the machinery's output: paragraphs in authored order with every announcement deleted into a reversible sidecar, so a stream demands the boundary inference instead of receiving it.

What would settle it

Run the paper's three-arm training test over identical content: marked, pure, and paired arms at about 100 billion tokens per arm; if the paired arm ends indistinguishable from the marked arm on the held-out announcement probe, reliability was not the operative variable and the factorization collapses to a notation preference. The cheaper precursor is replacing 5 to 25 billion tokens of a 50-billion-token long-context stage with pure-frame text; if no instrument moves, the locus argument fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that across long prose, the cue a trained language model actually uses is the announcement, a short standalone line such as a chapter title saying that a boundary is here, and not the markup sigil that writes the announcement down. On a fifty-document ground-truth corpus, deleting the announcement makes the following prose measurably harder to predict at every reader scale tested, from 0.60B to 8.19B parameters, while swapping Markdown for a bare line moves nothing, inside a pre-registered bound under two percent of measured information gain. A census of thirteen corpora measured with the paper's clean-window survival statistic finds the scarce resource is long unmarked text: survival falls to 0.153 in a vision-converted PDF slice versus 0.889 in a standard web corpus, and the longest clean runs live in books, the row that left modern mixtures. The paper concludes that faithful flattening of markup is a no-op for boundary inference, and ships a format, the pure frame, that deletes every announcement into a reversible sidecar, mixed against marked copies over announcement presence rather than notation.

Load-bearing premise

The recommendation stands or falls on the bet that the reader-side difficulty from deleted announcements translates into a trainable capability, so that a model trained on announcement-free text learns the boundary inference; the paper says the decisive training run has not been run.

Editorial extensions

If this is right

  • Faithful flattening of markup does not restore the missing inference demand; only deleting the announcement line does, so corpus operators should act on announcements, not sigils.
  • The pure frame should be mixed with marked copies over announcement presence, so the same underlying boundary is sometimes announced and sometimes not, attacking cue reliability at the level that matters.
  • The long-context training stage, where the sequence is the document, is the right target: it is under one percent of the token budget and currently fed the most heavily marked material in the census.
  • Data cards should record extractor identity, conversion target, and clean-window survival at training context lengths, a field the paper argues one flagship PDF corpus already partially carries.
  • The scarce resource is long unmarked text, not unmarked text: institutional proceedings such as parliamentary records supply it, consumer transcripts and library scans do not, and books left modern mixtures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, data-mixing methods that reweight documents as already built cannot reach notation; the only operators that can are deletion-style transforms such as the pure frame, so the actionable design space for long-context curation shifts from weighting to format operators.
  • The announcement cost concentrates in roughly the first thousand tokens after a boundary, which predicts a testable training signature: a model trained on pure-frame long text should show its largest gains on prose just after boundaries, not uniformly across a document.
  • The notation-invariance null implies that information-gain or attention-based selectors for long-context data, which ignore serialization, may be picking on the wrong coordinate; comparing such selectors over marked versus pure-frame versions of the same works would settle it cheaply.
  • If the format works as intended, model-generated prose should carry fewer explicit signpost announcements; a blind or automated comparison of announcement density between models trained on the two mixtures would be a direct product-level check.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper introduces clean-window survival S(W), a deterministic, seedless count of how much of a token stream still demands boundary inference, and uses it to measure structural notation in thirteen public corpora. It reports a pre-registered converter study whose own prediction failed, a three-arm reading probe across five base models (plus a sixth replication) showing that deleting an announcement line raises next-token prediction difficulty while swapping the notation changes long-range information gain by a measured zero, and a writer-side generation probe examining whether base models re-impose markup. On the basis of these measurements it proposes a 'pure frame' format that deletes announcements into a reversible sidecar and recommends mixing marked and pure copies at training time, together with a data-card field for notation.

Significance. The paper's reader-side core is genuinely well controlled: the three-arm design holds content byte-identical across arms, the anchor set is identical across all readers, the notation-swap null is tight and pre-registered, and the announcement deficit replicates across six readers and three pipelines. The paper also deserves explicit credit for shipping reproducible artifacts, a falsifier ledger with pre-committed directions, a costed and pre-registered decisive test, and unusually candid disclosure of its own failed predictions and limitations. If the central claim is taken as the reader-side finding that the announcement, not the sigil, is the operative cue for long-range reading at fixed checkpoints, the evidence is strong. However, the paper's format recommendation is explicitly a bet about training-time behavior, and several census and converter-study results are weaker than the abstract suggests, so the central design claim currently rests on an untested empirical link.

major comments (5)
  1. [§9–§10] The pure-frame recommendation is explicitly a bet: §9 says 'We are betting that the factorization exists and that reliability is what forces it,' and §10's Test 3 is described as 'Specified and costed; not run.' The reader-side evidence in §7 cannot distinguish 'the training pressure never existed' from 'these readers never formed the machinery,' a limit the paper itself states in §11. Because the abstract and §9 present the format as 'what those measurements imply,' the central design claim currently rests on an untested training-time link. I ask the authors either to run Test 3 or to reword the central claim so that the format proposal is explicitly conditional on the paired-arm outcome.
  2. [§5.1, Table 2; abstract] The census's flagship number is not reproducible as published: Table 2 reports re-measured S(8,192) for the olmOCR slice as 0.271 and 0.296 against the published 0.153, and seven of the ten published rows have no recorded sample size, document count, sampling rule, revision or date. Although the qualitative claim that olmOCR has the lowest survival survives (0.271 is still below DCLM's 0.558), the abstract's 'falls to 0.153' states a number the paper's own re-verification does not support. The corrected value, the limited window counts (23, 62, 138), and the provisional status of the table should appear prominently, including in the abstract.
  3. [§5.4] The pre-registered converter null was measured on nine non-visual configurations, and the paper states in the same section that olmOCR and MinerU 'were provisioned and produced no rows.' Since the census's most contaminated slice is produced by a vision-language converter and §6 identifies that slice as the input to the long-context stage, the claim that modern converters under-mark cannot be extended to the vision-converted frontier without an additional assumption. The null should be explicitly scoped to non-visual converters, or the missing VLM rows should be run.
  4. [§8, Table 7] The writer-side front's headline 'bounded null' is not supported by the primary greedy-decoding arm: the repetition guard fired on 84–100% of continuations in every cell, leaving 77 of 1,328 generations, and the paper itself calls this a selection effect. Only the temperature-1.0 secondary arm on 31 anchors supports the I3 exact zero, while I1, I2 and I4 are unevaluable in the primary arm. The front's conclusions should be rewritten around the secondary arm, with I1/I2/I4 reported as indeterminate rather than as components of a 'bounded null.'
  5. [§7, §8, §11] There is an internal inconsistency in the announcement-reconstruction claim. §7 reports that R2 fired at 1.7B: the information-gain contrast +0.270 [+0.010, +0.521] excludes zero, which the pre-registered rule defines as larger readers reconstructing a deleted announcement. Yet §8 states that 'no reader recovers a deleted announcement from long-range context either,' and §11 lists R2 as a fired rule that 'cuts against this paper.' The §8 sentence should be corrected to say that reconstruction was observed at one reader and did not replicate at three larger ones; as written, it contradicts the paper's own pre-registered outcome.
minor comments (3)
  1. [§5.2] The supply-side verdict 'institutional, not consumer' rests on one sampling-sensitive row: reading from the head of a shard changes Hansard's median document by almost a factor of eight and S(8,192) from 0.930 to 0.757. The abstract should carry the caveat that this conclusion depends on the seeded-shuffle sampling rule.
  2. [§5.5, Table 5] The adjusted narrative precision of 0.967 lands in the registered SUPPORTED, WEAKENED band, not SUPPORTED AS STATED. The paper does say 'near-perfect reliability' rather than 'perfect,' but the abstract and §9 should make equally explicit that the reliability claim is scoped to narrative prose and does not hold for sustained argument.
  3. [§6] The long-context stage argument cites Olmo 3's long-context pool as 'to a rounding error, entirely olmOCR-converted science PDFs.' Given the olmOCR census row's re-measurement divergence, this sentence should cite both the published and re-measured survival values, or at least flag which value is being used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the announcement-vs-notation contrasts are empirical, pre-registered, and not derived from the paper's own definitions or from fitted parameters.

full rationale

The central claim — that the announcement, not the sigil, is the operative cue — rests on paired probe experiments over fixed public checkpoints with an identical anchor set. Deleting versus swapping notation on byte-identical text is an empirical contrast, not a consequence of the definition of 'announcement,' which is structural (a short line, set off by whitespace, that says a boundary is here). Clean-window survival is a deterministic instrument, and the census figures are measurements rather than predictions implied by the definition. The pure-frame validator's S=1.000 row is explicitly labeled a fixed point of the deletion rule and not independent evidence, so no hidden reduction is being passed off as a result. The only self-citation, to the author's prior arXiv:2603.27006, is used as disclosed motivation or as a prior assertion in the limitations section, not as load-bearing support for the new measurements. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The paper also pre-commits to falsification if the paired training arm is indistinguishable from the marked arm, so the decisive link is an explicitly untested bet rather than a circular derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on operational definitions and on cited background assumptions about shortcut learning, text statistics, and reader-constructed segmentation. No fitted constants force the headline results; the pre-registered thresholds define verdicts but are not tuned to data. The pure-frame training benefit is an explicit, untested bet.

free parameters (2)
  • supply-rule thresholds at horizon W = median document >= W and S(W) >= 0.50
    Hand-set, pre-registered conjuncts that define 'supplies the missing format'. The supply-test verdict ('institutional, not consumer') is sensitive to these thresholds, though they are not fitted to measurements.
  • cue-reliability audit thresholds (E5) = precision >= 0.99 with lower bound >= 0.95; adjusted narrative 0.967
    Pre-registered pass/fail thresholds chosen by hand to decide whether the cue is 'near-perfect'. The paper's SUPPORTED, WEAKENED verdict depends on them, and they are not derived from theory.
assumptions (4)
  • domain assumption Reader-constructed segmentation: readers build discourse boundaries from content rather than reading them off the page.
    Invoked in §1 as motivation from Zacks et al. 2007 and Zwaan and Radvansky 1998, explicitly not as evidence.
  • domain assumption Shortcut adoption is governed by reliability and availability, and induction-head formation is more sensitive to reliability than frequency.
    Used in §2 and §5.3 to argue that a low-volume cue can matter. Relies on Geirhos et al. 2020, Hermann et al. 2024, and Aoyama et al. 2026.
  • domain assumption Long-range correlations in text are caused by hierarchical document structure.
    Used in §6 to argue that the main pre-training stage cannot capture long-range structure. Relies on Alvarez-Lacalle et al. 2006 and Altmann et al. 2012.
  • ad hoc to paper Training-time benefit of the pure frame follows from the reader-side difficulty increase.
    Stated in §9 and §10 as a bet ('We are betting that the factorization exists'). Test 3 has not been run, so this is assumed for the design recommendation rather than measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora." pith.science (2026). https://pith.science/paper/CQVMQLIL

@misc{pith2026260809093,
  author       = {Pith},
  title        = {Pith review of: The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQVMQLIL}},
  note         = {Machine review of arXiv:2608.09093}
}
read the original abstract

How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.

Figures

Figures reproduced from arXiv: 2608.09093 by the authors.

Figure 1
Figure 1. The supply problem in one plane. Horizontal: median document length. Vertical: the [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. The pre-registered supply rule as a region: a corpus qualifies at horizon [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Where the tokens go. For each family, the main pre-training stage (sequence length at [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The three-arm probe at the published 0.6B reader. Swapping the notation moves nothing; [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: The two contrasts across the reader ladder, with 95% cluster bootstrap intervals. Left: [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: The announcement deficit in space, over one entire book, at Qwen3-1.7B-Base. Left: nats [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: What a base model writes back, by the notation it was shown. Detector-visible markup [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

97 extracted references · 49 canonical work pages

  1. [1]

    OLMo-core: src/scripts/official/OLMo3/

    Allen Institute for AI. OLMo-core: src/scripts/official/OLMo3/. https://github.com/ allenai/OLMo-core, 2026. Read at commit064b172(2026-07-29)

    2026

  2. [2]

    Physics of language models: Part 3.1, knowledge storage and extraction.arXiv preprint arXiv:2309.14316, 2023

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction.arXiv preprint arXiv:2309.14316, 2023

    arXiv 2023

  3. [3]

    Altmann, Giampaolo Cristadoro, and Mirko Degli Esposti

    Eduardo G. Altmann, Giampaolo Cristadoro, and Mirko Degli Esposti. On the origin of long-range correlations in texts.Proceedings of the National Academy of Sciences, 109(29): 11582–11587, 2012. DOI 10.1073/pnas.1117723109

    doi:10.1073/pnas.1117723109 2012

  4. [4]

    Álvarez-Lacalle, B

    E. Álvarez-Lacalle, B. Dorow, J.-P. Eckmann, and E. Moses. Hierarchical structures induce long-range dynamical correlations in written texts.Proceedings of the National Academy of Sciences, 103(21):7956–7961, 2006. DOI 10.1073/pnas.0510673103

    doi:10.1073/pnas.0510673103 2006

  5. [5]

    Predicting the emergence of induction heads in language model pretraining.arXiv preprint arXiv:2511.16893v3, 2026

    Tatsuya Aoyama, Ethan Gotlieb Wilcox, and Nathan Schneider. Predicting the emergence of induction heads in language model pretraining.arXiv preprint arXiv:2511.16893v3, 2026. Version-pinned: v3 (ICML 2026 camera-ready), Eq. 7

    arXiv 2026

  6. [6]

    Docling technical report.arXiv preprint arXiv:2408.09869, 2024

    Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Fabian Lindlbauer, Kasper Dinkla, et al. Docling technical report.arXiv preprint arXiv:2408.09869, 2024

    arXiv 2024

  7. [7]

    Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

    Vedant Badoni, Danqi Chen, and Xinyi Wang. Hubs or fringes: Pretraining data selection via web graph centrality.arXiv preprint arXiv:2606.11499, 2026

    work page Pith review arXiv 2026

  8. [8]

    One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

    Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu, and Massoud Pedram. One token away from collapse: The fragility of instruction-tuned helpfulness.arXiv preprint arXiv:2604.13006, 2026

    work page Pith review arXiv 2026

Show all 97 references
  1. [9]

    trafilatura: trafilatura/xml.py

    Adrien Barbaresi. trafilatura: trafilatura/xml.py. https://github.com/adbar/ trafilatura, 2026. Version 2.2.0; read at commitc1bc953(2026-07-31)

    2026

  2. [10]

    so much depends / upon / a whitespace: Why whitespace matters for poets and LLMs

    Sriharsh Bhyravajjula, Melanie Walsh, Anna Preus, and Maria Antoniak. so much depends / upon / a whitespace: Why whitespace matters for poets and LLMs. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computation...

    2025

  3. [11]

    Datasheet for the Pile.arXiv preprint arXiv:2201.07311, 2022

    Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the Pile.arXiv preprint arXiv:2201.07311, 2022

    2022 arXiv

  4. [12]

    Document structure in long document transformers

    Jan Buchmann, Max Eichler, Jan-Micha Bodensohn, Ilia Kuznetsov, and Iryna Gurevych. Document structure in long document transformers. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Volume 1: Long Papers, pa...

    2024

  5. [13]

    Institutional books 1.0: A 242B token dataset from Harvard Library’s collections, refined for accuracy and usability.arXiv preprint arXiv:2506.08300, 2025

    Matteo Cargnelutti, Catherine Brobston, John Hess, Jack Cushman, Kristi Mukk, Aristana Scourtas, Kyle Courtney, Greg Leppert, Amanda Watson, Martha Whitehead, and Jonathan Zittrain. Institutional books 1.0: A 242B token dataset from Harvard Library’s collections, refined for a...

    2025 arXiv

  6. [14]

    The hyperfitting phenomenon: Sharpening and stabilizing LLMs for open-ended text generation

    Fredrik Carlsson, Fangyu Liu, Daniel Ward, Murathan Kurfali, and Joakim Nivre. The hyperfitting phenomenon: Sharpening and stabilizing LLMs for open-ended text generation. In International Conference on Learning Representations (ICLR), 2025

    2025

  7. [15]

    LADM: Long-context training data selection with attention-based dependency measurement for LLMs

    Jianghao Chen, Junhong Wu, Yangyifan Xu, and Jiajun Zhang. LADM: Long-context training data selection with attention-based dependency measurement for LLMs. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025

    2025

  8. [16]

    Chen, Xikun Zhang, Fan Bai, Zhengding Hu, and Zhen Wang

    Michael K. Chen, Xikun Zhang, Fan Bai, Zhengding Hu, and Zhen Wang. Demystifying training-time augmentation for data-constrained language model pretraining.arXiv preprint arXiv:2606.16246, 2026

    2026 arXiv

  9. [17]

    Data mixing for large language models pretraining: A survey and outlook.Data Intelligence, 8, 2026

    Zhuo Chen, Yuxuan Miao, Supryadi, and Deyi Xiong. Data mixing for large language models pretraining: A survey and outlook.Data Intelligence, 8, 2026

    2026

  10. [18]

    BeyondWeb: Lessons from scaling synthetic data for trillion-scale pretraining

    DatologyAI. BeyondWeb: Lessons from scaling synthetic data for trillion-scale pretraining. arXiv preprint arXiv:2508.10975, 2025

    2025 arXiv

  11. [19]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024

    DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  12. [20]

    Beyond length: Quantifying long-range information for long-context LLM pretraining data

    Haoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu, Yizhou Sun, Yi-An Ma, and Yeyun Gong. Beyond length: Quantifying long-range information for long-context LLM pretraining data. InInternational Conference on Learning Representations (ICLR), 2026. Venue confirmed via OpenReview; ...

    2026

  13. [21]

    Documenting large webtext corpora: A case study on the Colossal Clean Crawled Corpus

    Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the Colossal Clean Crawled Corpus. InProceedings of the 2021 Conference on Empirical Methods in Nat...

    2021

  14. [22]

    Smith, and Jesse Dodge

    Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. What’s in my big data? InInternational Conference on Learning Representat...

    2024

  15. [23]

    DoGE:Domainreweightingwithgeneralization estimation.arXiv preprint arXiv:2310.15393, 2024

    SiminFan, MatteoPagliardini, andMartinJaggi. DoGE:Domainreweightingwithgeneralization estimation.arXiv preprint arXiv:2310.15393, 2024

    2024 arXiv

  16. [24]

    E. M. Freeburg. The last fingerprint: How Markdown training shapes LLM prose.arXiv preprint arXiv:2603.27006, 2026

    2026

  17. [25]

    NExtLong: Toward effective long-context training without long documents

    Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. NExtLong: Toward effective long-context training without long documents. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  18. [26]

    The Pile: An 800GB dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  19. [27]

    Wichmann

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2:665–673, 2020. DOI 10.1038/s42256-020-00257-z. 39

    2020 doi

  20. [28]

    Hermann, Hossein Mobahi, Thomas Fel, and Michael C

    Katherine L. Hermann, Hossein Mobahi, Thomas Fel, and Michael C. Mozer. On the foundations of shortcut learning. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  21. [29]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  22. [30]

    FinePDFs dataset card.https://huggingface.co/datasets/HuggingFaceFW/ finepdfs, 2025

    Hugging Face. FinePDFs dataset card.https://huggingface.co/datasets/HuggingFaceFW/ finepdfs, 2025. Accessed 2026-08-09

    2025

  23. [31]

    SmolLM3: smol, multilingual, long-context reasoner.https://huggingface

    Hugging Face. SmolLM3: smol, multilingual, long-context reasoner.https://huggingface. co/blog/smollm3, 2025. Accessed 2026-08-09

    2025

  24. [32]

    datatrove: src/datatrove/pipeline/extractors/trafilatura.py

    Hugging Face. datatrove: src/datatrove/pipeline/extractors/trafilatura.py. https: //github.com/huggingface/datatrove, 2026. Read at commit0eb8e30(2026-08-06)

    2026

  25. [33]

    propella-1: Multi- property document annotation for LLM data curation at scale.arXiv preprint arXiv:2602.12414,

    Maximilian Idahl, Benedikt Droste, Björn Plüster, and Jan Philipp Harries. propella-1: Multi- property document annotation for LLM data curation at scale.arXiv preprint arXiv:2602.12414,

  26. [34]

    ISO 32000-1:2008 — document management: Portable document format — part 1: PDF 1.7.https://www.iso.org/standard/51502.html, 2008

    International Organization for Standardization. ISO 32000-1:2008 — document management: Portable document format — part 1: PDF 1.7.https://www.iso.org/standard/51502.html, 2008

    2008

  27. [35]

    EntropyLong: Effective long-context training via predictive uncertainty.arXiv preprint arXiv:2510.02330, 2025

    Junlong Jia, Ziyang Chen, Xing Wu, Chaochen Gao, Zijia Lin, Debing Zhang, Songlin Hu, and Binghui Guo. EntropyLong: Effective long-context training via predictive uncertainty.arXiv preprint arXiv:2510.02330, 2025

    2025

  28. [36]

    Characterizing narrative content in web-scale LLM pretraining data.arXiv preprint arXiv:2606.19468, 2026

    Teagan Johnson, Elliott Ash, Andrew Piper, and Maria Antoniak. Characterizing narrative content in web-scale LLM pretraining data.arXiv preprint arXiv:2606.19468, 2026

    2026 arXiv

  29. [37]

    Feder Cooper, Aviya Skowron, et al

    Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella Biderman, Baber Abbasi, Luca Soldaini, Enrico Shippole, A. Feder Cooper, Aviya Skowron, et al. The Com- mon Pile v0.1: An 8TB dataset of public domain and openly licensed text.arXiv preprint arXiv:2506.0...

    2025 arXiv

  30. [38]

    Nemotron-CC-Math: A 133 billion-token-scale high quality math pretraining dataset.arXiv preprint arXiv:2508.15096, 2025

    Rabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Moham- mad Shoeybi, and Bryan Catanzaro. Nemotron-CC-Math: A 133 billion-token-scale high quality math pretraining dataset.arXiv preprint arXiv:2508.15096, 2025

    2025 arXiv

  31. [39]

    Is a document educational or just Wikipedia- style? pitfalls of classifier-based quality filtering

    Mateusz Klimaszewski and Piotr Andruszkiewicz. Is a document educational or just Wikipedia- style? pitfalls of classifier-based quality filtering. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), 2026

    2026

  32. [40]

    Quality at a glance: An audit of web-crawled multilingual datasets.Transactions of the Association for Computational Linguistics, 10:50–72, 2022

    Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii- Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets.Transactions of the Association for C...

    2022

  33. [41]

    The format tax.arXiv preprint arXiv:2604.03616, 2026

    Ivan Yee Lee, Loris D’Antoni, and Taylor Berg-Kirkpatrick. The format tax.arXiv preprint arXiv:2604.03616, 2026. 40

    2026 arXiv

  34. [42]

    DataComp-LM: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. DataComp-LM: In search of the next generation of training sets for language models. InAdvances in Neural Information Processing Systems (N...

    2024

  35. [43]

    Jeffrey Li, Joshua P. Gardner, Doug Kang, Fangping Shi, Karanjeet Singh, Chun-Liang Li, Herumb Shandilya, David Leo Wright Hall, Oncel Tuzel, Percy Liang, Ludwig Schmidt, Hadi Pouransari, andFartashFaghri. Beyondasingleextractor: Re-thinkingHTML-to-textextraction for LLM pre-t...

  36. [44]

    Zehan Li, Yutong Zhu, Siyang Wu, Honglin Bao, and James A. Evans. Narrative flattening: How post-training compresses thematic, affective, and stylistic variation in LLM fiction.arXiv preprint arXiv:2605.27878, 2026

    2026 arXiv

  37. [45]

    URLhttps://aclanthology.org/2026

    Association for Computational Linguistics, 2026. URLhttps://aclanthology.org/2026. findings-eacl.307/

    2026

  38. [46]

    Liu, Shaomian Zheng, He Cao, Dingnan Jin, Qing Cui, and Jun Zhou

    June M. Liu, Shaomian Zheng, He Cao, Dingnan Jin, Qing Cui, and Jun Zhou. Improv- ing cross-format robustness in language models with multi-format training.arXiv preprint arXiv:2606.11643, 2026

    2026 arXiv

  39. [47]

    Lin and Max Tegmark

    Henry W. Lin and Max Tegmark. Criticality in formal languages and statistical physics.Entropy, 19(7):299, 2017

    2017

  40. [48]

    RegMix: Data mixture as regression for language model pre-training

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. RegMix: Data mixture as regression for language model pre-training. In International Conference on Learning Representations (ICLR), 2025

    2025

  41. [49]

    Dripper: Token-efficient main HTML extraction with a lightweight LM.arXiv preprint arXiv:2511.23119, 2026

    Mengjie Liu, Jiahui Peng, Wenchang Ning, Pei Chu, Jiantao Qiu, Ren Ma, He Zhu, Rui Min, Lindong Lu, Linfeng Hou, et al. Dripper: Token-efficient main HTML extraction with a lightweight LM.arXiv preprint arXiv:2511.23119, 2026

    2026

  42. [50]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Llama Team, AI @ Meta. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  43. [51]

    Sentence-level contextual entrainment in large language models

    Yang Liu and Chenhui Chu. Sentence-level contextual entrainment in large language models. arXiv preprint arXiv:2606.24077, 2026

    2026 arXiv

  44. [52]

    AICC: Parse HTML finer, make models better — a 7.3T AI-ready corpus built by a model-based HTML parser.arXiv preprint arXiv:2511.16397, 2025

    Ren Ma, Jiantao Qiu, Chao Xu, Pei Chu, Kaiwen Liu, Pengli Ren, Yuan Qu, Jiahui Peng, Linfeng Hou, Mengjie Liu, et al. AICC: Parse HTML finer, make models better — a 7.3T AI-ready corpus built by a model-based HTML parser.arXiv preprint arXiv:2511.16397, 2025

    2025

  45. [53]

    Klein, and Jesse Dodge

    Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren F. Klein, and Jesse Dodge. AboutMe: Using self-descriptions in webpages to document the effects of English pretraining data filters. InProceedings of the 62nd Annual Meeting of the Association for C...

    2024

  46. [54]

    Measuring fingerprints of web-filtered text datasets and fingerprint propagation through training

    Youssef Mansour and Reinhard Heckel. Measuring fingerprints of web-filtered text datasets and fingerprint propagation through training. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. Spotlight. 41

    2025

  47. [55]

    Rephrasing the web: A recipe for compute and data-efficient language modeling.arXiv preprint arXiv:2401.16380, 2024

    Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling.arXiv preprint arXiv:2401.16380, 2024

    2024 arXiv

  48. [56]

    Register always matters: Analysis of LLM pretraining data through the lens of language variation

    Amanda Myntti, Erik Henriksson, Veronika Laippala, and Sampo Pyysalo. Register always matters: Analysis of LLM pretraining data through the lens of language variation. InProceedings of the Second Conference on Language Modeling (COLM), 2025

    2025

  49. [57]

    Autocorrelations decay in texts and applicability limits of language models

    Nikolay Mikhaylovskiy and Ilya Churilov. Autocorrelations decay in texts and applicability limits of language models. InComputational Linguistics and Intellectual Technologies (Dialogue), 2023

    2023

  50. [58]

    OctoLong: Mid-training on cross-repository code contexts enhances long-context modeling.arXiv preprint arXiv:2608.05141, 2026

    Indraneil Paul, Falko Helm, Goran Glavaš, and Iryna Gurevych. OctoLong: Mid-training on cross-repository code contexts enhances long-context modeling.arXiv preprint arXiv:2608.05141, 2026

    2026 arXiv

  51. [59]

    How can we synthesize high-quality pretraining data? a systematic study of prompt design, generator model, and source data

    Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik, Guilherme Penedo, Hynek Kydlíček, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, and Thomas Wolf. How can we synthesize high-quality pretraining data? a systematic study of...

    2026

  52. [60]

    Chapter captor: Text segmentation in novels

    Charuta Pethe, Allen Kim, and Steven Skiena. Chapter captor: Text segmentation in novels. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8373–8383. Association for Computational Linguistics, 2020. URLhttps: //aclantholog...

    2020

  53. [61]

    The FineWeb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. InAdvances in Neural Information Processing Systems (NeurIPS), Dat...

    2024

  54. [62]

    epub2txt-all (the Books3 converter)

    Shawn Presser. epub2txt-all (the Books3 converter). https://github.com/shawwn/scrap,

  55. [63]

    olmOCR: Unlocking trillions of tokens in PDFs with vision language models.arXiv preprint arXiv:2502.18443, 2025

    Jake Poznanski, Aman Rangapur, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. olmOCR: Unlocking trillions of tokens in PDFs with vision language models.arXiv preprint arXiv:2502.18443, 2025

    2025

  56. [64]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling.arXiv preprint arXiv:1911.05507, 2019

    1911 arXiv

  57. [65]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21(140):1–67, 2020

    2020

  58. [66]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  59. [67]

    From reflection to repair: A scoping review of dataset documentation tools.arXiv preprint arXiv:2602.15968, 2026

    Pedro Reynolds-Cuéllar, Marisol Wong-Villacres, Adriana Alvarado Garcia, and Heila Precel. From reflection to repair: A scoping review of dataset documentation tools.arXiv preprint arXiv:2602.15968, 2026

    2026

  60. [68]

    Scale-dependent relationships in natural language.arXiv preprint arXiv:1912.07506, 2019

    Aakash Sarkar and Marc Howard. Scale-dependent relationships in natural language.arXiv preprint arXiv:1912.07506, 2019. 42

    1912 arXiv

  61. [69]

    RolmOCR model card.https://huggingface.co/reducto/RolmOCR, 2025

    Reducto AI. RolmOCR model card.https://huggingface.co/reducto/RolmOCR, 2025. Ac- cessed 2026-08-09

    2025

  62. [70]

    Dolma: an open corpus of three trillion tokens for language model pretraining research

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. InProceedings of the 62nd Annua...

    2024

  63. [71]

    Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset. InProceedings of the 63rd Annual Meeting of the Associa...

    2025

  64. [72]

    Mutual information scaling and expressive power of sequence models.arXiv preprint arXiv:1905.04271, 2019

    Huitao Shen. Mutual information scaling and expressive power of sequence models.arXiv preprint arXiv:1905.04271, 2019

    1905 arXiv

  65. [73]

    ChapterBreak: A challenge dataset for long-range language models

    Simeng Sun, Katherine Thai, and Mohit Iyyer. ChapterBreak: A challenge dataset for long-range language models. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 37...

    2022

  66. [74]

    Accessed 2026-08-09

    Aaron Swartz, Chris Balakrishnan, et al.html2text: Convert HTML to Markdown-formatted text.https://github.com/Alir3z4/html2text, 2024. Accessed 2026-08-09

    2024

  67. [75]

    Zico Kolter, and Zhuang Liu

    Mingjie Sun, Yida Yin, Zhiqiu Xu, J. Zico Kolter, and Zhuang Liu. Idiosyncrasies in large language models. InProceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  68. [76]

    Olmo 3.arXiv preprint arXiv:2512.13961, 2026

    Team Olmo. Olmo 3.arXiv preprint arXiv:2512.13961, 2026

    2026 arXiv

  69. [77]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  70. [78]

    Beyond rephrasing: Book-level organization improves synthetic textbook data for mid-training.arXiv preprint arXiv:2607.28109, 2026

    Jiawen Tao, Miao Peng, Yaoming Li, Xiaokun Yuan, Mengzhou Wu, Wenhan Yu, Guoan Wang, Nuo Chen, Tong Yang, and Maxm Pan. Beyond rephrasing: Book-level organization improves synthetic textbook data for mid-training.arXiv preprint arXiv:2607.28109, 2026

    2026 arXiv

  71. [79]

    MinerU: An open-source solution for precise document content extraction.arXiv preprint arXiv:2409.18839, 2024

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. MinerU: An open-source solution for precise document content extraction.arXiv preprint arXiv:2409.18839, 2024

    2024 arXiv

  72. [80]

    Gonzalez, and Sewon Min

    Yichuan Wang, Zhifei Li, Zirui Wang, Paul Teiletche, Lesheng Jin, Matei Zaharia, Joseph E. Gonzalez, and Sewon Min. PIXELRAG: Web screenshots beat text for retrieval-augmented generation.arXiv preprint arXiv:2606.28344, 2026

    2026 arXiv

  73. [81]

    Right reset: Chunking by prefix removal.arXiv preprint arXiv:2608.04330, 2026

    Mike Vegeto. Right reset: Chunking by prefix removal.arXiv preprint arXiv:2608.04330, 2026. Preprint

    2026 arXiv

  74. [82]

    LongAttn: Selecting long-context training data via token-level attention

    Longyun Wu, Dawei Zhu, Guangxiang Zhao, Zhuocheng Yu, Junfeng Ran, Xiangyu Wong, Lin Sun, and Sujian Li. LongAttn: Selecting long-context training data via token-level attention. arXiv preprint arXiv:2502.16860, 2025

    2025 arXiv

  75. [83]

    Morris, and Lionel Levine

    Wilson Wu, John X. Morris, and Lionel Levine. Do language models plan ahead for future tokens? InConference on Language Modeling (COLM), 2024

    2024

  76. [84]

    Datasheets aren’t enough: DataRubrics for automated quality metrics and accountability.arXiv preprint arXiv:2506.01789, 2025

    Genta Indra Winata, David Anugraha, Emmy Liu, Alham Fikri Aji, Shou-Yi Hung, Aditya Parashar, Patrick Amadeus Irawan, Ruochen Zhang, Zheng-Xin Yong, Jan Christian Blaise Cruz, Niklas Muennighoff, et al. Datasheets aren’t enough: DataRubrics for automated quality metrics and ac...

    2025 arXiv

  77. [85]

    Learning to break the loop: Analyzing and mitigating repetitions for neural text generation

    Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  78. [86]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance

    Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  79. [87]

    Le, Tengyu Ma, and Adams Wei Yu

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  80. [88]

    Zacks, Nicole K

    Jeffrey M. Zacks, Nicole K. Speer, Khena M. Swallow, Todd S. Braver, and Jeremy R. Reynolds. Event perception: A mind-brain perspective.Psychological Bulletin, 133(2):273–293, 2007. Cited as motivation, not as evidence, for the claim that discourse segmentation is constructed ...

    2007

  81. [89]

    From lists to emojis: How format bias affects model alignment

    Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists to emojis: How format bias affects model alignment. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 26940–26961, 2025

    2025

  82. [90]

    The price of format: Diversity collapse in LLMs.arXiv preprint arXiv:2505.18949, 2025

    Longfei Yun, Chenyang An, Zilong Wang, Letian Peng, and Jingbo Shang. The price of format: Diversity collapse in LLMs.arXiv preprint arXiv:2505.18949, 2025

    2025 arXiv

  83. [91]

    The structural attention tax: How retrieval format hijacks in-context learning independent of content.arXiv preprint arXiv:2606.11198, 2026

    Yuqi Zhang and Di Zhang. The structural attention tax: How retrieval format hijacks in-context learning independent of content.arXiv preprint arXiv:2606.11198, 2026. Preprint

    2026 arXiv

  84. [92]

    Where does long-context supervision actually go? effective-context exposure balancing.arXiv preprint arXiv:2605.10544, 2026

    Jinchang Zhu, Jindong Li, Chengyu Zou, Rong Fu, Chao Wang, Haowei He, and Menglin Yang. Where does long-context supervision actually go? effective-context exposure balancing.arXiv preprint arXiv:2605.10544, 2026

    2026 arXiv

  85. [93]

    Scalable visual pretraining for language intelligence.arXiv preprint arXiv:2607.09657, 2026

    Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Huanze Tang, Yunhua Zhou, Demin Song, Kuikun Liu, Haochen Ye, et al. Scalable visual pretraining for language intelligence.arXiv preprint arXiv:2607.09657, 2026

    2026 arXiv

  86. [94]

    Zwaan and Gabriel A

    Rolf A. Zwaan and Gabriel A. Radvansky. Situation models in language comprehension and memory.Psychological Bulletin, 123(2):162–185, 1998. Cited as motivation, not as evidence. 44

    1998

  87. [96]

    Enhancing LLM knowledge learning through generalization.arXiv preprint arXiv:2503.03705, 2025

    Mingkang Zhu, Xi Chen, Zhongdao Wang, Bei Yu, Hengshuang Zhao, and Jiaya Jia. Enhancing LLM knowledge learning through generalization.arXiv preprint arXiv:2503.03705, 2025

    2025 arXiv

  88. [2020]

    Read at commit31fd266(2026-07-30)

    2026

  89. [2026]

    Preprint; no venue on the arXiv record

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.