Pith. sign in

REVIEW 4 major objections 6 minor 17 references

Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The Too Long, Didn't Model benchmark measures seven frontier LLMs on whole novels and finds that none retain stable understanding beyond 64k tokens, despite advertised context windows up to 10M.

desk verdict Useful long-context benchmark with a clever reference-construction trick, but the abstract's universal negative is contradicted by the paper's own Table 2. read the letter →

arxiv 2505.14925 v1 pith:EMBLEMST submitted 2025-05-20 cs.CL cs.AIcs.LG

Sil Hamilton , Rebecca M. M. Hicke , Matthew Wilkens , David Mimno This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords long-contextlanguagemodelsLLMbenchmarknovelunderstandingsummarizationentitytrackingnarrativetimelostinthemiddleevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that advertised long-context windows of up to ten million tokens do not translate into stable understanding of long, structurally complex texts. To show this, it introduces the Too Long, Didn't Model (TLDM) benchmark, which asks language models to summarize forty novels, track characters' locations, and estimate elapsed narrative time at four window sizes. The benchmark compares a model's full-novel answer to a reference assembled from the model's own per-chapter answers, avoiding the need for human annotations. Across seven frontier models, performance stays consistent for texts under 64k tokens and then declines, sharply for summarization and storyworld tracking. The result matters because it suggests current long-context evaluation misses exactly the integrative, multi-step understanding that real long-document use demands.

What carries the argument

The load-bearing mechanism is the TLDM benchmark's 'associative ground construction': because a novel-level answer can be decomposed into chapter-level answers, the authors build a model-specific reference by running the model on each chapter separately and concatenating the outputs (concatenated summaries, last recorded locations per character, summed time estimates). Full-novel predictions under five treatments — title and author only, unaltered text, one chapter per user message, truncation to the window of interest, and shuffled chapters — are then scored against this reference with semantic similarity, Jaccard plus semantic similarity, and absolute relative error. The distance between a model's long-context answer and its own short-context answer is the stability metric that carries the argument; it is what allows a benchmark with no human ground truth to locate where in the context window understanding begins to fail.

What would settle it

Pick a small set of TLDM novels and have human annotators produce reference summaries, final character locations, and elapsed time. Then score each model's full-novel outputs against these human references instead of against its own concatenated chapter outputs; if similarity no longer declines with length across the <32k to >128k bins, the reported 64k wall is an artifact of the synthetic-reference design rather than a genuine loss of long-context understanding.

Watch

Extended reading notes

Core claim

The central claim is that none of the seven tested 2025 frontier LLMs retain stable understanding beyond roughly 64k tokens, despite context windows ranging from 128k to 10 million tokens. Stability is defined operationally: the similarity between what a model reports when given an entire novel and what it reports when each chapter is processed independently and the outputs are concatenated. Above the 64k–128k length bin, this similarity falls for every model on at least one task, with the steepest drops in storyworld reporting (last known locations of characters) and, for several open-weight models, in summarization. Narrative time estimation degrades less, and no model performs well at storyworld tracking even on shorter texts. The authors conclude that models' effective context for complex understanding is far smaller than their advertised context, and that document order matters: shuffled chapters reduce summary and storyworld performance, while truncating the input to the requested window improves it.

Load-bearing premise

The load-bearing premise is that a reference assembled from the model's own chapter-level outputs is a faithful proxy for correct novel-level answers; if chapter-level outputs are inaccurate, or if concatenating them introduces artifacts, the stability metric measures agreement with a synthetic reference rather than real understanding of the full novel.

Editorial extensions

If this is right

  • Advertised context windows of 1M–10M tokens overstate practical capability: for complex narrative understanding, the effective limit found here is near 64k tokens.
  • Long-context evaluation should move beyond needle-in-a-haystack retrieval to tasks that require integrating information across the whole input.
  • Larger models are more stable at long contexts, so open-weight models that run on modest hardware are likely to remain worse at whole-document understanding.
  • Document linearity helps: shuffling chapters degrades summaries and storyworld tracking, implying some models rely on inelastic mechanisms for tracking narrative order.
  • Truncating inputs to the asked-for window improves performance, indicating that extraneous context actively hurts focused long-context tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reference is built from the model's own chapter-level outputs, TLDM measures self-consistency rather than correctness; a model that is confidently wrong in the same way at both scales could score high, so the absolute capability numbers are probably optimistic.
  • The 64k ceiling is a property of current models, not of the benchmark alone; if future training changes how attention distributes over long inputs, the same protocol should reveal a moving threshold rather than a fixed wall.
  • The same decomposition could apply to non-fiction books, technical manuals, or legal corpora, and might be a cheap way to test retrieval-augmented or hierarchical-summarization systems without human labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces TLDM, a long-context benchmark built from forty English-language novels in four token-length bins (<32k to >128k), with three narrative-understanding tasks (one-sentence-per-chapter summarization, storyworld character-location tracking, and narrative-time estimation), five text treatments (title-only, whole novel, per-chapter user messages, truncation to a window of interest, and shuffled chapters) and their combinations. Seven 2025-vintage LLMs are evaluated by measuring the similarity between whole-novel outputs and a model-specific reference built from concatenated chapter-level outputs. The authors report that performance degrades above 64k tokens, conclude that no tested model retains stable understanding beyond 64k tokens, and argue that model scale and text linearity are the main drivers of long-context performance.

Significance. The benchmark and its released code/data are a genuine contribution: the treatment design (especially the shuffle and window-of-interest manipulations) gives a more informative decomposition of long-context failure than needle-in-a-haystack tests, the random-baseline columns are good practice, the T1 memorization control is thoughtful, and the paper is transparent about the absence of human ground truth. If the central empirical claim were supported, the result would be an important caution about advertised million-token context windows. However, the headline claim is not supported by the paper's own Table 2 under any pre-specified notion of 'stable,' the stability criterion is never defined, and the evaluation measures self-consistency rather than understanding, a gap the Limitations section acknowledges but the abstract does not. The finding that is actually demonstrated is weaker and model-dependent: a subset of models decline sharply on some tasks above 64k tokens, with small open-weight models declining most steeply.

major comments (4)
  1. [Abstract / §4, Table 2] The abstract's universal negative—'none of seven tested frontier LLMs retain stable understanding beyond 64k tokens'—is contradicted by the paper's own results. In Table 2, GPT-4.1 summary similarity is 0.80/0.81/0.81/0.82 across the four bins, Llama 4 Scout summary is 0.76/0.77/0.74/0.77 and time is 0.55/0.60/0.57/0.58, and Gemini 2.0 Flash time is 0.55/0.67/0.54/0.63. These are flat within plausible noise, and §4 itself says GPT-4.1 is 'most consistent across all context lengths' and that Llama 4 Scout and Gemini 2.0 Flash 'achiev[e] reliable performance in summary and time estimation over all lengths.' Since 'stable understanding' is never defined (per task, averaged over tasks, or thresholded), the central claim is underdetermined; under any per-task threshold, at least one model is stable on at least one task. The claim needs to be replaced with a pre-specified stability criterion and per-model, per-task statements that match the table.
  2. [§3 Evaluation; Limitations] The evaluation replaces ground truth with each model's own chapter-level outputs, so the reported scores measure self-consistency between long- and short-context behavior, not correctness of novel understanding. The Limitations paragraph concedes this ('we lack maximally robust assessments of model vs. human capabilities'), but the abstract's 'understanding' framing inherits the fragility: if chapter-level outputs are inaccurate, agreement between long and short contexts is uninformative. The storyworld task is the clearest case: random baselines are B=0.00 and even short-length scores are 0.18–0.27, near the floor, so the observed 'decline' at longer lengths is partly a floor effect. Additionally, footnote 8 says the storyworld reference is built by 'recurrently passing in characters from previous chapters to stabilize predictions,' a chaining mechanism that the long-context treatment does not receive; the comparison therefore mixes task ability with reference-construction artifacts.
  3. [§4, Table 2 and Figure 3] No measure of variance or significance is reported anywhere. Each length bin averages only 10 novels, the API models are presumably sampled once, and the qualitative contrast between 'performance is consistent below 64k' and 'declines considerably above 64k' is not backed by error bars, confidence intervals, or significance tests. Given that several adjacent-bin differences in Table 2 are 0.01–0.04 (e.g., GPT-4.1 summary 0.80→0.82; Llama 4 Scout time 0.55→0.58), the paper's central contrast is not distinguishable from noise without such statistics. The authors should report per-model, per-bin variance or at least multiple runs, and re-derive their qualitative conclusions from a pre-specified comparison.
  4. [§5 Discussion] The claim that 'long-context abilities improve linearly with model size' is asserted without any supporting regression, figure, or statistical test, and the design is confounded: models differ in native context-window support (128k to 10M), API serving conditions, and open versus closed weights, and the open-weight models that collapse at 64–128k (Gemma 3 27b, Qwen 3 32b) are also the smallest. The observed pattern is consistent with a context-length-support or serving artifact rather than a scaling law, and the statement should be removed or supported with a proper analysis that controls for these factors.
minor comments (6)
  1. [Appendix A / §3 Data] Several items in the corpus are not complete novels but fragments of multi-volume serials (The Caxtons: Part 12, Godolphin: Volume 5, What Will He Do With It: Book 10, The Earl's Promise, A Novel: Volume 2), and the <32k bin contains children's chapter books; this stretches the 'novel' premise and the claimed 'long-range semantic dependencies often over 128k tokens,' and it weakens construct validity for the shorter bins.
  2. [Footnote 9] The evaluation metrics are underspecified: the semantic-similarity measure for summaries and storyworlds does not state which embedding model or similarity function is used, how the Jaccard and semantic components are combined, or how scores are normalized to [0, 1]; the 'random baselines (B) averaged over 1,440 random pairs' also lack a definition of what is paired. These details are needed for the released benchmark to be reproducible from the paper itself.
  3. [Figure 3] The third panel is labeled 'Network' rather than 'Storyworld,' the y-axis is labeled 'Accuracy' even though the underlying measures are similarity scores and error, and the caption's 'Timepoint' axis label is unclear; the figure should be reconciled with the terminology used in Table 2.
  4. [Footnote 1 / §1] The text says 'we make our data and code available here' but provides no URL; given that code/data release is a stated contribution, the link needs to appear in the paper.
  5. [§4, Table 2] Only four of the seven models have entries in the >128k column because the other three models' context windows are 128k tokens, yet the abstract's 'none of seven' claim spans that regime; the caption should state this explicitly, and the claim should be re-scoped to the four models actually tested beyond 128k.
  6. [§3 Evaluation / Time task] The narrative-time reference is built by summing per-chapter predictions in seconds while the whole-novel prompt asks for a single number and unit; the resulting absolute-relative-error score mixes arithmetic and calibration ability with narrative-time understanding, and this confound should be discussed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's self-referential reference is explicitly disclosed, and the stability metric is a direct measurement rather than a fitted parameter or imported uniqueness claim.

full rationale

The paper's central comparison is between a model's full-novel output and a reference constructed by concatenating that same model's chapter-level outputs (Section 3, Evaluation). This is self-referential, and the Limitations section explicitly discloses it: 'the TLDM benchmark compares novels only to their own short context performance.' No fitted parameter is fed back into a formula, no uniqueness theorem is imported from the authors' prior work, and no predicted quantity is equivalent to an input by construction. The stability score is the operational definition of the measurement, and the paper transparently labels the comparison as stability rather than claiming independent ground truth. Calling the measured quantity 'stable understanding' is an interpretive label, not a hidden derivation. Any dispute about whether specific models such as GPT-4.1, Llama 4 Scout, or Gemini 2.0 Flash are stable on summary and time tasks in Table 2 is a question of aggregation, thresholding, and task weighting—a correctness and construct-validity concern, not circularity. The self-referential reference does create risk for the term 'understanding,' but because the paper acknowledges this limitation and does not derive a result from a prior claim that already contains the result, the derivation chain is not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on evaluation-protocol assumptions rather than fitted parameters or invented entities. The absence of human ground truth means every score is conditional on the reference-construction equivalence, and the below-random score for Gemma 3 in the 64k-128k bin indicates metric noise.

assumptions (4)
  • domain assumption Concatenated chapter-level model outputs are a valid proxy for a novel-level gold response.
    Section 3, Evaluation: 'We do not have human-labeled ground truth, but instead compare individual model performance on short contexts to performance on long contexts.' The benchmark's stability metric depends on this equivalence.
  • domain assumption Semantic similarity between generated summaries tracks narrative equivalence.
    Section 3, Evaluation: the difference between predictions is computed 'using a similarity heuristic.' No human validation of the similarity metric is reported; Gemma 3's summary score of 0.16 at 64-128k falling below its random baseline of 0.25 suggests metric noise.
  • domain assumption Ten novels per length bin are representative of novels in that bin.
    Section 3, Data: 'randomly sample books... keep the first ten suitable texts' from public-domain fiction. The corpus skews to obscure 19th and early 20th century works, so representativeness for modern prose is untested.
  • domain assumption Storyworld and narrative-time tasks measure narrative understanding.
    Section 3, Tasks: the tasks are presented as probes of narrative understanding, but no evidence links them to human reading comprehension or to an established theory of narrative competence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels." pith.science (2026). https://pith.science/paper/EMBLEMST

@misc{pith2026250514925,
  author       = {Pith},
  title        = {Pith review of: Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMBLEMST}},
  note         = {Machine review of arXiv:2505.14925}
}
read the original abstract

Although the context length of large language models (LLMs) has increased to millions of tokens, evaluating their effectiveness beyond needle-in-a-haystack approaches has proven difficult. We argue that novels provide a case study of subtle, complicated structure and long-range semantic dependencies often over 128k tokens in length. Inspired by work on computational novel analysis, we release the Too Long, Didn't Model (TLDM) benchmark, which tests a model's ability to report plot summary, storyworld configuration, and elapsed narrative time. We find that none of seven tested frontier LLMs retain stable understanding beyond 64k tokens. Our results suggest language model developers must look beyond "lost in the middle" benchmarks when evaluating model performance in complex long-context scenarios. To aid in further development we release the TLDM benchmark together with reference code and data.

Figures

Figures reproduced from arXiv: 2505.14925 by the authors.

Figure 1
Figure 1. Token lengths of ten popular novels as tok [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The five treatments considered in our study [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Accuracy scores per task averaged over all models. Values for each window of interest are presented. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

  1. [2]

    Mrs. Brown, after preparing a mustard plas- ter for her supposedly ill husband in a Paris hotel, accidentally applies it to the throat of a sleeping stranger in the wrong room, panics at her mistake, considers—but ultimately avoids—confessing to the night porter, and returns distraught and chilled to her own room

    work page

  2. [3]

    Brown, desperate to escape a French hotel after mistakenly putting a mustard plaster on a stranger instead of her ailing husband Mr

    Mrs. Brown, desperate to escape a French hotel after mistakenly putting a mustard plaster on a stranger instead of her ailing husband Mr. Brown, anxiously concocts excuses to persuade her reluc- tant husband to leave early while fearing discovery by the night porter and the injured stranger

    work page

  3. [4]

    In Proceedings of the 41 St International Conference on Machine Learning, volume 235

    The Illusion of State in State-Space Models. In Proceedings of the 41 St International Conference on Machine Learning, volume 235. PMLR. William Merrill and Ashish Sabharwal. 2025. A Little Depth Goes a Long Way: The Expres- sive Power of Log-Depth Transformers. Preprint, arXiv:2503.03961. Meta. 2025. The Llama 4 Herd: The beginning of a new era of native...

    arXiv 2025

  4. [5]

    The Bat- tle of the Strong: A Romance of Two Kingdoms

    Needle threading: Can llms follow threads through near-million-scale haystacks? arXiv preprint arXiv:2411.05000. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. RoFormer: En- hanced Transformer with Rotary Position Embedding. Preprint, arXiv:2104.09864. Simeng Sun, Kalpesh Krishna, Andrew Mattarella- Micke, and Mohit Iyyer...

    arXiv 2023

  5. [8]

    Mrs. Brown, after mistakenly entering a stranger’s hotel room at night and inadvertently placing a mustard plaster on his throat instead of her husband’s, faces public embarrassment and sus- picion but ultimately explains the misunderstand- ing, allowing her and Mr. Brown to narrowly avoid greater trouble and continue their journey, despite lingering awkw...

    work page

  6. [9]

    Brown accidentally ap- plies a mustard plaster to a stranger, Mr

    A series of misunderstandings and mistaken identity occur when Mrs. Brown accidentally ap- plies a mustard plaster to a stranger, Mr. Jones, at a hotel, leading to awkwardness and eventual reconciliation when it is revealed that Mr. Jones is engaged to Mrs. Brown’s sister and all parties meet at Thompson Hall. Treatment 2 (whole input). 1. During a bitter...

    work page

  7. [10]

    Brown mistakenly applies the mustard poultice to a stranger, Mr

    In the dark, Mrs. Brown mistakenly applies the mustard poultice to a stranger, Mr. Jones, in- stead of her husband, and flees in panic rather than wake or help him

    work page

  8. [11]

    Brown invents excuses to hurry her husband’s departure despite guilt and fear of discovery

    Distraught and desperate to leave the hotel to avoid a confrontation, Mrs. Brown invents excuses to hurry her husband’s departure despite guilt and fear of discovery

    work page

Show all 17 references
  1. [12]

    Brown is confronted about the incident when the porter discovers her handkerchief in Mr

    Mrs. Brown is confronted about the incident when the porter discovers her handkerchief in Mr. Jones’s room, leading to an embarrassing explana- tion and apology to Mr. Jones, which allows them to continue their journey

  2. [13]

    Brown and her husband awkwardly discover that the stranger from Paris, Mr

    Upon arrival at Thompson Hall, Mrs. Brown and her husband awkwardly discover that the stranger from Paris, Mr. Jones, is her sister’s fi- ancé, leading to the revelation of her mistake and eventual forgiveness during the family’s Christmas celebration. Treatment 3 (separate us...

  3. [14]

    Brown be- comes lost in the hotel while seeking mustard, then mistakenly applies the plaster to a stranger’s throat instead of her husband’s

    In the middle of the night, Mrs. Brown be- comes lost in the hotel while seeking mustard, then mistakenly applies the plaster to a stranger’s throat instead of her husband’s

  4. [15]

    Brown desperately tries to avoid detection and persuade her husband that they must leave quickly for England

    Discovering her error and unable to face the consequences, Mrs. Brown desperately tries to avoid detection and persuade her husband that they must leave quickly for England

  5. [16]

    Brown to confess, after which her husband explains the accident to the injured stranger, Mr

    The Browns are confronted by hotel staff and the porter with evidence of the mishap, forcing Mrs. Brown to confess, after which her husband explains the accident to the injured stranger, Mr. Jones, allowing them to depart

  6. [17]

    Jones is the fiancé of Mrs

    Unexpectedly reunited at Thompson Hall, it is revealed that Mr. Jones is the fiancé of Mrs. Brown’s sister, Jane, and after an awkward Christ- mas gathering, Mrs. Brown’s mistake becomes a family anecdote, ultimately forgiven by all. Treatment 5 (shuffled chapters). Certainly!...

  7. [2023]

    In The Eleventh International Conference on Learning Representations

    Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024. Long-context LLMs Struggle with Long In-context Learn...

    2024 arXiv

  8. [2024]

    Preprint, arXiv:2401.18058

    LongAlign: A Recipe for Long Context Alignment of Large Language Models. Preprint, arXiv:2401.18058. Pei Chen, Hongye Jin, Cheng-Che Lee, Rulin Shao, Jingfeng Yang, Mingyu Zhao, Zhaoyu Zhang, Qin Lu, Kaiwen Men, Ning Xie, Huasheng Li, Bing Yin, Han Li, and Lingyun Wang. 2025. ...

    2025 arXiv

  9. [2025]

    Preprint, arXiv:2412.19437

    DeepSeek-V3 Technical Report. Preprint, arXiv:2412.19437. Sil Hamilton and Andrew Piper. 2023. Multihathi: A complete collection of multilingual prose fiction in the hathitrust digital library. Journal of Open Hu- manities Data. Michael S. Hart. 1971. Project Gutenberg. Cheng-...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.