Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HAMLET introduces an automated, hierarchy-aware benchmark for LLM comprehension of book-length texts, and its evaluation agrees with expert judgments more than 90% of the time.

desk verdict Useful new benchmark with a real human-validation study, but the recall numbers should be read as conditional on GPT-4o's own notion of important facts. read the letter →

arxiv 2508.19578 v1 pith:YTLF5DB7 submitted 2025-08-27 cs.CL cs.AI

Jiaqi Deng , Yuho Lee , Nicole Hee-Yeon Kim , Hyangsuk Min , Taewon Yun , Minjeong Ban , Kim Yul , Hwanjun Song This is my paper
classification cs.CLcs.AI
keywords long-contextevaluationkey-facttreequery-focusedsummarizationmulti-levelrecallfaithfulnesslost-in-the-middlebook-lengthcomprehensionLLMbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-context benchmarks usually ask for one holistic summary or short QA, so they miss whether a model can hold both the gist and the fine-grained facts of a book-length input. HAMLET instead evaluates LLMs on query-focused summaries at three levels of detail—root themes, branch-level supporting ideas, and leaf-level details—by extracting a key-fact tree from each 4K-token chunk of the book and checking the model's summary against that tree. The paper reports that this fully automated pipeline agrees with expert human judgments on recall and faithfulness over 90% of the time and cuts evaluation cost by up to 25 times compared with a prior fine-grained evaluator. Using the framework on sixteen novels, the paper finds that recall steadily drops from root to leaf, that the lost-in-the-middle effect is much sharper for leaf-level details, and that analytical queries are harder than narrative ones.

What carries the argument

The key-fact tree is the load-bearing object. For each non-overlapping 4K-token chunk of a novel, GPT-4o is prompted to produce two trees—one analytical (theme, characterization, symbolic pattern) and one narrative (events, causal progression)—with every fact filtered through faithfulness, objectivity, and significance checks. The tree plays a dual role: it generates a self-contained query used to elicit a summary from the tested LLM, and it serves as the reference against which that summary's recall and faithfulness are computed. This chunk-level anchoring is what makes the evaluation possible without rereading the whole book for each judgment.

What would settle it

Have expert readers independently list the facts they consider important at root, branch, and leaf levels for a sample of chunks. If GPT-4o's key-fact trees systematically omit facts those experts flag, or include facts experts call trivial, then the paper's measured leaf-level recall and lost-in-the-middle effects would be artifacts of the reference construction rather than properties of the tested LLMs.

Watch

Extended reading notes

Core claim

The central claim is that comprehension of book-length text can and should be measured hierarchically. HAMLET structures each book chunk into a key-fact tree—a root capturing the central theme, branches for supporting ideas or narrative arcs, and leaves for specific facts—and generates one query per tree, half analytical and half narrative. A test model then reads the entire book and writes a query-focused summary. Automatic evaluators, using the same key-fact trees, score recall as the fraction of key-facts at each level that appear in the summary, and faithfulness by checking each summary sentence against the matching chunk. The paper claims this is the first automated framework to evaluat

Load-bearing premise

The framework's ground truth is the set of key-facts that GPT-4o extracts from each 4K-token chunk, so every recall score presupposes that these trees capture what a careful human reader would consider the important content.

Editorial extensions

If this is right

  • If HAMLET's validation holds, long-context benchmark designers can replace expensive expert reading with automated key-fact evaluation for recall and faithfulness, at least for narrative prose.
  • The root-to-leaf recall gradient gives a concrete diagnostic: when a model fails a long-context task, the failure is likely to appear first in fine-grained leaf details rather than high-level gist.
  • Because each chunk carries its own key-fact tree, HAMLET's recall metric can localize which part of the book a model failed to use, making positional effects like lost-in-the-middle measurable at multiple abstraction levels.
  • The analytical/narrative split implies that long-context evaluations should report both perspectives separately, since aggregate summary scores mix two very different difficulties.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reference trees are single-model artifacts, so rankings among LLMs could shift if trees were built by a committee of models or by human experts; a testable extension is to measure how much leaf-level recall differences survive when the reference set changes.
  • The same chunk-tree design could be carried to non-fiction, scientific papers, or transcripts, where root-branch-leaf maps onto section headings, claims, and supporting evidence; the paper's limitations section notes this as future work.
  • The sharp leaf-level lost-in-the-middle effect suggests retrieval-augmented systems that feed only selected chunks to the model may mitigate exactly the failure mode HAMLET exposes, since leaf details are the ones most likely to be dropped from full-context reading.
  • Reasoning-specialized training appears to trade recall for faithfulness in this setup; an obvious next experiment is to test whether that trade-off persists when the reasoning model is given more output tokens or a retrieval step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces HAMLET, a fully automated benchmark for evaluating LLMs on book-length inputs. It segments 16 novels into 4K-token chunks, uses GPT-4o to build a three-level key-fact hierarchy (root/branch/leaf) for each chunk, validates those trees with a GPT-4o filter, and generates one query per tree. LLMs then produce query-focused summaries over the full book, and HAMLET scores recall and faithfulness using GPT-4o-based evaluators anchored to the chunk key-fact trees. A human study with three expert annotators reports over 90% agreement for automated evaluation and a 25x cost reduction. Main benchmark results show that leaf-level recall is low, lost-in-the-middle is more pronounced at leaf level, analytical queries are harder than narrative ones, and proprietary/larger models outperform open-source/smaller models.

Significance. If the reference-validity concerns can be addressed, HAMLET would be a valuable contribution: the key-fact hierarchy is a natural decomposition of book-length comprehension, the chunk-anchored evaluation strategy is novel and cost-effective, and the framework is fully automated with public code and data. The expert validation, while partial, is stronger than typical LLM-as-judge reports, and the cost/agreement numbers are concrete and checkable. The main risk is that the recall component is built on a reference representation whose completeness is not validated, so downstream conclusions about leaf-level deficits and model ordering are conditional on GPT-4o's fact-selection behavior.

major comments (3)
  1. [§3.1.2, Eq. (1), §4 Table 2] The recall denominator in Eq. (1) is the set of GPT-4o-extracted, GPT-4o-filtered key-facts. The human validation in §4 checks only whether the three-dimensional filter's PASS/FAIL decisions are correct on sampled key-facts (Table 2); it does not evaluate whether the key-fact tree omits information that expert readers would consider important. The generation prompt (Table 15) instructs the model not to omit significant information, but no completeness audit is reported. Consequently, the reported leaf-level recall ranges (0.113-0.265 analytical, 0.153-0.383 narrative) and the sharper lost-in-the-middle at leaf level could be artifacts of GPT-4o's own notion of importance rather than properties of the evaluated LLMs. I would require a completeness study: independently have experts list important facts from a sample of chunks, compute recall of the key-fact trees against those lists, and r
  2. [§3.1.2-§3.3.2, Tables 3 and 11] There is a self-referential loop: GPT-4o creates the reference key-fact trees, validates them, generates the queries, acts as scorer, and is itself one of the evaluated models (Table 11). The expert agreement in Table 3 shows that the labels produced by the automated evaluator agree with experts, but it does not break the reference-construction circularity. A model whose summary style matches GPT-4o's likely has an advantage in recall/faithfulness scores. I ask for a stability test: rebuild the benchmark on a subset using at least one different backbone (e.g., Claude or Llama-3.1-405B) for tree construction and scoring, and demonstrate that model rankings and lost-in-the-middle patterns are preserved. Alternatively, provide expert-built reference trees for a subset and show rankings are unchanged.
  3. [§5.1, Figure 3, Tables 4-7] All headline findings are reported as point estimates without confidence intervals or significance tests. For example, Table 4 reports gaps such as 0.025 vs 0.048 for GPT-4o-Mini leaf-level analytical recall, and Figure 3 shows differences between models on the order of 0.01-0.05. Given the data have a hierarchical structure (queries nested in chunks, chunks in books), these differences may not be reliable. Report per-query standard errors, bootstrap confidence intervals, or mixed-effects models; or explicitly label the comparisons as exploratory. This is needed to support claims like 'consistent performance gaps' and 'sharper lost-in-the-middle at leaf level.'
minor comments (5)
  1. [§3.1.2 and Table 2] The key-fact total is reported as 22,333 in §3.1.2 but 23,333 in Table 2 as the sampling universe. Please reconcile these numbers and give the exact count with breakdown by level and book.
  2. [Table 9] Header 'Vadility' should be 'Validity'.
  3. [Table 13] Model names 'GTP-4o' appear twice; should be 'GPT-4o'.
  4. [§3.3.2] Typo: 'faithfulnes' should be 'faithfulness'.
  5. [§4] The sentence about sample size ('over 98% confidence level with ±5% margin of error') should be moved to Appendix F and should specify the calculation for each component; as written, it is unclear which sample size is meant.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's reference and evaluation are human-validated, and reported recall/faithfulness scores are measurements, not fitted predictions.

full rationale

HAMLET's derivation chain is: source chunks -> GPT-4o key-fact trees -> GPT-4o queries -> model summaries -> GPT-4o key-fact alignment / fact-verification against the reference trees. Eq. (1) defines recall as the proportion of reference key-facts present in the summary; the reference key-facts are extracted from the source chunks, not from the summaries being scored, so the metric is not defined in terms of the target output. The human study (Section 4, Tables 2-3) validates the filter's PASS/FAIL decisions and the key-fact alignment/fact-checking labels against expert annotations, providing an external anchor. No parameter is fitted to a subset and then reported as a prediction; the scores in Tables 4-7 are empirical aggregates. Self-citations to FineSurE (Song et al., 2024) and UniSumEval (Lee et al., 2024) describe the adaptation of an evaluation framework and are not load-bearing for the paper's central claims. The most serious concern is a validity limitation rather than circularity: the key-fact trees are generated and filtered by GPT-4o, and completeness of the trees is not checked by the expert study (Table 2 only checks filter judgments), so leaf-level recall deficits could partly reflect GPT-4o's fact selection. The paper even prompts 'Do not omit any significant information' but reports no completeness verification. There is also an internal numeric inconsistency (22,333 key-facts in Section 3.1.2 vs. 23,333 in Table 2). These are reliability issues, not instances where an equation reduces to its inputs; the evaluation pipeline is externally anchored by expert labels for the sampled judgments. Therefore no circular step meeting the quoted-equation threshold is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical or conceptual entities in the sense of the graviton problem; it is a data structure and a scoring metric. The main hidden choices are the chunk size and the reliance on GPT-4o as both reference generator and evaluator.

free parameters (1)
  • chunk size = 4K tokens
    Chosen by hand based on LLM-as-judge comparison across 1K, 2K, 4K, and 8K chunks (Appendix B, Table 9). The choice affects how key-fact trees are built and all downstream scores.
assumptions (3)
  • domain assumption Expert human judgments are the gold standard for comprehension and summary quality.
    Used throughout Section 4 as the ground truth against which the automated pipeline is validated. The paper does not question this premise.
  • domain assumption Key-fact trees built from independent 4K chunks faithfully represent the important information of the book for comprehension evaluation.
    The recall metric counts whether model summaries contain these tree key-facts; if trees omit or misrepresent crucial facts, the metric is biased. Appendix B validates chunk quality with an LLM judge, but the final key-fact trees are only partly human-checked.
  • domain assumption GPT-4o, the model used to create trees, queries, and evaluate summaries, is a reliable proxy for human evaluation after calibration.
    GPT-4o is used at every stage of pipeline construction and evaluation. The human study calibrates it on samples, but the assumption that it generalizes to all test instances is an untested domain assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts." pith.science (2026). https://pith.science/paper/YTLF5DB7

@misc{pith2026250819578,
  author       = {Pith},
  title        = {Pith review of: Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTLF5DB7}},
  note         = {Machine review of arXiv:2508.19578}
}
read the original abstract

We introduce HAMLET, a holistic and automated framework for evaluating the long-context comprehension of large language models (LLMs). HAMLET structures source texts into a three-level key-fact hierarchy at root-, branch-, and leaf-levels, and employs query-focused summarization to evaluate how well models recall and faithfully represent information at each level. To validate the reliability of our fully automated pipeline, we conduct a systematic human study, showing that our automatic evaluation achieves over 90% agreement with expert human judgments, while reducing the cost by up to 25 times. HAMLET reveals that LLMs struggle with fine-grained comprehension, especially at the leaf level, and are sensitive to positional effects like the lost-in-the-middle. Analytical queries pose greater challenges than narrative ones, and consistent performance gaps emerge between open-source and proprietary models, as well as across model scales. Our code and dataset are publicly available at https://github.com/DISL-Lab/HAMLET.

Figures

Figures reproduced from arXiv: 2508.19578 by the authors.

Figure 1
Figure 1. Overview of HAMLET: HAMLET constructs queries from a key-fact hierarchy (root, branch, leaf) per chunk, generates summaries using the full book, and evaluates recall and faithfulness using LLM-based evaluators. LLM’s ability to extract information across varying levels of abstraction, categorized into two perspec￾tives: analytical, which focuses on deeper meaning and thematic interpretation, and narrative, which emp… view at source ↗
Figure 2
Figure 2. Example of an analytical key-fact tree ex [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Recall scores of six LLMs across four key-fact levels, based on query chunk locations for lost-in-the-middle [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    On a new extended needle-in-a-haystack benchmark, explicit anti-hallucination prompts and dispersed fact placement cause some long-context LLMs to over-refuse or collapse in accuracy, while others remain robust.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [2]

    Evaluating very long-term conversational memory of llm agents. In ACL. Raymond A Mar, Jingyuan Li, Anh TP Nguyen, and Cindy P Ta. 2021. Memory and comprehension of narrative versus expository texts: A meta-analysis. Psychonomic Bulletin & Review, 28:732–749. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. FineSurE: Fine-grained ...

    arXiv 2021

  2. [6]

    Focus on analyzing what is explicitly stated, supported, or implied within reasonable bounds, without adding subjective opinions or unsupported inferences

    work page

  3. [8]

    All key-facts should NEVER be based on over-interpretation or logical leaps beyond the information provided in the text

    work page

  4. [11]

    Output format: - Provide your answer in JSON format

    Make each key-fact as concise as possible, ensuring that each contain at most 2-3 entities. Output format: - Provide your answer in JSON format. - The answer should ONLY be a dictionary with the valid JSON format as follows: <Tree> - Include only the tree dictionary in the answer. The excerpt: {excerpt} Table 15: Analytical key-fact tree generation prompt...

    work page

  5. [12]

    Each key-fact is NOT a statement of theme or topic of the text, but a specific piece of information that can be directly extracted from the text

    work page

  6. [13]

    Do not omit any significant information from the text

    work page

  7. [14]

    Ensure clear relationships between roots, branches, and leaves

    work page

  8. [15]

    All key-facts must be directly supported by the text

    work page

Show all 17 references
  1. [16]

    Create as many roots, branches and leaves as needed to fully capture the text’s key-facts

  2. [17]

    Focus on how the story progresses from beginning to end, including any critical pivots or climaxes

  3. [18]

    the protagonist

    NEVER use pronouns, such as he, she, it, that, or "the protagonist". ALW AYS USE PROPER NOUNS

  4. [19]

    sentence

    Make each key-fact as concise as possible, ensuring that each contain at most 2-3 entities. Output format: - Provide your answer in JSON format. - The answer should ONLY be a dictionary with the valid JSON format as follows: <Tree> - Include only the tree dictionary in the ans...

  5. [20]

    • Less than 25 MFs → Score 1-2 • 25 - 35 MFs and at least a two-level hierarchy → Score 3-4 • More than 35 MFs and a well-defined multi-level hierarchy → Score 5

    HKF_validity — Validity of Hierarchical Key-Fact Tree Extraction Count the MFs and judge whether they can be arranged into a clear multi-level tree (root → branches → leaves). • Less than 25 MFs → Score 1-2 • 25 - 35 MFs and at least a two-level hierarchy → Score 3-4 • More th...

  6. [21]

    1-2 / 1" range • 3 points= A = YES but exactly one of B-D in the

    Content_coherence - Evaluate the chunk using the following four signals: • A. Structural completeness (beginning → middle → end) : YES / NO • B. Abrupt transitions (N_abrupt) : 0, 1-2, ≥3 • C. Unresolved references or unfinished plotlines : 0, 1, ≥2 • D. Logical/temporal/spati...

  7. [22]

    HKF_validity

    Cross_content_reasoning — Support for Cross-Content Reasoning Count the ICs that enable reasoning across different parts of the chunk. • Less than 8 ICs → Score 1-2 • 8 - 16 ICs → Score 3-4 • More than 16 ICs → Score 5 OUTPUT FORMAT: { "HKF_validity": <score from 1 to 5>, "Con...

  8. [2023]

    arXiv preprint arXiv:2303.04048

    Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048. Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. 2023. Less is more for long document summary evaluation by llms. arXiv preprint arXiv:2309.07382. Ruochen Xu, ...

    2023 arXiv

  9. [2024]

    Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In ICLR. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: extending llm context window beyond 2 million tokens. In ...

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.