Literary Evidence Retrieval via Long-Context Language Models

Katherine Thai; Mohit Iyyer

REVIEW 4 major objections 4 minor 13 references

Literary Evidence Retrieval via Long-Context Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper reports that a long-context language model given the full novel text can retrieve the exact quotation cited by literary criticism, outperforming a single human expert on the curated benchmark.

desk verdict Useful new benchmark, but the 'exceeds human' claim is a three-example difference from a single annotator and should not be stated as established. read the letter →

arxiv 2506.03090 v1 pith:UHHNVZX4 submitted 2025-06-03 cs.CL

Katherine Thai , Mohit Iyyer This is my paper

classification cs.CL

keywords literaryevidenceretrievallong-contextlanguagemodelsRELiCcriticismclosereadingquotationreasoningbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper repurposes the RELiC dataset into a long-context benchmark: a model receives the full text of a public-domain novel and an excerpt of literary criticism in which a quotation is masked, and must generate the missing quotation. On a human-verified subset of 292 examples, the authors report that the best closed-source reasoning model, Gemini Pro 2.5, retrieves the correct quotation 62.5% of the time on a 40-example human-evaluated fold, versus 55.0% for one human expert, while the best open-weight model reaches 29.1% on the full set. The result matters because it moves literary evidence retrieval from embedding-based search to full-text interpretive reasoning, and it quantifies a large gap between closed and open models. The authors also report that all models overgenerate and can miss subtle literary signals.

What carries the argument

The central object is the masked-citation retrieval task: a window of one to five consecutive sentences from the novel is removed from an excerpt of literary criticism, and the model must regenerate that window given the full novel text and the remaining criticism. The design forces joint global narrative reasoning and close textual reading. The evaluation machinery includes two prompt templates, Simple and Explanation, and fuzzy string matching against the ground-truth quotation, with thresholds selected after manual inspection. Two special folds, the 40-example human-evaluated set and the 39-example close-reading set, isolate the human comparison and the potential exploitation of lexical overlap.

What would settle it

Run the same 40-example fold with several independent literary scholars who have not seen the dataset-construction process, and compare their pooled accuracy with Gemini Pro 2.5's 62.5%; if the pooled experts match or exceed that rate, the claim of surpassing a human expert is not supported. A second check would remove quotations that appear verbatim in large public corpora and test whether the closed-model advantage persists, which would indicate whether memorization rather than reasoning drives the result.

Watch

Extended reading notes

Core claim

The central claim is that modern long-context language models, when prompted with an entire novel and a masked literary-analysis excerpt, can retrieve the cited quotation at a rate that exceeds a human expert's performance on the same task. The authors construct this task by taking RELiC's literary criticism excerpts, filtering to 292 high-quality examples, and evaluating twelve LLMs plus an embedding baseline. On the 40-example fold the human expert attempted, Gemini Pro 2.5 scores 62.5% with an explanation prompt, while the human scores 55.0%. On the full 292-example set, the best model reaches 64.7%, the best open-weight model, DeepSeek-R1, reaches 29.1%, and the embedding baseline reaches 4.5%. The paper interprets this as evidence that interpretive reasoning over the whole text, not just long-context capacity or lexical matching, is the decisive factor.

Load-bearing premise

The headline comparison assumes that the single author who curated, validated, and then attempted 40 examples is a representative human-expert baseline; if that baseline is unrepresentative, the claim that a model exceeds human expertise loses its footing even though the benchmark itself may remain sound.

Editorial extensions

If this is right

Closed-source reasoning models can serve as quotation finders for literary criticism, reducing an eight-hour manual search task to minute-scale API calls.
Open-weight models are not yet competitive: the best open model trails the best closed model by more than a factor of two, so long-context capacity alone is insufficient for interpretive retrieval.
All models, and the human annotator, tend to output longer windows than the ground truth, so evaluation and prompting must account for overgeneration.
On close-reading examples, where the criticism itself partially quotes the target, large models improve but the smallest models do not, indicating a capacity threshold for exploiting lexical overlap.
Models can select plausible alternative quotations that differ from the ground truth, which the paper suggests could help scholars explore multiple textual connections.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the reported human-expert comparison is a single-reader baseline, so the result establishes competence at one verifiable retrieval task rather than general expert-level literary interpretation.
Inference: a natural ranking-based extension would ask models to output several candidate windows and judge whether the ground truth falls in the ranked list, separating retrieval recall from overgeneration.
Inference: because the dataset is English and Western-canon, a cross-lingual extension with long-context models trained on other literary traditions would test whether the capability generalizes beyond the Anglophone canon.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Benchmark is worth having; the 'beats a human expert' headline is not. The long-context RELiC set is carefully built: 292 examples, explicit filters for leakage, location disclosure, and memorization, plus a human validation pass with a reported f1 of 89.8 on the filter itself. That is real work, and the release of data and code makes it reusable. The model-to-model comparisons are also informative, especially the open-weight gap and the close-reading fold. The overgeneration analysis is a nice touch.

The soft spot is exactly where the stress-test lands. The human-superiority claim rests on 40 examples from one annotator, who also curated and validated the dataset. 62.5% vs 55% is 25/40 vs 22/40; a three-example difference, well within chance. No confidence intervals, no significance test, no inter-annotator agreement. And the abstract says 50% while the body/table says 55%, which is a concrete error. The appendix confirms the fuzzy-match thresholds were selected after manual inspection, so the metric is to some degree tuned on the data. These issues are real, but they are fixable. The benchmark itself does not collapse; the claim that a closed model 'exceeds human expert performance' should be downgraded to 'is competitive with a single expert on a small sample' until better evidence appears.

I also note the Limitations section concedes the single-annotator point explicitly. That honesty counts in the paper's favor. The reader's conditional verdict seems right, and the stress-test note holds up. I would not block on the human baseline; I would send it to review with a request for a stronger human evaluation or a reframed claim. The curation methodology and the model comparison results justify referee time.

Referee Report

4 major / 4 minor

Summary. The paper repurposes the RELiC dataset to construct a long-context literary evidence retrieval benchmark in which a model receives a full public-domain novel and an excerpt of literary criticism with a masked quotation, and must generate the missing quotation. The authors curate 292 examples through automated filtering and manual validation, evaluate eight instruction-following LLMs plus an embedding baseline under two prompt variants, and introduce two small folds: a 40-example human-evaluated set and a 39-example close-reading set. The central results are that Gemini Pro 2.5 achieves 62.5% accuracy on the human-evaluated fold versus 55.0% for a human expert, that closed-source models substantially outperform open-weight models, and that all models tend to overgenerate relative to the ground-truth quotation length. The paper also releases the dataset and evaluation code.

Significance. If the benchmark is sound, it is a useful resource for studying long-context interpretive reasoning, and the curation pipeline is more transparent than many comparable efforts: the authors document filtering steps, provide manual validation statistics, and release code and data. The model-to-model comparisons on the full 292-example set are independent of the human baseline and show a clear gap between closed and open-weight models. However, the headline claim of surpassing human expert performance is not established by the evidence as presented: it rests on a three-example difference on 40 items annotated by the same author who curated and filtered the dataset, with no significance testing, no inter-annotator agreement, and conflicting numbers between the abstract and the body. These issues affect the paper's central claim rather than only its presentation, and they need to be addressed before the result can be taken at face value.

major comments (4)

[Section 4, Table 2] The claim that Gemini Pro 2.5 exceeds human expert performance is supported only by a 62.5% (25/40) versus 55.0% (22/40) difference on the /usr fold, which is a three-example gap. With n=40, the exact two-sided 95% binomial confidence intervals for these proportions overlap substantially (approximately [46%, 77%] and [38%, 71%]), and a two-proportion comparison is far from significant (p >> 0.05). No confidence intervals, significance tests, or effect-size measures are reported. As it stands, the observed difference is fully consistent with chance, so the sentence 'exceed human expert performance' should be removed or substantially qualified, or the human evaluation must be expanded and statistically analyzed.
[Abstract vs. Section 4/Table 2] The abstract reports '62.5% vs. 50% accuracy' for the human comparison, while Section 4 and Table 2 report the human expert accuracy as 55.0%. This is not a trivial typo: the headline comparison is the paper's main claim, and readers cannot tell whether the human baseline is 50% or 55%. The inconsistency must be resolved, and the final version should state one number with a clear description of which prompt condition and which examples it corresponds to.
[Section 3.1 and Limitations] The human baseline is not an independent expert evaluation. The same author who manually validated and filtered the dataset (Section 3.1) also attempted the 40 human-evaluation examples, and the Limitations section concedes that 'our human evaluation was limited in scope and scale, as it was conducted by a single annotator, one of the authors of this paper.' This creates selection and familiarity effects: the annotator knows the curation criteria and has already read the source novels, and no inter-annotator agreement is reported. At minimum, the manuscript should describe this as a pilot or illustrative human reference point rather than a validated human-expert baseline, and the 'exceeds human' framing should be adjusted accordingly.
[Section 3.2 and Appendix D] The automatic evaluation relies on fuzzy-match thresholds of 95 (source membership) and 90 (quote overlap), and Appendix D states that these 'thresholds were determined after manual inspection of outputs at varying thresholds.' Because all model accuracy numbers depend on these thresholds, the reported rankings could be sensitive to threshold choice, and selecting thresholds after seeing outputs risks overfitting the evaluation. The authors should report accuracy across a range of thresholds (e.g., 80, 85, 90, 95, 100) or otherwise justify that the conclusions are robust to reasonable threshold variation.

minor comments (4)

[Section 2.1] The phrase 'These sentences of make up the context' contains a typo ('of' should be removed).
[Section 2.1 and Table 2] The notation '/usr' for the human-evaluated fold and the adjacent text in Section 2.1 appear with missing spaces or formatting artifacts (e.g., 'the/usrHUMAN EVAL SET'), which makes the fold definitions hard to parse.
[Table 2] The column headers 'MODEL+ SIMPLEPROMPT' and 'MODEL+ EXPLANATIONPROMPT' lack spaces, and the table would be easier to read with the model names and prompt types separated clearly.
[Appendix B.1] The EZ2MEM filter identifies examples that a model can answer without the primary source, but the paper notes that such examples are not excluded from the final dataset. This is a reasonable design choice, but it should be discussed explicitly in the main text because it bears on data-contamination concerns for closed-source models.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity; the benchmark reuses an external dataset and evaluates models independently. The self-referential single-author human baseline is a validity limitation, not a circular derivation.

full rationale

This paper contains no analytic derivation chain whose output is equivalent to an input. The benchmark is built from the public RELiC dataset (Thai et al., 2022), and the 292-example subset is produced by LLM-aided filtering plus manual review; none of these steps computes the reported model accuracies. Model scores in Table 2 are measured against RELiC ground-truth quotations via fuzzy matching, independent of the filtering decisions. The only self-referential element is the human baseline: Section 3.1 states 'The author who manually validated high quality examples also attempted 40 examples from four previously-read primary source novels,' and the Limitations section concedes 'our human evaluation was limited in scope and scale, as it was conducted by a single annotator, one of the authors of this paper.' This overlap, together with the three-example margin (62.5% vs. 55.0%, while the abstract even reports 50%), weakens the 'exceeds human expert' claim as an external benchmark, but it does not make the model accuracy a fitted parameter or a logical consequence of the curation. Model-to-model comparisons are unaffected by this issue. No circular step can be exhibited from the text, so the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central numbers rest on thresholds, human curation judgments, and inherited RELiC labels rather than on a derived equation. The ledger records the fitted evaluation parameters and domain assumptions that a replication attempt would need to fix.

free parameters (5)

Fuzzy match threshold for source membership = 95
Appendix D: threshold 95 for checking the model response exists in the primary source; chosen after manual inspection of outputs at varying thresholds.
Fuzzy match threshold for quote overlap = 90
Appendix D: threshold 90 for overlap between the model response and the ground truth quotation; manually selected after inspecting outputs.
Leakage filter fuzzy match threshold = 95
Appendix B.1: context sentences are excluded if they fuzzy-match at threshold 95 with the primary source text.
Token generation limits = 800/1200 for simple/explanation prompts; 12000/25000 for reasoning models
Appendix C: generation limits were chosen to accommodate reasoning tokens and affect output length and overgeneration metrics.
Temperature settings = 0.0 except Qwen3 at 0.6
Appendix C: temperature was set to 0.0 for most models for determinism; Qwen3 models used the default 0.6.

assumptions (5)

domain assumption RELiC ground truth quotations are correct and canonical citations from the primary sources.
The benchmark inherits RELiC's ground truth labels without re-checking them against the original published criticism; Section 2.1.
domain assumption The 292 filtered examples are well-formed literary analysis with a uniquely identifiable quotation.
Section 2.1: one author manually verified 400 examples and marked 292; no inter-annotator agreement is reported.
ad hoc to paper Fuzzy string matching at thresholds 95 and 90 correctly approximates human judgment of quote correctness.
Appendix D: thresholds were manually determined after inspecting outputs and were not validated against independent human ratings.
domain assumption The single author's performance on 40 examples represents human expert performance.
Section 3.1: one annotator with a literature degree attempted 40 examples in about 8 hours; Limitations acknowledges the single-annotator limitation.
domain assumption Models do not rely on memorized quotations from training data despite contamination concerns.
Section 2.1: the EZ2MEM filter identified easily memorized quotes but examples were not excluded; the authors note data contamination as a concern.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Literary Evidence Retrieval via Long-Context Language Models." pith.science (2026). https://pith.science/paper/UHHNVZX4

@misc{pith2026250603090,
  author       = {Pith},
  title        = {Pith review of: Literary Evidence Retrieval via Long-Context Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UHHNVZX4}},
  note         = {Machine review of arXiv:2506.03090}
}

read the original abstract

How well do modern long-context language models understand literary fiction? We explore this question via the task of literary evidence retrieval, repurposing the RELiC dataset of That et al. (2022) to construct a benchmark where the entire text of a primary source (e.g., The Great Gatsby) is provided to an LLM alongside literary criticism with a missing quotation from that work. This setting, in which the model must generate the missing quotation, mirrors the human process of literary analysis by requiring models to perform both global narrative reasoning and close textual examination. We curate a high-quality subset of 292 examples through extensive filtering and human verification. Our experiments show that recent reasoning models, such as Gemini Pro 2.5 can exceed human expert performance (62.5% vs. 50% accuracy). In contrast, the best open-weight model achieves only 29.1% accuracy, highlighting a wide gap in interpretive reasoning between open and closed-weight models. Despite their speed and apparent accuracy, even the strongest models struggle with nuanced literary signals and overgeneration, signaling open challenges for applying LLMs to literary analysis. We release our dataset and evaluation code to encourage future work in this direction.

Figures

Figures reproduced from arXiv: 2506.03090 by the authors.

**Figure 2.** An example of long context RELiC where the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

[1]

CLEAN (GPT-4 O-MINI ): We asked the model to remove any OCR artifacts and to ensure the context and ground truth quotation flow seam- less and grammatically without changing any meaning. Additionally, we asked the model to remove any remaining in-line citations that revealed the page number of the ground truth quotation, which allowed us to preserve some ...

work page
[2]

LEAKAGE (heuristic): If the sentences in the context preceding or following the ground truth quotation were fuzzy-matches (thresh- old: 95) with any text from the primary source, the example was excluded from our dataset

work page
[3]

If it was not, we excluded it from our dataset

LIT ANALYSIS (GPT-4 O-MINI ): We asked the model to classify whether each RELiC in- stance was an example of literary analysis. If it was not, we excluded it from our dataset. See the prompt in Table 9

work page
[4]

If it did, we excluded it from our dataset

LOCATION (GPT-4 O-MINI ): We asked the model if the context revealed the location of the ground truth quotation. If it did, we excluded it from our dataset. See the prompt in Table 10

work page
[5]

FIRST SENT (heuristic): We identified cases where the ground truth quotation was the first sentence of its primary source novel. The intuition behind this filter (and the following Book Title Author Publication Year Token Count Word Count Brave New World Aldous Huxley 1932 91,472 65,278 What Maisie Knew Henry James 1897 124,544 95,988 Ethan Frome Edith Wh...

work page 1932
[6]

LAST SENT (heuristic): Similarly, we identi- fied cases where the ground truth quotation was the last sentence of its primary source novel

work page
[7]

OUTLIER (heuristic): We identified cases where the ground truth quotation was cited much more frequently than any of the other quotations from the same primary source novel

work page
[8]

prefix," and the part of the excerpt after the quotation is called the

EZ2MEM (GPT-4): We asked the model to per- form the RELiC task without providing the primary source text and identified cases where the model was able to correctly generate at least one sentence of the ground truth quota- tion. Note that we did not automatically exclude any of the examples identified by the last four filters. Though we did not use them in...

work page 2024

Show all 13 references

[11]

Artifacts include page numbers, chapter headings, footnotes, image captions, etc

Remove all textual artifacts from the OCR process by deleting them. Artifacts include page numbers, chapter headings, footnotes, image captions, etc
[12]

Correct the grammar, punctuation, and spelling of the prefix and suffix without altering the meaning so that the primary source quotation fits seamlessly and grammatically between the prefix and suffix
[13]

(my emphasis)

Remove all in-line citations following quotations, especially those that say things like "(my emphasis)." Respond with ONLY the cleaned prefix and suffix in the following format: <clean_prefix>CLEAN PREFIX </clean_prefix> <clean_suffix>CLEAN SUFFIX </clean_suffix> Table 8: Pro...
[2024]

In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 4471–4500, Miami, Florida, USA

BookWorm: A dataset for character descrip- tion and analysis. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 4471–4500, Miami, Florida, USA. Association for Computational Linguistics. Andrew Piper and Sunyam Bagga. 2024. Using large language m...

2024
[2025]

facilitate further research in this direction

Qwen2.5 Technical Report. arXiv preprint. ArXiv:2412.15115 [cs]. Jocelyn Shen, Joel Mire, Hae Won Park, Cynthia Breazeal, and Maarten Sap. 2024. HEART-felt narra- tives: Tracing empathy and narrative style in personal stories with LLMs. In Proceedings of the 2024 Con- ference ...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

CLEAN (GPT-4 O-MINI ): We asked the model to remove any OCR artifacts and to ensure the context and ground truth quotation flow seam- less and grammatically without changing any meaning. Additionally, we asked the model to remove any remaining in-line citations that revealed the page number of the ground truth quotation, which allowed us to preserve some ...

work page

[2] [2]

LEAKAGE (heuristic): If the sentences in the context preceding or following the ground truth quotation were fuzzy-matches (thresh- old: 95) with any text from the primary source, the example was excluded from our dataset

work page

[3] [3]

If it was not, we excluded it from our dataset

LIT ANALYSIS (GPT-4 O-MINI ): We asked the model to classify whether each RELiC in- stance was an example of literary analysis. If it was not, we excluded it from our dataset. See the prompt in Table 9

work page

[4] [4]

If it did, we excluded it from our dataset

LOCATION (GPT-4 O-MINI ): We asked the model if the context revealed the location of the ground truth quotation. If it did, we excluded it from our dataset. See the prompt in Table 10

work page

[5] [5]

FIRST SENT (heuristic): We identified cases where the ground truth quotation was the first sentence of its primary source novel. The intuition behind this filter (and the following Book Title Author Publication Year Token Count Word Count Brave New World Aldous Huxley 1932 91,472 65,278 What Maisie Knew Henry James 1897 124,544 95,988 Ethan Frome Edith Wh...

work page 1932

[6] [6]

LAST SENT (heuristic): Similarly, we identi- fied cases where the ground truth quotation was the last sentence of its primary source novel

work page

[7] [7]

OUTLIER (heuristic): We identified cases where the ground truth quotation was cited much more frequently than any of the other quotations from the same primary source novel

work page

[8] [8]

prefix," and the part of the excerpt after the quotation is called the

EZ2MEM (GPT-4): We asked the model to per- form the RELiC task without providing the primary source text and identified cases where the model was able to correctly generate at least one sentence of the ground truth quota- tion. Note that we did not automatically exclude any of the examples identified by the last four filters. Though we did not use them in...

work page 2024