{"id":"69714a14-e541-48a1-87e3-706ed4b349aa","arxiv_id":"2506.03090","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new 292-example benchmark shows the best closed-source LLM retrieves literary quotations from full novels better than one human expert, while open-weight models lag sharply.","lead":"This paper turns a database of literary criticism into a test where an AI reads an entire novel and must find the exact quote a critic cites. The best closed model beat one human expert on a small set, while open-weight models fell far behind.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline human-superiority claim rests on 3 examples: 25/40 (62.5%) vs 22/40 (55%) from a single annotator, with no confidence interval or significance test; this comparison is not established.","rationale":"The paper's most striking and load-bearing claim is that a closed-source reasoning LLM exceeds human expert performance at literary evidence retrieval. That claim is anchored entirely in a 40-example comparison where the model scored 25/40 and a single human annotator scored 22/40. The reader's weakest-assumption analysis correctly identifies the human baseline as the fragile point, and the internal abstract/body discrepancy on the human score (50% vs 55%) strengthens that concern. Because the observed gap is three examples and no statistical test or confidence interval is supplied, the superiority claim is not established even if the benchmark is sound. The rest of the paper—dataset curation, open/closed model gap, overgeneration analysis, and released code—does not depend on this specific comparison and remains valuable, so the appropriate outcome is the same conditional verdict: accept the benchmark contribution while requiring stronger human-baseline evidence before endorsing the human-superiority conclusion. No ad hominem is intended; a single-annotator baseline is a methodological limitation, not a sign of bad faith.","tokens_in":11001,"tokens_out":2507,"duration_ms":31189,"concrete_test":"Run a pre-registered human replication on the same 40-item /usr fold with at least two additional expert annotators (English-literature background, blind to model outputs and to the original annotator's choices, with the same searchable-text tools). Report each annotator's accuracy, exact binomial 95% confidence intervals, and pairwise McNemar tests against Gemini Pro 2.5. If any human accuracy is ≥62.5%, or if Gemini's 62.5% falls inside the pooled human CI, revise the claim from 'exceeds human expert performance' to 'performs comparably to one human annotator on a small sample'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4 / Table 2, that Gemini Pro 2.5 exceeds a human expert (62.5% vs 55.0%), depends entirely on the /usr fold of 40 examples and is a three-example difference. The human baseline is one author, who also curated and validated the dataset (Section 3.1; Limitations), making the comparison vulnerable to selection and familiarity effects. No inter-annotator agreement, confidence intervals, or significance tests are reported. With n=40, an exact two-sided 95% binomial CI for 62.5% is roughly [46%, 77%], and for 55% roughly [38%, 71%]; the observed difference is fully consistent with chance. Additionally, the abstract reports 50% human accuracy while the body and Table 2 report 55%, so the headline comparison is internally inconsistent. The benchmark itself may be useful, but the specific claim of surpassing human expert performance is unsupported by the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper repurposes the RELiC dataset to construct a long-context literary evidence retrieval benchmark in which a model receives a full public-domain novel and an excerpt of literary criticism with a masked quotation, and must generate the missing quotation. The authors curate 292 examples through automated filtering and manual validation, evaluate eight instruction-following LLMs plus an embedding baseline under two prompt variants, and introduce two small folds: a 40-example human-evaluated set and a 39-example close-reading set. The central results are that Gemini Pro 2.5 achieves 62.5% accuracy on the human-evaluated fold versus 55.0% for a human expert, that closed-source models substantially outperform open-weight models, and that all models tend to overgenerate relative to the ground-truth quotation length. The paper also releases the dataset and evaluation code.","tokens_in":11387,"tokens_out":3096,"duration_ms":34559,"significance":"If the benchmark is sound, it is a useful resource for studying long-context interpretive reasoning, and the curation pipeline is more transparent than many comparable efforts: the authors document filtering steps, provide manual validation statistics, and release code and data. The model-to-model comparisons on the full 292-example set are independent of the human baseline and show a clear gap between closed and open-weight models. However, the headline claim of surpassing human expert performance is not established by the evidence as presented: it rests on a three-example difference on 40 items annotated by the same author who curated and filtered the dataset, with no significance testing, no inter-annotator agreement, and conflicting numbers between the abstract and the body. These issues affect the paper's central claim rather than only its presentation, and they need to be addressed before the result can be taken at face value.","major_comments":[{"comment":"The claim that Gemini Pro 2.5 exceeds human expert performance is supported only by a 62.5% (25/40) versus 55.0% (22/40) difference on the /usr fold, which is a three-example gap. With n=40, the exact two-sided 95% binomial confidence intervals for these proportions overlap substantially (approximately [46%, 77%] and [38%, 71%]), and a two-proportion comparison is far from significant (p >> 0.05). No confidence intervals, significance tests, or effect-size measures are reported. As it stands, the observed difference is fully consistent with chance, so the sentence 'exceed human expert performance' should be removed or substantially qualified, or the human evaluation must be expanded and statistically analyzed.","section":"Section 4, Table 2"},{"comment":"The abstract reports '62.5% vs. 50% accuracy' for the human comparison, while Section 4 and Table 2 report the human expert accuracy as 55.0%. This is not a trivial typo: the headline comparison is the paper's main claim, and readers cannot tell whether the human baseline is 50% or 55%. The inconsistency must be resolved, and the final version should state one number with a clear description of which prompt condition and which examples it corresponds to.","section":"Abstract vs. Section 4/Table 2"},{"comment":"The human baseline is not an independent expert evaluation. The same author who manually validated and filtered the dataset (Section 3.1) also attempted the 40 human-evaluation examples, and the Limitations section concedes that 'our human evaluation was limited in scope and scale, as it was conducted by a single annotator, one of the authors of this paper.' This creates selection and familiarity effects: the annotator knows the curation criteria and has already read the source novels, and no inter-annotator agreement is reported. At minimum, the manuscript should describe this as a pilot or illustrative human reference point rather than a validated human-expert baseline, and the 'exceeds human' framing should be adjusted accordingly.","section":"Section 3.1 and Limitations"},{"comment":"The automatic evaluation relies on fuzzy-match thresholds of 95 (source membership) and 90 (quote overlap), and Appendix D states that these 'thresholds were determined after manual inspection of outputs at varying thresholds.' Because all model accuracy numbers depend on these thresholds, the reported rankings could be sensitive to threshold choice, and selecting thresholds after seeing outputs risks overfitting the evaluation. The authors should report accuracy across a range of thresholds (e.g., 80, 85, 90, 95, 100) or otherwise justify that the conclusions are robust to reasonable threshold variation.","section":"Section 3.2 and Appendix D"}],"minor_comments":[{"comment":"The phrase 'These sentences of make up the context' contains a typo ('of' should be removed).","section":"Section 2.1"},{"comment":"The notation '/usr' for the human-evaluated fold and the adjacent text in Section 2.1 appear with missing spaces or formatting artifacts (e.g., 'the/usrHUMAN EVAL SET'), which makes the fold definitions hard to parse.","section":"Section 2.1 and Table 2"},{"comment":"The column headers 'MODEL+ SIMPLEPROMPT' and 'MODEL+ EXPLANATIONPROMPT' lack spaces, and the table would be easier to read with the model names and prompt types separated clearly.","section":"Table 2"},{"comment":"The EZ2MEM filter identifies examples that a model can answer without the primary source, but the paper notes that such examples are not excluded from the final dataset. This is a reasonable design choice, but it should be discussed explicitly in the main text because it bears on data-contamination concerns for closed-source models.","section":"Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim of human-level or superhuman performance is currently overstated relative to the evidence, and the author-as-annotator issue is significant enough that I would ask the editor to ensure the revised version either provides proper statistical support or reframes the claim as a pilot result. The benchmark and model comparison portions are otherwise valuable and could be published after the human-comparison issue is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Benchmark is worth having; the 'beats a human expert' headline is not. The long-context RELiC set is carefully built: 292 examples, explicit filters for leakage, location disclosure, and memorization, plus a human validation pass with a reported f1 of 89.8 on the filter itself. That is real work, and the release of data and code makes it reusable. The model-to-model comparisons are also informative, especially the open-weight gap and the close-reading fold. The overgeneration analysis is a nice touch.\n\nThe soft spot is exactly where the stress-test lands. The human-superiority claim rests on 40 examples from one annotator, who also curated and validated the dataset. 62.5% vs 55% is 25/40 vs 22/40; a three-example difference, well within chance. No confidence intervals, no significance test, no inter-annotator agreement. And the abstract says 50% while the body/table says 55%, which is a concrete error. The appendix confirms the fuzzy-match thresholds were selected after manual inspection, so the metric is to some degree tuned on the data. These issues are real, but they are fixable. The benchmark itself does not collapse; the claim that a closed model 'exceeds human expert performance' should be downgraded to 'is competitive with a single expert on a small sample' until better evidence appears.\n\nI also note the Limitations section concedes the single-annotator point explicitly. That honesty counts in the paper's favor. The reader's conditional verdict seems right, and the stress-test note holds up. I would not block on the human baseline; I would send it to review with a request for a stronger human evaluation or a reframed claim. The curation methodology and the model comparison results justify referee time.","headline":"Useful new benchmark, but the 'exceeds human' claim is a three-example difference from a single annotator and should not be stated as established.","tokens_in":11729,"tokens_out":1819,"would_cite":true,"duration_ms":21221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that a long-context language model given the full novel text can retrieve the exact quotation cited by literary criticism, outperforming a single human expert on the curated benchmark.","keywords":["literary evidence retrieval","long-context language models","RELiC","literary criticism","close reading","quotation retrieval","reasoning models","benchmark"],"falsifier":"Run the same 40-example fold with several independent literary scholars who have not seen the dataset-construction process, and compare their pooled accuracy with Gemini Pro 2.5's 62.5%; if the pooled experts match or exceed that rate, the claim of surpassing a human expert is not supported. A second check would remove quotations that appear verbatim in large public corpora and test whether the closed-model advantage persists, which would indicate whether memorization rather than reasoning drives the result.","tokens_in":10816,"feed_emoji":"📚","tokens_out":6020,"duration_ms":62572,"temperature":0.7,"pith_summary":"The paper repurposes the RELiC dataset into a long-context benchmark: a model receives the full text of a public-domain novel and an excerpt of literary criticism in which a quotation is masked, and must generate the missing quotation. On a human-verified subset of 292 examples, the authors report that the best closed-source reasoning model, Gemini Pro 2.5, retrieves the correct quotation 62.5% of the time on a 40-example human-evaluated fold, versus 55.0% for one human expert, while the best open-weight model reaches 29.1% on the full set. The result matters because it moves literary evidence retrieval from embedding-based search to full-text interpretive reasoning, and it quantifies a large gap between closed and open models. The authors also report that all models overgenerate and can miss subtle literary signals.","feed_headline":"A long-context model beats one human expert at finding cited quotes","feed_subtitle":"On a curated benchmark, Gemini Pro 2.5 retrieves the right quotation 62.5% of the time versus 55% for the human expert.","key_machinery":"The central object is the masked-citation retrieval task: a window of one to five consecutive sentences from the novel is removed from an excerpt of literary criticism, and the model must regenerate that window given the full novel text and the remaining criticism. The design forces joint global narrative reasoning and close textual reading. The evaluation machinery includes two prompt templates, Simple and Explanation, and fuzzy string matching against the ground-truth quotation, with thresholds selected after manual inspection. Two special folds, the 40-example human-evaluated set and the 39-example close-reading set, isolate the human comparison and the potential exploitation of lexical overlap.","core_discovery":"The central claim is that modern long-context language models, when prompted with an entire novel and a masked literary-analysis excerpt, can retrieve the cited quotation at a rate that exceeds a human expert's performance on the same task. The authors construct this task by taking RELiC's literary criticism excerpts, filtering to 292 high-quality examples, and evaluating twelve LLMs plus an embedding baseline. On the 40-example fold the human expert attempted, Gemini Pro 2.5 scores 62.5% with an explanation prompt, while the human scores 55.0%. On the full 292-example set, the best model reaches 64.7%, the best open-weight model, DeepSeek-R1, reaches 29.1%, and the embedding baseline reaches 4.5%. The paper interprets this as evidence that interpretive reasoning over the whole text, not just long-context capacity or lexical matching, is the decisive factor.","pith_inferences":["Inference: the reported human-expert comparison is a single-reader baseline, so the result establishes competence at one verifiable retrieval task rather than general expert-level literary interpretation.","Inference: a natural ranking-based extension would ask models to output several candidate windows and judge whether the ground truth falls in the ranked list, separating retrieval recall from overgeneration.","Inference: because the dataset is English and Western-canon, a cross-lingual extension with long-context models trained on other literary traditions would test whether the capability generalizes beyond the Anglophone canon."],"forward_implications":["Closed-source reasoning models can serve as quotation finders for literary criticism, reducing an eight-hour manual search task to minute-scale API calls.","Open-weight models are not yet competitive: the best open model trails the best closed model by more than a factor of two, so long-context capacity alone is insufficient for interpretive retrieval.","All models, and the human annotator, tend to output longer windows than the ground truth, so evaluation and prompting must account for overgeneration.","On close-reading examples, where the criticism itself partially quotes the target, large models improve but the smallest models do not, indicating a capacity threshold for exploiting lexical overlap.","Models can select plausible alternative quotations that differ from the ground truth, which the paper suggests could help scholars explore multiple textual connections."],"supporting_citations":[{"why":"Supplies the original RELiC dataset of literary-criticism excerpts with quotations from public-domain novels; it is the source material repurposed into the long-context benchmark.","marker":"Thai et al., 2022"},{"why":"Establishes the million-token long-context capability that motivates the task framing and serves as the Gemini 1.5 baseline.","marker":"Gemini Team, 2024"},{"why":"Identifies Gemini 2.5 Pro, the model that achieves the best reported accuracy in the evaluation.","marker":"Google, 2025"},{"why":"Provides the NOCHA long-context evaluation prompts that the Simple and Explanation prompts are adapted from.","marker":"Karpinska et al., 2024"},{"why":"Documents GPT-4O, used both as a filtering model during dataset construction and as a baseline model in the evaluation.","marker":"OpenAI et al., 2024"},{"why":"The cited DeepSeek technical report is the reference behind the DeepSeek-R1 open-weight model, which sets the best open-weight accuracy in the comparison.","marker":"DeepSeek-AI et al., 2025"}],"fun_headline_variants":["Gemini Pro 2.5 beats a human critic at literary quote recall","Long-context LLM outperforms human expert on literary evidence","AI finds missing book quotes better than a human scholar","Closed model tops human on literary quote retrieval task","Model exceeds human accuracy in literary quote hunting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that the single author who curated, validated, and then attempted 40 examples is a representative human-expert baseline; if that baseline is unrepresentative, the claim that a model exceeds human expertise loses its footing even though the benchmark itself may remain sound.","fun_headline_variants_meta":{"raw":{"variants":["Gemini Pro 2.5 beats a human critic at literary quote recall","Long-context LLM outperforms human expert on literary evidence","AI finds missing book quotes better than a human scholar","Closed model tops human on literary quote retrieval task","Model exceeds human accuracy in literary quote hunting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0011,"raw_usage":{"total_tokens":4594,"prompt_tokens":954,"completion_tokens":3640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":3560}},"tokens_in":570,"tokens_out":3640,"duration_ms":28553,"temperature":1.0,"reasoning_tokens":3560,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:08:31.823254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 40-example fold with several independent literary scholars who have not seen the dataset-construction process, and compare their pooled accuracy with Gemini Pro 2.5's 62.5%; if the pooled experts match or exceed that rate, the claim of surpassing a human expert is not supported. A second check would remove quotations that appear verbatim in large public corpora and test whether the closed-model advantage persists, which would indicate whether memorization rather than reasoning drives the result.","supporting_citations":[],"review_version":1}