Pith. sign in

REVIEW 4 major objections 7 minor 39 references

NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NovelHopQA, a new benchmark built from 83 public-domain novels, shows that language models lose accuracy as both reasoning depth (1–4 hops) and context length (64k–128k tokens) grow, even for frontier models.

desk verdict Useful diagnostic benchmark with a real validation gap: labels are checked on short golden chains, not on the 64k-128k windows that carry the headline claims. read the letter →

arxiv 2506.02000 v2 pith:4ITNTIQX submitted 2025-05-20 cs.CL

Abhay Gupta , Michael Lu , Kevin Zhu , Sean O'Brien , Vasu Sharma This is my paper
classification cs.CL
keywords multi-hopreasoninglong-contextquestionansweringbenchmarknarrativecomprehensionretrieval-augmentedgenerationfailuremodeanalysislanguagemodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NovelHopQA is a new benchmark for question answering over long narrative contexts. It jointly varies two things that earlier benchmarks kept separate: reasoning depth (1 to 4 hops) and context length (64k, 96k, or 128k tokens). Built from 83 public-domain novels, it contains 4,000 human-validated QA pairs whose questions are generated by chaining together paragraphs connected through related keywords. Evaluating seven frontier models, the paper finds consistent accuracy drops as hop count increases and as context length grows, with even the strongest models falling below 80% on 4-hop questions at 128k tokens. The authors argue this shows that larger context windows alone do not buy robust multi-hop reasoning, and position the benchmark as a controlled diagnostic for long-context failure modes.

What carries the argument

The pipeline that carries the argument is keyword-guided paragraph chaining with incremental QA regeneration. For each example, an anchor keyword selects the first paragraph; at each subsequent hop a related keyword is extracted and a new paragraph containing both keywords is appended; then GPT-4o regenerates the question-answer pair over the full growing context so the final H-hop item integrates all H paragraphs. The paper also applies golden-context filtering (dropping questions any model cannot answer on the concatenated golden paragraphs) and human validation of alignment and hop depth, then evaluates seven models with chain-of-thought prompting on 64k–128k windows.

What would settle it

A concrete check would be to take a sample of NovelHopQA questions, have human annotators judge the required hop depth on the full 128k windows (not just the golden paragraphs), and see whether the majority of items still require exactly the labeled number of hops. If many items are judged to require fewer or more hops at length, the benchmark's central comparison collapses.

Watch

Extended reading notes

Core claim

The central claim is that current LLMs, including reasoning-oriented frontier models, degrade reliability on multi-hop QA when both the number of reasoning steps and the narrative context length increase. Using NovelHopQA, the paper shows average accuracy falls roughly 12 points from 1-hop to 4-hop at 64k, and most models lose another 4–6 points from 64k to 128k on 1-hop questions. RAG evaluations with top-k=7 retrieved chunks drop to around 50%, indicating retrieval helps but does not close the gap. The paper also isolates four failure modes: missed final-hop integration, entity confusion, incomplete evidence combination, and contextual drift.

Load-bearing premise

The load-bearing assumption is that a question labeled H-hop from the short concatenated golden paragraphs remains answerable at exactly H hops when embedded in a 64k–128k continuous narrative; if the surrounding text introduces conflicting or distracting evidence that changes the real reasoning depth, the reported hop-length interaction may reflect label noise rather than model ability.

Editorial extensions

If this is right

  • Model accuracy on NovelHopQA declines monotonically with hop depth across all seven tested models, so the benchmark provides a controlled axis for measuring reasoning depth independent of context length.
  • Raising context length from 64k to 128k consistently lowers accuracy, showing that nominal context-window size overstates usable reasoning context for narrative input.
  • RAG retrieval with top-7 chunks does not recover full-context performance, indicating that better generation alone is insufficient and retrieval must preserve complete hop chains.
  • The four failure modes—missed final-hop integration, entity confusion, incomplete evidence combination, and contextual drift—give concrete targets for diagnosing and improving long-context multi-hop systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If NovelHopQA results generalize, they imply that simply enlarging pretraining context windows will not yield proportional reasoning gains; model developers may need to change how evidence is gathered or integrated (e.g., iterative retrieval or explicit reasoning scaffolds).
  • The benchmark could serve as a diagnostic for attribution and faithfulness: asking a model to cite the paragraph for each hop would test whether failures come from missing retrieval or from failing to combine retrieved pieces.
  • A natural extension is to vary narrative genre or document type (news, legal, technical) to see whether the hop-depth penalty is peculiar to literary prose or reflects a universal long-context limitation.
  • One could also construct a synthetic variant where distractors are controlled (e.g., adding near-miss paragraphs) to separate the effect of reasoning depth from the effect of narrative distraction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces NovelHopQA, a benchmark of 4,000 question-answer pairs built from 64k-128k-token continuous windows of 83 public-domain novels, with intended reasoning depths of 1-4 hops. The construction pipeline selects anchor keywords, chains paragraphs through related-keyword steps, and asks GPT-4o to regenerate the QA pair incrementally at each hop so that the final question integrates all accumulated paragraphs. Quality control consists of a golden-context filter that removes questions missed by any of the seven models later evaluated, human validation of alignment and hop match on the concatenated golden chains, and irrelevant/no-context controls. Seven LLMs are evaluated with chain-of-thought prompting on full windows, and Table 3 reports consistent accuracy drops as hop depth and context length increase; RAG evaluations and qualitative failure-mode analyses are also included. The central claim is that current LLMs, including frontier models, degrade reliably along both the hop-depth and context-length axes in natural narrative.

Significance. If the reported trends hold, NovelHopQA fills a real gap by jointly varying reasoning depth and context length in coherent narrative, complementing benchmarks that fix one axis (e.g., LongBench, RULER on length; HotpotQA, MuSiQue on hops). The consistency of the downward trend across seven models and three context lengths, the near-zero performance in no-context and irrelevant-context controls, and the public release of data and code are concrete strengths that support the diagnostic value of the benchmark. The main caveat is that the hop/answerability labels are validated only on the short golden chains, not on the long windows used for the headline results, so the exact magnitudes of the reported drops are not yet fully anchored. The paper is a useful contribution, but the validity of its central empirical claim requires the additional validation described below.

major comments (4)
  1. [§3.4–§3.5 and §4 vs. Table 3] Answerability and hop-depth labels are established only on the short golden chains (the concatenated hop paragraphs), not on the 64k–128k windows used for the headline results in Table 3. Section 3.4 only requires the hop paragraphs to fall inside a continuous span; it does not check whether the surrounding narrative already answers the question in fewer hops, requires additional hops, or contains material that contradicts or supersedes the gold answer. Since later narrative events can change the correct answer to a question about an earlier state, the monotone drops in Table 3 could in principle reflect label noise that is correlated with hop depth and window length rather than with reasoning failure. The authors should validate a sample of long-window items with human annotators (or with an independent model) and report per-hop and per-length label-noise rates; if those rates are flat, the concern is directly answered.
  2. [§3.5] The golden-context filter evaluates all seven models and discards any question missed by any of them, and the same seven models are then evaluated on the filtered set in Section 5. This creates a circularity for the golden-context accuracy report (Table 1) and for the claim that the dataset is genuinely answerable: a question can survive only if at least one of the seven models that is also a test subject can answer it on the golden chain. The small removal counts in Table 5 (about 4% per hop) make the practical effect modest, but the authors should either filter with a held-out model not used in the main evaluation, or replace model filtering with human filtering on the long windows, and report how many items are removed under that alternative.
  3. [§3.4 step (2c) and §4] The QA pair is regenerated after each hop so that the final question integrates evidence from all h paragraphs, but nothing in the pipeline guarantees that each paragraph is necessary; the final question may be answerable from a subset of the chain. The human Hop Match metric asks only whether exactly H steps are required, and it is applied to the concatenated paragraphs rather than to the long windows. The authors should report a necessity check (for example, human or model performance on prefixes of the chain) to confirm that H=2, 3, 4 items indeed require H paragraphs; without this, the hop-depth axis may overstate the reasoning depth required.
  4. [§5 and §7] All accuracy numbers in Table 3 and the RAG table are produced by GPT-4.1 grading with no reported human agreement. Section 7 acknowledges this limitation, but the central quantitative claims (e.g., accuracy falls about 12 points from 1-hop to 4-hop at 64k) rest entirely on that grader. The authors should provide a human-machine agreement study on a sample (or substitute manual adjudication for a subset) and report grader bias by hop and length if any. Without this, the exact magnitudes of the drops are not anchored.
minor comments (7)
  1. [Abstract] The phrase "evaluate k1–4 hop QA" appears to contain a typo and should read "evaluate 1–4 hop QA."
  2. [Section 5] The informal "We noticed consistent accuracy drops" should be rephrased as "We observe consistent accuracy drops" for a formal register.
  3. [Table 3 caption] The caption states that bold indicates the row-wise maximum, but no bold formatting is visible in the table; the formatting should be fixed or the caption should be revised.
  4. [Tables 1–3] Model names are inconsistent across the tables (e.g., "LLaMa" vs. "LLaMA", "Gemini 2.5 P" vs. "Gemini 2.5 Pro"); these should be standardized.
  5. [References] The FAISS reference is incomplete and the author field "Facebook Research" should be updated to "Facebook AI Research" (or the appropriate institutional name).
  6. [Appendix D] The failure-mode examples read as constructed or idealized illustrations rather than verbatim model outputs; the paper should state explicitly whether these are transcripts or illustrative reconstructions.
  7. [Related Work, Section 2] Given that NovelQA is described as including both single- and multi-hop questions over documents exceeding 200k tokens, the abstract's "first benchmark" claim should be qualified relative to NovelQA's multi-hop subset and NoCha.

Circularity Check

1 steps flagged · score 4.0 of 10

Golden-context 'validation' is the filter criterion itself; long-window hop-depth results are independent but evaluated on a model-filtered subset.

  1. self definitional [Section 3.5 (Golden-Context Filtering), Table 1]
    "To verify answerability, we evaluate all seven models on the original golden contexts used to generate each QA pair. As shown in Table 1, all models score above 90% on average, confirming the validity of most questions. We discard any question missed by any model in the final dataset used in Section 5."

    The validation claim is the filter. 'Validity' is operationalized as 'answered correctly by all seven models on the golden context,' and the dataset reported in Table 1 is exactly the dataset remaining after discarding every question any of those seven models missed. Therefore Table 1's high accuracy is a restatement of the inclusion criterion, not an independent confirmation that the questions are answerable. The paper presents this accuracy as evidence of dataset quality, but it is by construction the selection rule itself. This does not force the long-window accuracy drops in Table 3, because long-window performance was not used in the filter and human validation plus no-context controls provide independent support; the circularity is limited to the golden-context validation claim.

full rationale

NovelHopQA is a benchmark-construction and evaluation paper, not a mathematical derivation, so most circularity patterns do not apply. No fitted parameter is renamed as a prediction; no uniqueness theorem from the authors is invoked; no ansatz is smuggled in via self-citation; and no known result is merely renamed. The only genuine circular step is in Section 3.5: the paper 'validates' answerability by measuring the same seven models' golden-context accuracy that it then uses as a filter, so the reported >90% Table 1 scores are guaranteed by the discarding rule ('We discard any question missed by any model'). That is a self-definitional validation claim. However, the central empirical claim—monotone accuracy decay with hop depth and context length in Table 3—is an evaluation result on long windows, which was not used in the filtering step, so it is not forced by construction. Human annotators independently confirm alignment and hop depth on concatenated golden paragraphs, and the no-context and irrelevant-context controls show near-chance performance, giving the benchmark independent grounding. A more serious validity threat is that answerability and hop labels are verified only on short golden chains and may not transfer to the 64k-128k continuous windows used in Table 3; this is a label-validity limitation, not circularity, because the reported drops are measured rather than derived from the filter. Overall, one auxiliary validation claim reduces to its own inputs, while the headline results retain independent content, warranting a score of 4.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central empirical claims do not depend on fitted parameters. They rest on dataset quality assumptions: GPT-generated and graded QA, filtering by the evaluated models, human validation on a subset, and the transfer of answerability from golden chains to long windows.

assumptions (5)
  • ad hoc to paper Golden-context filtering discards any question missed by any of the seven evaluated models, and the same seven models are later evaluated on the filtered set.
    Section 3.5 and Table 5. This is a design choice specific to this paper; it assumes filtering by the test subjects does not bias the benchmark.
  • domain assumption GPT-4.1 with chain-of-thought prompting accurately grades answer correctness.
    Used for all accuracy numbers in Section 5; Section 7 acknowledges grading bias risk.
  • domain assumption Human annotation on a sample of 2,600 of 4,000 examples is representative for alignment and hop-depth labels.
    Section 4 and Table 6; the full dataset is not human-validated.
  • domain assumption The no-context and irrelevant-context controls, 100 questions per hop per model, are sufficient to rule out memorization.
    Appendix E; small sample, but results near zero for all models.
  • domain assumption Continuous 64k-128k windows containing the hop paragraphs preserve the intended multi-hop structure and answerability.
    Section 3.4 states hop paragraphs fall within the window, but no validation is performed on the long windows.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts." pith.science (2026). https://pith.science/paper/4ITNTIQX

@misc{pith2026250602000,
  author       = {Pith},
  title        = {Pith review of: NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ITNTIQX}},
  note         = {Machine review of arXiv:2506.02000}
}
read the original abstract

Current large language models (LLMs) struggle to answer questions that span tens of thousands of tokens, especially when multi-hop reasoning is involved. While prior benchmarks explore long-context comprehension or multi-hop reasoning in isolation, none jointly vary context length and reasoning depth in natural narrative settings. We introduce NovelHopQA, the first benchmark to evaluate 1-4 hop QA over 64k-128k-token excerpts from 83 full-length public-domain novels. A keyword-guided pipeline builds hop-separated chains grounded in coherent storylines. We evaluate seven state-of-the-art models and apply oracle-context filtering to ensure all questions are genuinely answerable. Human annotators validate both alignment and hop depth. We additionally present retrieval-augmented generation (RAG) evaluations to test model performance when only selected passages are provided instead of the full context. We noticed consistent accuracy drops with increased hops and context length increase, even for frontier models-revealing that sheer scale does not guarantee robust reasoning. Failure-mode analysis highlights common breakdowns such as missed final-hop integration and long-range drift. NovelHopQA offers a controlled diagnostic setting to test multi-hop reasoning at scale. All code and datasets are available at https://novelhopqa.github.io.

Figures

Figures reproduced from arXiv: 2506.02000 by the authors.

Figure 1
Figure 1. Keyword-guided paragraph-chaining pipeline used to build [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Accuracy (%) on NOVELHOPQA across context lengths and hop depths H ∈ {1, 2, 3, 4}. This heatmap shows how model accuracy declines as both narrative length and multi-hop reasoning depth increase. To complement the heatmap, we include detailed line plots illustrating model-specific trends across each axis independently [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Model performance across context lengths for each hop level [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Model performance across hop levels for each context length (64k, 96k, 128k). These plots isolate the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Left: Heatmap of RAG-augmented accuracy across hop depths for all models. Right: Accuracy curves for [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Prompt for extracting five high-frequency anchor keywords from a book title [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Prompt for generating a single-hop question from one paragraph. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Prompt for extracting a related keyword at hop h. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Prompt for generating the final multi-hop question over H paragraphs. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 10 canonical work pages

  1. [1]

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. https://arxiv.org/abs/2307.11088 L-eval: Instituting standardized evaluation for long context language models . Preprint, arXiv:2307.11088

    arXiv 2023

  2. [2]

    BAAI. 2023. Baai general embedding (bge-large-en). https://huggingface.co/BAAI/bge-large-en

    work page 2023

  3. [3]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://arxiv.org/abs/2308.14508 Longbench: A bilingual, multitask benchmark for long context understanding . Preprint, arXiv:2308.14508

    arXiv 2024

  4. [4]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. https://arxiv.org/abs/2004.05150 Longformer: The long-document transformer . Preprint, arXiv:2004.05150

    arXiv 2020

  5. [5]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. https://arxiv.org/abs/1901.02860 Transformer-xl: Attentive language models beyond a fixed-length context . Preprint, arXiv:1901.02860

    arXiv 2019

  6. [6]

    Google DeepMind. 2025 a . https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-0-flash Gemini 2.0 flash and flash lite . Online documentation. Google Cloud, Vertex AI, and Google AI Studio documentation

    work page 2025

  7. [7]

    Google DeepMind. 2025 b . Gemini model and thinking updates: March 2025. https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/. Accessed: 2025-05-16

    work page 2025

  8. [8]

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. https://arxiv.org/abs/2402.13753 Longrope: Extending llm context window beyond 2 million tokens . Preprint, arXiv:2402.13753

    arXiv 2024

Show all 39 references
  1. [9]

    Abhay Gupta, Jacob Cheung, Philip Meng, Shayan Sayyed, Austen Liao, Kevin Zhu, and Sean O'Brien. 2025. https://arxiv.org/abs/2504.07100 Endive: A cross-dialect benchmark for fairness and performance in large language models . Preprint, arXiv:2504.07100

    2025 arXiv

  2. [10]

    Abhay Gupta, Philip Meng, Ece Yurtseven, Sean O'Brien, and Kevin Zhu. 2024. https://arxiv.org/abs/2408.14845 Aavenue: Detecting llm biases on nlu tasks in aave via a novel benchmark . Preprint, arXiv:2408.14845

    2024

  3. [11]

    Project Gutenberg. 2025. https://www.gutenberg.org/ Project gutenberg . Accessed: 2025-04-17

    2025

  4. [12]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. https://arxiv.org/abs/2404.06654 Ruler: What's the real context size of your long-context language models? Preprint, arXiv:2404.06654

    2024 arXiv

  5. [13]

    Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. https://arxiv.org/abs/2406.16264 One thousand and one pairs: A "novel" challenge for long-context language models . Preprint, arXiv:2406.16264

    2024 arXiv

  6. [14]

    Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2017. https://arxiv.org/abs/1712.07040 The narrativeqa reading comprehension challenge . Preprint, arXiv:1712.07040

    2017 arXiv

  7. [15]

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. https://arxiv.org/abs/2406.10149 Babilong: Testing the limits of llms with long context reasoning-in-a-haystack . Preprint, arXiv:2406.10149

    2024 arXiv

  8. [16]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. https://arxiv.org/abs/2005.11401 Retrieval-augmented generation for knowledge-int...

    2021 arXiv

  9. [17]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024 a . https://arxiv.org/abs/2311.04939 Loogle: Can long-context language models understand long contexts? Preprint, arXiv:2311.04939

    2024 arXiv

  10. [18]

    Mo Li, , Songyang Zhang, Yunxin Liu, and Kai Chen. 2024 b . https://arxiv.org/abs/2407.11963 Needlebench: Can llms do retrieval and reasoning in 1 million context window? Preprint, arXiv:2407.11963

    2024

  11. [19]

    Lyu, and Liwei Wang

    Yanyang Li, Shuo Liang, Michael R. Lyu, and Liwei Wang. 2024 c . https://arxiv.org/abs/2408.03246 Making long-context language models better multi-hop reasoners . Preprint, arXiv:2408.03246

    2024 arXiv

  12. [21]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023 b . https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

    2023 arXiv

  13. [22]

    Meta. 2024. https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct Llama 3.3 70b instruct . Hugging Face

    2024

  14. [23]

    OpenAI. 2024 a . https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  15. [24]

    OpenAI. 2024 b . https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api . Accessed: 2025-05-17

    2024

  16. [25]

    OpenAI. 2024 c . Introducing openai o1. https://openai.com/o1/. Accessed: 2025-05-16

    2024

  17. [26]

    Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. 2022. https://arxiv.org/abs/2112.08608 Quality: Question answering with long input texts, yes! Preprint, ar...

    2022 arXiv

  18. [27]

    Facebook Research. 2017. Faiss: A library for efficient similarity search and clustering of dense vectors. https://github.com/facebookresearch/faiss

    2017

  19. [28]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://arxiv.org/abs/2108.00573 Musique: Multihop questions via single-hop question composition . Preprint, arXiv:2108.00573

    2022 arXiv

  20. [29]

    Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Xiangkun Hu, Zheng Zhang, Qian Wang, and Yue Zhang. 2024 a . https://arxiv.org/abs/2403.12766 Novelqa: Benchmarking question answering on documents exceeding 200k tokens . Preprint, arXiv:...

    2024 arXiv

  21. [30]

    Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024 b . https://arxiv.org/abs/2406.17419 Leave no document behind: Benchmarking long-context llms with extend...

    2024 arXiv

  22. [31]

    Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. https://arxiv.org/abs/1710.06481 Constructing datasets for multi-hop reading comprehension across documents . Preprint, arXiv:1710.06481

    2018 arXiv

  23. [32]

    Rabe, DeLesley Hutchins, and Christian Szegedy

    Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. 2022. https://arxiv.org/abs/2203.08913 Memorizing transformers . Preprint, arXiv:2203.08913

    2022 arXiv

  24. [33]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024. https://arxiv.org/abs/2310.03025 Retrieval meets long context large language models . Preprint, arXiv:2310.03025

    2024 arXiv

  25. [34]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://arxiv.org/abs/1809.09600 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . Preprint, arXiv:1809.09600

    2018 arXiv

  26. [35]

    Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2024. https://arxiv.org/abs/2402.05136 Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k . Prepr...

    2024

  27. [36]

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2021. https://arxiv.org/abs/2007.14062 Big bird: Transformers for longer sequences . Preprint, arXiv:2007.14062

    2021 arXiv

  28. [37]

    Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi yang, Junhao Liu, Longze Chen, Run Luo, and Min Yang. 2024. https://arxiv.org/abs/2312.09542 Marathon: A race through the realm of long context with large language models . Preprint, arXiv:2312.09542

    2024 arXiv

  29. [38]

    Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. 2024. https://arxiv.org/abs/2402.14116 Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models . Preprint, arXiv:2402.14116

    2024 arXiv

  30. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.