Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MINTQA is a benchmark for multi-hop QA over new and tail knowledge, and it shows that current LLMs, including GPT-4o, fail badly on such queries.

desk verdict MINTQA is a substantial and useful multi-hop QA dataset, but its 'new knowledge' label rests on a weak Wikidata-first-appearance proxy, so the headline temporal finding needs qualification. read the letter →

arxiv 2412.17032 v3 pith:R356WUSX submitted 2024-12-22 cs.CL

Jie He , Nan Hu , Wanqiu Long , Jiaoyan Chen , Jeff Z. Pan This is my paper · ORCID
classification cs.CL
keywords multi-hopquestionansweringlong-tailknowledgenewretrieval-augmentedgenerationLLMevaluationWikidatabenchmarkdecompositionboundary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MINTQA, a benchmark of 28,366 multi-hop question-answer pairs built from Wikidata and Wikipedia, split into MINTQA-POP (17,887 questions on popular versus unpopular knowledge) and MINTQA-TI (10,479 questions on old versus new knowledge). Its purpose is to measure whether LLMs can handle queries that require several reasoning hops and facts that are rare or have only recently appeared. The paper evaluates 22 LLMs and reports that accuracy collapses on new and unpopular knowledge, with GPT-4o falling to 21.17% on single-hop new-knowledge questions and performance degrading further as hop count rises. Retrieval helps, but less on the hardest cases, and even giving models the right sub-questions does not close the gap on new knowledge. The central claim is that current multi-hop QA evaluation misses a distinct capability: coordinating decomposition and retrieval when knowledge is fresh or tail.

What carries the argument

The central object is the fact chain: an ordered sequence of Wikidata triples $(s_1,r_1,o_1), \dots, (s_n,r_n,o_n)$ with $o_i = s_{i+1}$, no cycles, no chain that is a sub-chain of another, and no shortcut triple connecting non-adjacent entities. Each chain grounds a multi-hop question whose answer is $o_n$, with per-hop sub-questions generated and validated by GPT-4o. The popularity and time splits rest on two proxies (Wikipedia counts for popular versus unpopular, Wikidata dump versions for old versus new), and the evaluation machinery isolates five capabilities: parametric answering, strategy choice, retrieval benefit, sub-question generation, and iterative or dynamic retrieval.

What would settle it

Take a matched sample of MINTQA-TI 'new' facts and 'old' facts with the same answer-entity popularity and relation types, and probe an LLM whose training cutoff predates 2021; if accuracy on the two sets is statistically indistinguishable, the new/old split is not isolating knowledge recency but is instead tracking popularity or entity type.

Watch

Extended reading notes

Core claim

MINTQA is constructed by chaining connected Wikidata triples into fact chains of one to four hops, with constraints that forbid cycles, shortcuts, and duplicate sub-chains, and by using GPT-4o to turn each chain into a question plus validated sub-questions. Popularity is defined by occurrence frequency in Wikipedia (unpopular: 1 to 9 occurrences; popular: at least 50), and newness by presence in the 2024-06-05 Wikidata dump but absence from the 2021-06-21 dump. Evaluating 22 instruction-tuned LLMs across five protocols, the paper finds that parametric accuracy on MINTQA-TI single-hop questions ranges from 7.78% to 21.17%, that knowledge-boundary awareness improves with model size but is unreliable, that direct retrieval gives roughly a 30-point gain on TI, that self-generated sub-questions hurt TI accuracy, and that decomposition-then-retrieval underperforms direct retrieval on new knowledge. The paper concludes that MINTQA measures a real capability gap in coordinating retrieval and decomposition on fresh and tail knowledge, and that current models, even with retrieval, are far from closing it.

Load-bearing premise

The load-bearing premise is that Wikipedia mention counts and a triple's absence from an older Wikidata dump really mark the edge of what LLMs know, and if that proxy is wrong the central contrast between new/tail and old/popular knowledge stops measuring actual knowledge.

Editorial extensions

If this is right

  • Benchmarks that only test popular or static facts overstate LLM multi-hop capability, so including unpopular and newly added facts is necessary to expose the boundary.
  • Retrieval-augmented generation is not a uniform fix: direct retrieval helps new knowledge, but iterative decomposition-then-retrieval can hurt, indicating the decomposition step itself is a bottleneck.
  • Giving a model gold sub-questions substantially improves tail-knowledge accuracy but leaves new-knowledge accuracy low, meaning the missing ingredient is the facts themselves, not just the reasoning plan.
  • Models show systematic overconfidence: smaller models often answer directly or assume the answer is already in hand, while even large models struggle to judge when their own knowledge is insufficient.
  • Confidence-gated dynamic retrieval does not yet maintain accuracy while cutting retrieval calls, so efficiency gains come at a real accuracy cost on this benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the popularity proxy is imperfect, the headline gap may partly reflect entity-type confounds, because unpopular triples in Wikipedia tend to involve niche entities that are also less likely to appear in training corpora; a controlled test matching entity types would separate the two effects.
  • A natural extension is to use MINTQA's sub-question pairs as supervision for training a decomposition model, then test whether that training transfers to unseen fresh facts; the paper does not train such a model.
  • The new/old split could be made dynamic by re-running the construction on later Wikidata dumps, turning current 'new' facts into 'old' ones and letting researchers measure how quickly model knowledge catches up.
  • The near-ceiling performance with gold sub-questions and gold retrieval suggests the next bottleneck is document-level synthesis, not decomposition or retrieval alone; this goes beyond the paper's explicit claims.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MINTQA, a multi-hop question answering benchmark constructed from Wikidata triples and Wikipedia frequencies, comprising two subsets: MINTQA-POP (17,887 pairs) for popular/unpopular knowledge and MINTQA-TI (10,479 pairs) for old/new knowledge, with one- to four-hop chains and GPT-4o-generated sub-questions. The authors evaluate 22 LLMs across five settings (parametric knowledge, strategy selection, direct retrieval, sub-question generation, and decomposition-with-retrieval) and report that models perform markedly worse on new and unpopular knowledge, with even the best system achieving only 62.33% overall accuracy.

Significance. If the benchmark's construct validity holds, MINTQA would fill a clear gap in multi-hop QA evaluation, and the systematic comparison of 22 models under multiple reasoning strategies would be a valuable community resource. The dataset is released with a permissive license, includes sub-question annotations, and the paper provides a substantial amount of evaluation detail. However, the validity of the core 'new knowledge' claim and the integrity of the multi-hop annotation in retrieval settings are not fully established, so the significance depends on the authors' ability to address the issues below.

major comments (3)
  1. [Section 3.2; Appendix E, Table 22] The operationalization of 'new knowledge' as triples present in the 2024-06-05 Wikidata dump but absent from the 2021-06-21 dump conflates Wikidata ingestion time with fact recency. A concrete instance is Appendix E, Table 22, where the triple (Pigeon Bay Domain, country, New Zealand) is labeled Type: New, even though this is a stable geographical fact that almost certainly predates 2021 and may well appear in LLM pretraining corpora. As a result, the low parametric accuracy on MINTQA-TI (e.g., GPT-4o at 21.17% for single-hop questions, Section 5) cannot be unambiguously attributed to knowledge that postdates training. The paper's Limitation section acknowledges that Wikidata distribution patterns may not reflect LLM pretraining data, but it does not address this specific conflation. To support the abstract's claim that models are 'particularly' limited on new knowledge, the authors should either re-annotate the 'new' subset using a genuine temporal signal (e.g., entity inception dates or event dates) or explicitly reinterpret MINTQA-TI as measuring 'recently added to Wikidata' and adjust the conclusions accordingly.
  2. [Section 3.1, constraint 4; Section 7.1] The No Shortcuts check is defined only over Gpop, but the retrieval corpus used in the evaluation is G = Gpop ∪ Gti (Section 7.1). A chain sampled from Gpop can therefore be shortcut by a triple that exists only in Gti (and vice versa for chains from Gti), so the effective number of hops in the retrieval experiments may be lower than annotated. Because the paper's multi-hop validity claims apply to the retrieval-augmented settings (Sections 7 and 9), the No Shortcuts constraint should be verified against the full retrieval corpus, not just the subgraph from which the chain was sampled.
  3. [Section 3.1] The popularity thresholds (unpopular: [1,10), popular: [50,∞)) are arbitrary and no sensitivity analysis is provided. Since the central dimension of MINTQA-POP is defined by this binarization, the paper should report how the main conclusions (e.g., the performance gap between popular and unpopular questions) change under reasonable alternative thresholds, or justify the chosen cutoffs empirically.
minor comments (5)
  1. [Table 8] The caption and column layout are difficult to parse; the reader cannot easily determine which columns correspond to MINTQA-POP versus MINTQA-TI and how the 'proportion' values map to the actual hop counts. Please clarify the column headers or separate the POP and TI results into distinct tables.
  2. [Section 5] The specific numbers '21.17%' (GPT-4o) and '7.78%' (LLaMA-3.2-1B) for single-hop new-knowledge questions should be accompanied by an explicit pointer to the table that contains these results, since Table 8 in the appendix does not report hop-disaggregated values in an obvious way.
  3. [Appendix B] The phrase 'MINTQAQA dataset' contains a typo; it should read 'MINTQA dataset'.
  4. [Figure 2] The label 'Parametric Konwlodege Exploration' contains a spelling error; it should read 'Parametric Knowledge Exploration'.
  5. [Table 5] The phrase 'LLaMA-3.1-8B sees a 33.41% increase' actually refers to a 33.41 percentage-point increase (from 37.28 to 70.69); please rephrase for precision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's new/tail labels come from external Wikidata/Wikipedia data, and its accuracy results are empirical measurements against Wikidata answers.

full rationale

MINTQA is a benchmark-construction and evaluation paper rather than a derivation from first principles. The popularity and newness labels are operationalized from external data: Wikipedia occurrence frequency for popularity, and presence in the 2024-06-05 Wikidata dump but not the 2021-06-21 dump for newness. These labels are not derived from the models being evaluated. The QA answers are the Wikidata object entities of sampled triples, so accuracy scores are measured against an external knowledge base rather than against any fitted parameter or model output. No parameter is fitted and then renamed as a prediction, and no load-bearing self-citation, uniqueness theorem, or ansatz-importing citation appears in the construction chain. The one methodological overlap is that GPT-4o generates and validates the natural-language questions and is also among the evaluated models; however, this does not make any reported result true by construction, because questions are filtered by whether GPT-4o can answer from source contexts while the ground-truth answers remain the original Wikidata triples, and a human check of 500 sampled items found only 2% contextually insufficient cases. The skeptical concern that 'new' as absent from the 2021 Wikidata dump may not mean temporally new relative to LLM pretraining corpora is a construct-validity limitation, explicitly acknowledged in the Limitation section ('our definition of long-tail and new facts relies solely on Wikidata distribution patterns, which may not very accurately reflect knowledge representation in LLMs' diverse pre-training corpora'), not a circular derivation of the benchmark's headline results. Therefore no circular step is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims depend on the popularity and temporal proxies for knowledge, on the completeness of the shortcut check, and on the validity of GPT-4o-generated questions. These are domain assumptions rather than fitted numerical parameters; the numerical choices (thresholds, top-k, max hops) are hand-selected design parameters.

free parameters (5)
  • popularity threshold for unpopular facts = P_unpop = [1, 10)
    Used to label triples as unpopular; boundary chosen without justification beyond corpus frequency in Section 3.1.
  • popularity threshold for popular facts = P_pop = [50, ∞)
    Used to label triples as popular; Section 3.1.
  • max hops = n ≤ 4
    Design choice for multi-hop chains, Section 3.1/3.2.
  • top-k retrieved passages = 5
    Used in all retrieval evaluations, Section 7.2.
  • max iterations in decomposition-then-retrieval = 5
    Section 9.1.
assumptions (4)
  • domain assumption The Wikipedia triple frequency is proportional to the likelihood that an LLM has memorized the fact.
    This underlies the popularity split in Section 3.1; flagged as weak in the Limitation.
  • ad hoc to paper The No Shortcuts constraint only needs to be checked against Gpop, the popularity-sampled graph.
    Section 3.1 constraint 4; if a shortcut exists in the full retrieval corpus G but not Gpop, the question may be answerable with fewer hops than intended.
  • domain assumption GPT-4o-generated questions are valid, natural, and unambiguous after self-validation.
    Section 3.3 and Table 13; validation uses the same model that generates questions; only a 500-sample human check is performed.
  • domain assumption Wikidata version differences (2021-06-21 vs 2024-06-05) capture temporal 'newness' of knowledge.
    Section 3.2; new entities/relations in the newer dump are treated as new knowledge.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge." pith.science (2026). https://pith.science/paper/R356WUSX

@misc{pith2026241217032,
  author       = {Pith},
  title        = {Pith review of: MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R356WUSX}},
  note         = {Machine review of arXiv:2412.17032}
}
read the original abstract

Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks but face significant challenges with complex, knowledge-intensive multi-hop queries, particularly those involving new or long-tail knowledge. Existing benchmarks often fail to fully address these challenges. To bridge this gap, we introduce MINTQA (Multi-hop Question Answering on New and Tail Knowledge), a comprehensive benchmark to evaluate LLMs' capabilities in multi-hop reasoning across four critical dimensions: question handling strategy, sub-question generation, retrieval-augmented generation, and iterative or dynamic decomposition and retrieval. MINTQA comprises 10,479 question-answer pairs for evaluating new knowledge and 17,887 pairs for assessing long-tail knowledge, with each question equipped with corresponding sub-questions and answers. Our systematic evaluation of 22 state-of-the-art LLMs on MINTQA reveals significant limitations in their ability to handle complex knowledge base queries, particularly in handling new or unpopular knowledge. Our findings highlight critical challenges and offer insights for advancing multi-hop reasoning capabilities. The MINTQA benchmark is available at https://github.com/probe2/multi-hop/.

Figures

Figures reproduced from arXiv: 2412.17032 by the authors.

Figure 1
Figure 1. A example for our benchmark: Given a complex question, the model must decide whether to de￾compose it into sub-questions and determine if external knowledge retrieval is required. Consider a complex question: “What is the high￾est point in the country that hosted the 2010 Win￾ter Olympics?” As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Two components of our work: (a) we sample different types of facts from Wikidata to generate complex [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Zero-shot accuracy of different LLMs across [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The performance of Qwen2.5-72B with gold [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of LLMs on MINTQA-TI using different retrieval methods: "Oracle" uses gold-standard retrieval passages, while "Vanilla" involves models answering without retrieval content. Notably, in the Oracle setting, where gold-standard passages are used, ev…
Figure 8
Figure 8. Figure 8: Popularity Related Data Distribution N O N,N N,O O,NO,O N,N,N N,N,O N,O,N N,O,O O,N,N O,N,O O,O,N O,O,O N,N,N,N N,N,N,O N,N,O,N N,N,O,O N,O,N,N N,O,N,O N,O,O,N N,O,O,O O,N,N,N O,N,N,O O,N,O,N O,N,O,O O,O,N,N O,O,N,O O,O,O,N O,O,O,O Hop Patterns 0 250 500 750 1000 1250 …
Figure 9
Figure 9. Figure 9: Time Related Data Distribution B Human Sampling Check Despite our data filtering strategy’s success in im￾proving data quality compared to the initial dataset, some failure cases still exist. Our analysis revealed that certain samples in the MINTQAQA dataset share a co…
Figure 10
Figure 10. Figure 10: Recall performance of retrieval methods across two datasets for varying question hops. veals that, despite differences in the datasets, large models exhibit similar distributions in terms of ac￾tual step counts and the number of sub-questions generated by the models. …
Figure 11
Figure 11. Figure 11: Heatmaps (a) and (c) show Qwen2.5- 72B with BM25 performance on two datasets, while heatmaps (b) and (d) shows BM25 recall. The X-axis represents the proportion of popular knowledge required in the question, and the Y-axis indicates question hops [PITH_FULL_IMAGE:fig…
Figure 12
Figure 12. Figure 12: Performance of all models with the three [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Performance comparison of LLMs on MINTQA-POP and MINTQA-TI using different retrieval methods. "Oracle" uses gold-standard retrieval passages, while "Vanilla" involves models answering without retrieval content. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: The confusion matrix of the number of sub-questions generated by the large language models for main [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation

    cs.IR 2025-05 conditional novelty 6.0 of 10

    InfoDeepSeek is a 245-question benchmark that measures how well AI agents seek information on the live web, with new metrics for answer accuracy, evidence quality, and compactness.

  2. Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs

    cs.CL 2025-07 unverdicted novelty 3.0 of 10

    A survey organizing RAG-reasoning systems into three stages: reasoning-enhanced RAG, RAG-enhanced reasoning, and synergized agentic RAG-reasoning.

Reference graph

Works this paper leans on

57 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, and Martin Cai etc. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . Preprint, ...

    arXiv 2024

  2. [2]

    Yufei Feng, Mo Yu, Wenhan Xiong, Xiaoxiao Guo, Junjie Huang, Shiyu Chang, Murray Campbell, Michael Greenspan, and Xiaodan Zhu. 2020. https://arxiv.org/abs/2004.02393 Learning to recover reasoning chains for multi-hop question answering via cooperative games . Preprint, arXiv:2004.02393

    work page Pith review arXiv 2020

  3. [3]

    Christian Keller etc

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    arXiv 2024

  4. [4]

    Nguyen, Saku Sugawara, and Akiko Aizawa

    Xanh Ho, A. Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://api.semanticscholar.org/CorpusID:226236740 Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps . ArXiv, abs/2011.01060

    arXiv 2020

  5. [5]

    Jie Huang and Kevin Chen-Chuan Chang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.67 Towards reasoning in large language models: A survey . In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049--1065, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-acl.67 2023

  6. [6]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. https://arxiv.org/abs/2409.12186 Qwen2.5-coder technica...

    arXiv 2024

  7. [7]

    Joty, and Md

    Shayekh Bin Islam, Md Asib Rahman, K S M Tozammel Hossain, Enamul Hoque, Shafiq R. Joty, and Md. Rizwan Parvez. 2024. https://api.semanticscholar.org/CorpusID:273026102 Open-rag: Enhanced retrieval-augmented reasoning with open-source large language models . ArXiv, abs/2410.01782

    arXiv 2024

  8. [8]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. https://api.semanticscholar.org/CorpusID:249097975 Unsupervised dense information retrieval with contrastive learning . Trans. Mach. Learn. Res., 2022

    2021

Show all 57 references
  1. [9]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C. Park. 2024. https://api.semanticscholar.org/CorpusID:268553748 Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity . In North American Chapter of the Associati...

    2024

  2. [10]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  3. [11]

    Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. 2023. https://api.semanticscholar.org/CorpusID:258615193 Evaluating open-domain question answering in the era of large language models . ArXiv, abs/2305.06984

    2023 arXiv

  4. [12]

    Daniel Khashabi, Erfan Sadeqi Azer, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2019. https://api.semanticscholar.org/CorpusID:218469981 On the possibilities and limitations of multi-hop reasoning under linguistic imperfections . arXiv: Computation and Language

    2019

  5. [13]

    Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, an...

    2019

  6. [14]

    Gonzalez, Haotong Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, and Ion Stoica. 2023. https://api.semanticscholar.org/CorpusID:261697361 Efficient memory management for large language model serving with pagedattention . Procee...

    2023

  7. [15]

    Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://api.semanticscholar.org/CorpusID:218869575 Retrieval-augmented gener...

    2020 arXiv

  8. [16]

    Yu, and Surinder Kumar

    Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix X. Yu, and Surinder Kumar. 2022. https://api.semanticscholar.org/CorpusID:253420654 Large language models with controllable working memory . ArXiv, abs/2211.05110

    2022 arXiv

  9. [17]

    Zekai Li and Wei Peng. 2023. https://api.semanticscholar.org/CorpusID:258536349 Self-adaptive reasoning on sub-questions for multi-hop question answering . ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5

    2023

  10. [18]

    Yanming Liu, Xinyue Peng, Xuhong Zhang, Weihao Liu, Jianwei Yin, Jiannan Cao, and Tianyu Du. 2024. https://api.semanticscholar.org/CorpusID:268363612 Ra-isf: Learning to answer and understand from retrieval augmentation via iterative self-feedback . In Annual Meeting of the As...

    2024

  11. [19]

    Seiji Maekawa, Hayate Iso, Sairam Gurajada, and Nikita Bhutani. 2024. https://api.semanticscholar.org/CorpusID:267770347 Retrieval helps or hurts? a deeper dive into the efficacy of retrieval augmentation to language models . ArXiv, abs/2402.13492

    2024 arXiv

  12. [20]

    Alex Troy Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. https://api.semanticscholar.org/CorpusID:254877603 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Annual Mee...

    2022

  13. [21]

    Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. https://api.semanticscholar.org/CorpusID:174801764 Compositional questions do not necessitate multi-hop reasoning . In Annual Meeting of the Association for Computational Linguistics

    2019

  14. [22]

    Hall, Ming-Wei Chang, and Yinfei Yang

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hern \'a ndez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021. https://api.semanticscholar.org/CorpusID:245144556 Large dual encoders are generalizable retrievers . ArXiv, abs/2112.07899

    2021 arXiv

  15. [23]

    Guo, and Xueqi Cheng

    Shiyu Ni, Keping Bi, J. Guo, and Xueqi Cheng. 2024. https://api.semanticscholar.org/CorpusID:267751438 When do llms need retrieval augmentation? mitigating llms' overconfidence helps retrieval augmentation . In Annual Meeting of the Association for Computational Linguistics

    2024

  16. [24]

    Liu, Hao Tian, Huaqin Wu, Ji rong Wen, and Haifeng Wang

    Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, J. Liu, Hao Tian, Huaqin Wu, Ji rong Wen, and Haifeng Wang. 2023. https://api.semanticscholar.org/CorpusID:259991467 Investigating the factual knowledge boundary of large language models with retrieval augmentation . ArXiv, a...

    2023 arXiv

  17. [25]

    Robertson and Hugo Zaragoza

    Stephen E. Robertson and Hugo Zaragoza. 2009. https://api.semanticscholar.org/CorpusID:207178704 The probabilistic relevance framework: Bm25 and beyond . Found. Trends Inf. Retr., 3:333--389

    2009

  18. [26]

    Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. https://api.semanticscholar.org/CorpusID:252693442 Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering . ArXiv, abs/2210.01613

    2022 arXiv

  19. [27]

    Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong, Kaixiong Zhou, and Ninghao Liu. 2024 a . https://api.semanticscholar.org/CorpusID:268733173 Retrieval-enhanced knowledge editing in language models for multi-hop question answering . In International Conference on Informat...

    2024

  20. [28]

    Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. 2024 b . https://api.semanticscholar.org/CorpusID:270688739 Generate-then-ground in retrieval-augmented generation for multi-hop question answering . In Annual Meeting of the Associat...

    2024

  21. [29]

    Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. 2024. https://doi.org/10.1145/3673791.3698415 Fine tuning vs. retrieval augmented generation for less popular knowledge . In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in...

    2024

  22. [30]

    Smith, Luke Zettlemoyer, and Tao Yu

    Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. https://arxiv.org/abs/2212.09741 One embedder, any task: Instruction-finetuned text embeddings

    2022 arXiv

  23. [31]

    Xu, Hanwen Zha, Yue Liu, and Xinhsuai Dong

    Kai Sun, Y. Xu, Hanwen Zha, Yue Liu, and Xinhsuai Dong. 2023. https://api.semanticscholar.org/CorpusID:261048922 Head-to-tail: How knowledgeable are large language models (llms)? a.k.a. will llms replace knowledge graphs? ArXiv, abs/2308.10168

    2023 arXiv

  24. [32]

    Yixuan Tang and Yi Yang. 2024. https://api.semanticscholar.org/CorpusID:267312593 Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries . ArXiv, abs/2401.15391

    2024 arXiv

  25. [33]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  26. [34]

    Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

    H. Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2021. https://api.semanticscholar.org/CorpusID:236771976 Musique: Multihop questions via single-hop question composition . Transactions of the Association for Computational Linguistics, 10:539--554

    2021

  27. [35]

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2023. https://api.semanticscholar.org/CorpusID:263672149 Freshllms: Refreshing large language models with search engine augmentation . In Annu...

    2023

  28. [36]

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. https://doi.org/10.18653/v1/2024.findings-acl.813 F resh LLM s: Refreshing large language models with search engine augmentation . In Fi...

    2024 doi

  29. [37]

    Jinyuan Wang, Junlong Li, and Hai Zhao. 2023. https://api.semanticscholar.org/CorpusID:264406215 Self-prompted chain-of-thought on large language models for open-domain multi-hop reasoning . ArXiv, abs/2310.13552

    2023 arXiv

  30. [38]

    Shouhui Wang and Biao Qin. 2024. https://api.semanticscholar.org/CorpusID:269804292 No need for large-scale search: Exploring large language models in complex knowledge base question answering . In International Conference on Language Resources and Evaluation

    2024

  31. [39]

    Siyuan Wang, Zhongyu Wei, Zhihao Fan, Qi Zhang, and Xuanjing Huang. 2022. https://api.semanticscholar.org/CorpusID:251718892 Locate then ask: Interpretable stepwise reasoning for multi-hop question answering . In International Conference on Computational Linguistics

    2022

  32. [40]

    Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. 2024. Searching for best practices in retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Langu...

    2024

  33. [41]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. https://api.semanticscholar.org/CorpusID:246411621 Chain of thought prompting elicits reasoning in large language models . ArXiv, abs/2201.11903

    2022 arXiv

  34. [42]

    Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, and Jack Hessel. 2024. https://api.semanticscholar.org/CorpusID:272694661 Promptriever: Instruction-trained retrievers can be prompted like language models . ArXiv, abs/2409.11136

    2024 arXiv

  35. [43]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. 2023. https://api.semanticscholar.org/CorpusID:265351794 Lm-cocktail: Resilient tuning of language models via model merging . ArXiv, abs/2311.13534

    2023 arXiv

  36. [44]

    Siheng Xiong, Ali Payani, Yuan Yang, and Faramarz Fekri. 2025. https://doi.org/10.18653/v1/2025.acl-long.1540 Deliberate reasoning in language models as structure-aware planning with an accurate world model . In Proceedings of the 63rd Annual Meeting of the Association for Com...

    2025 doi

  37. [45]

    Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen tau Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oğuz. 2020. https://api.semanticscholar.org/CorpusID:221970302 Answering complex open-domain questions with multi-ho...

    2020 arXiv

  38. [46]

    Shicheng Xu. 2023. https://api.semanticscholar.org/CorpusID:267938725 Search-in-the-chain: Towards accurate, credible and traceable large language models for knowledge-intensive tasks

    2023

  39. [47]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://api.semanticscholar.org/CorpusID:52822214 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . In Conference on Empiric...

    2018

  40. [48]

    Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Yang Wang, Zhiguo Wang, and Bing Xiang. 2023. https://openreview.net/forum?id=XHc5zRPxqV9 Decaf: Joint decoding of answers and logical forms for question answering over knowledge ...

    2023

  41. [49]

    Yu, Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang

    W. Yu, Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2020. https://api.semanticscholar.org/CorpusID:222272210 A survey of knowledge-enhanced text generation . ACM Computing Surveys, 54:1 -- 38

    2020

  42. [50]

    Zihan Zhang, Meng Fang, and Ling Chen. 2024. https://api.semanticscholar.org/CorpusID:268033124 Retrievalqa: Assessing adaptive retrieval-augmented generation for short-form open-domain question answering . ArXiv, abs/2402.16457

    2024 arXiv

  43. [51]

    Joty, Chengwei Qin, and Lidong Bing

    Ruochen Zhao, Xingxuan Li, Shafiq R. Joty, Chengwei Qin, and Lidong Bing. 2023. https://api.semanticscholar.org/CorpusID:258547173 Verify-and-edit: A knowledge-enhanced chain-of-thought framework . ArXiv, abs/2305.03268

    2023 arXiv

  44. [52]

    Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. 2024 a . https://api.semanticscholar.org/CorpusID:267782780 Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models . In Annual Meeting of the Association for Computational Linguistics

    2024

  45. [53]

    Yun Zhu, Jia-Chen Gu, Caitlin Sikora, Ho Ko, Yinxiao Liu, Chu-Cheng Lin, Lei Shu, Liangchen Luo, Lei Meng, Bang Liu, and Jindong Chen. 2024 b . https://api.semanticscholar.org/CorpusID:270062557 Accelerating inference of retrieval-augmented generation via sparse context select...

    2024 arXiv

  46. [54]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji rong Wen. 2023. https://api.semanticscholar.org/CorpusID:260887838 Large language models for information retrieval: A survey . ArXiv, abs/2308.07107

    2023

  47. [55]

    Rajmohan, Dongmei Zhang, and Qi Zhang

    Ziyuan Zhuang, Zhiyang Zhang, Sitao Cheng, Fangkai Yang, Jia Liu, Shujian Huang, Qingwei Lin, S. Rajmohan, Dongmei Zhang, and Qi Zhang. 2024. https://api.semanticscholar.org/CorpusID:271769059 Efficientrag: Efficient retriever for multi-hop question answering . ArXiv, abs/2408.04259

    2024 arXiv

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.