Pith. sign in

REVIEW 2 major objections 5 minor 57 references

At enterprise scale, BM25 lexical retrieval outperforms file-system agents, dense retrieval, and graph-based RAG, crossing over near 10 million corpus tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:39 UTC pith:PWJ4D3EE

load-bearing objection A carefully controlled scaling study with a real but benchmark-conditional result: BM25 overtakes agentic search at scale on this corpus, and the caveat is in the paper's own trap construction. the 2 major comments →

arxiv 2607.26497 v3 pith:PWJ4D3EE submitted 2026-07-29 cs.CL

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Pengyu Wang , Benfeng Xu , Shaohan Wang , Mingxuan Du , Xin Zeng , Huarui Wu , Lei Zhang , Licheng Zhang This is my paper
classification cs.CL
keywords retrieval-augmented generationBM25lexical retrievalagentic searchdense retrievalgraph-based RAGcorpus scalingcost-accuracy tradeoff
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that retrieval-augmented generation has no single best paradigm across corpus sizes: a file-system agent that explores raw documents with LLM tool calls leads at small scale, but BM25 lexical retrieval overtakes it at roughly 10 million corpus tokens and leads at every larger size tested, by nearly 20 points at 601 million tokens. The study controls corpus scale along 28 strictly nested tiers while holding questions, gold evidence, and adversarial distractors fixed. It finds that dense retrieval is efficient but less accurate, and that graph-based RAG hits construction walls before deployment scale. A mechanism control shows that swapping raw file search for BM25 inside the same agent harness raises full-scale accuracy from 36.9 to 69.4 while using about one ninth the query tokens. A sympathetic reader would care because enterprise corpora are large and growing, so knowing which paradigm actually scales is a practical default-setting question.

Core claim

The central measured claim is a scale-dependent crossover rather than an unconditional winner: the file-system agent scores higher at the smallest shared corpus sizes, but around 10 million corpus tokens BM25 catches it, and BM25 then leads at every larger tier, with its advantage approaching 20 points at the 601-million-token full corpus. The paper argues the mechanism is corpus-wide candidate discovery: BM25 amortizes global ranking in an inverted index that grows cheaply, while the agent's sequential local file exploration pays increasing per-question cost and loses relevant branches as the corpus grows. A matched control changes only the retrieval primitive: replacing raw file-tree searc

What carries the argument

The load-bearing object is a 28-tier strictly nested corpus ladder: tiers grow by about 1.25 per rung from 1,144 documents (1.7 million tokens) to 511,959 documents (601 million tokens), while a fixed 'bedrock' of gold documents, adversarial traps, and not-found lures stays constant across all sizes. The mechanism that separates the paper's two claims (crossover and cause) is the retrieval-primitive control: the same agent harness, model, prompt, budget, and judge, with only the raw file-tree tools replaced by a BM25 search tool whose first call is forced to use the original question, guaranteeing its top-5 matches native BM25. This isolates global candidate ranking from agentic reasoning.

Load-bearing premise

The ranking depends on the benchmark's trap-design assumption that hard distractors are same-topic documents with wrong versions, dates, or decisions; if real enterprise corpora instead mislead through paraphrase without lexical overlap, BM25's measured advantage could shrink or reverse.

What would settle it

Re-run the same 28-tier ladder on a corpus built from the same questions and gold evidence but with adversarial distractors generated by paraphrasing gold documents so they share almost no surface terms with the answer, and check whether BM25's full-scale lead over the file-system agent remains outside the paired bootstrap interval; if it collapses, the scaling conclusion is an artifact of exact-match trap mining.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A BM25 index is the strongest default retriever for enterprise-scale corpora above roughly ten million tokens; agentic reasoning adds value after ranked discovery, not as a replacement for it.
  • LLM-based graph construction is hard to justify at 10^5–10^6 documents unless build cost is near-linear and relational questions dominate, since its construction-token cost is orders of magnitude above index-free retrieval.
  • Raw file-system agents degrade as corpora grow: their per-question token budget can grow by a factor of 39 at the bedrock and they reach 31% budget exhaustion at full scale.
  • Retrieval-ordering conclusions are not safe from single-corpus-size evaluations: the winner flips with scale, so benchmarks should report accuracy and cost at multiple nested sizes.
  • Dense retrieval, while cheap, stays below lexical BM25 at shared tiers in this enterprise-style corpus, so lexical first-stage ranking remains competitive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark's distractors are deliberately mined to share a topic but state the wrong version, date, or decision, the lexical advantage may be partly an artifact of that adversarial design; on corpora that mislead via paraphrase, the crossover could shift or disappear.
  • A direct test would hold the 28-tier ladder and questions fixed while varying the distractor generation rule (lexical overlap vs paraphrase vs multi-hop), and check whether the BM25-agent crossover moves.
  • The Agent+BM25 recipe — global ranking for discovery plus a small number of agentic refinement calls — looks like a practical default for large corpora, but the optimal number of agentic calls and how it grows with corpus size is left open.
  • At even larger scales (billions of tokens), either BM25's lead widens if lexical anchors persist, or a hybrid of lexical and learned ranking emerges; the paper's data only bound the regime up to 601 million tokens.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper compares four RAG paradigms—lexical, dense, graph-based, and agentic file search—on a controlled nested corpus ladder built from EnterpriseRAG-Bench. It fixes 500 questions, gold evidence, and adversarial distractors in a 1,144-document bedrock, then appends 27 strictly nested tiers up to 511,959 documents (1.7M–601M tokens). Using one reader (Qwen3.6-27B), one official judge, and token/latency metering, it reports a scale-dependent crossover: File-System Agent has higher point estimates at small tiers, BM25 overtakes it around 10M tokens and leads at all larger shared tiers, reaching about 50.5 vs 30.7 at full scale. Matched controls (Agent+BM25, graph-substrate agents, paraphrase/top-k) isolate retrieval primitive from agency and support the mechanism that global ranked discovery, followed by agentic reading, is the effective combination. Graph-based pipelines either fail to build at scale or remain below BM25.

Significance. If accepted, the result provides a practical default for enterprise RAG and a methodological template for scaling evaluations. The study's strengths are substantial: 28 strictly nested, checksummed tiers; a fixed bedrock; one reader and judge; phase-separated token metering; paired bootstrap confidence intervals; an independent judge and binary protocol; matched retrieval/harness/substrate controls; an artifact-to-claim map; and a direct audit of the trap proposal pool. The main limitation is external: the adversarial trap distribution is constructed to reward lexical exact match over semantically similar but factually wrong distractors, so the size and location of the crossover may be benchmark-specific. This does not undermine the internal validity of the measurements, but it tempers the general 'BM25 Wins at Scale' claim.

major comments (2)
  1. [§3.2 / §6.1] The load-bearing external claim depends on the trap-mining protocol. Traps are produced by BM25 top-10 and dense reranking of a BM25 top-200 pool, filtered by a single 'same entity/topic, wrong version/date/decision' criterion (§3.2). This creates distractors with high lexical overlap and factual divergence, a regime where BM25's exact matching is advantaged; §6.1 concedes 'exact matching is therefore advantaged.' The paraphrase control in Table 6/Table 14 rewrites questions, not distractors, and runs only at N=1,144 and 2,254. Thus the crossover at ~10M tokens may be a property of this adversarial distribution rather than a general scaling law. Please add a distractor-sensitivity experiment (e.g., paraphrase-based or low-lexical-overlap traps; multi-hop distractors) at shared tiers and report whether the crossover moves or disappears. Without this, the title and §6.5 recommendation over
  2. [§4.2 / Appendix D] The central measured claim 'Around 10 million corpus tokens, BM25 overtakes it' is supported only by point estimates. Appendix D states that the 10M marker is 'a rounded regime marker rather than a fitted threshold or significance boundary.' The manuscript should either add a formal test of the crossover (e.g., paired bootstrap proportions or confidence intervals for the ordering at each tier; confidence intervals around the crossing tier) or consistently use hedged language in the abstract and §4.2. As written, the abstract's 'Around 10 million' reads as a statistically established boundary.
minor comments (5)
  1. [Table 1] The caption says 'completeness' and 'document recall' are bedrock values; please state explicitly that all cost columns are bedrock values as well, to avoid ambiguity.
  2. [Figure 5 (left)] The coverage-adjusted zero assignment is clearly labeled, but the cross-scale summary would benefit from error bars or point labels; without them, uncertainty in the coverage-adjusted frontier is not visible.
  3. [§6.3] The claim that 'agentic reasoning works best after ranked discovery' rests on a single two-scale, 150-question control. Please state this scope explicitly in the main text, since it is a mechanism control rather than a broad empirical law.
  4. [§3.3] Minor typographical issues: 'Tiert' and 'nt −1,144' have awkward spacing; also 'roughly 450-fold' should be tied to 1.25^27 ≈ 444 for precision.
  5. [Table 2] The build-cost fits C(x)=a x^b report no goodness-of-fit or confidence intervals. Adding R² or residual information would strengthen the scaling-wall claims.

Circularity Check

0 steps flagged

No circularity: the scaling crossover is measured, not derived; the trap-mining overlap is an external benchmark-representativeness caveat.

full rationale

This paper is a controlled empirical scaling study, not a derivation whose output is equivalent to its input. The central claim—'Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier'—is a direct measurement on a fixed external benchmark (EnterpriseRAG-Bench) with a fixed bedrock, nested 28-tier ladder, shared reader, and official judging protocol. The only step that could look circular is the benchmark's trap-mining procedure: 'for each target question, BM25 contributes its full-corpus top-10, while dense retrieval reranks a BM25 top-200 pool... An LLM filter retains every candidate that concerns the same entity or topic as a gold document while reporting the wrong version, date, or decision' (§3.2). That design makes the adversarial distractors lexically close to BM25-preferable candidates, so it creates a benchmark-representativeness caveat about generality to other distractor distributions. It does not, however, make the measured crossover a logical consequence of the paper's own fitted parameters or definitions: no parameter is fitted to a subset of the accuracy data and then used to 'predict' the same subset; the crossover is reported as an observed point-estimate ordering ('a rounded regime marker rather than a fitted threshold or significance boundary', Appendix D). The paper also directly addresses the mining dependency with 'a direct full-corpus DenseRAG audit' (§3.2) and paraphrase/top-k controls (§6.1, Table 14), showing the advantage persists under altered wording and matched depth, and that traps are recoverable through a direct dense proposal path. There is no load-bearing self-citation chain or imported uniqueness theorem. The build-cost extrapolations in Table 2 are explicitly labeled fitted estimates and are not used as accuracy predictions ('These projections are not used as hidden accuracy estimates', Appendix D). Therefore the derivation chain is self-contained; what remains is a legitimate external-validity concern about the benchmark's adversarial distribution, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on experimental design choices rather than mathematical derivation. No parameter was fitted to produce the crossover itself, but the benchmark's adversarial construction and the agent budget are load-bearing choices that shape the result.

free parameters (3)
  • File-System Agent call budget = 80 LLM calls per question
    Chosen budget bounds sequential exploration; 31% of questions exhaust it at full scale, so the measured agentic collapse is partly budget-limited and a different budget could move the crossover.
  • Retrieval depth = top-5 chunks
    All retrievers use top-5; a top-10 control at small tiers shows BM25 still leads there, but absolute gaps could change with depth.
  • Chunking parameters = 1,200-token chunks, 100-token overlap
    Shared chunker affects all retrieval quality and is not tuned per system; different segmentation could alter retrieval comparisons.
axioms (5)
  • domain assumption EnterpriseRAG-Bench's gold labels and trap/lure sets are a valid proxy for enterprise RAG difficulty
    The central claim 'BM25 wins at scale' generalizes from this synthetic fictional-company corpus to real enterprise corpora; no real-world validation is provided.
  • domain assumption The adversarial-trap construction criterion (topic overlap plus wrong fact) is representative of realistic retrieval failure modes
    Section 3.2 defines traps under a single topicality-and-wrong-fact criterion, which favors exact-match lexical retrieval and is not independently motivated by enterprise user studies.
  • domain assumption The File-System Agent interface (list_dir, grep, read_doc, 80-call budget) captures state-of-the-art agentic retrieval
    The agent is modeled on coding-agent interfaces, but its grep-only raw search and fixed budget may understate more capable semantic or learned search policies.
  • domain assumption Qwen3.6-27B as reader and policy, with the official LLM judge, yields stable rankings
    One reader and one primary judge are used; robustness is checked with an independent judge (96.2% agreement), but all systems share the same reader, so reader-specific interactions are possible.
  • domain assumption Build-cost power-law fits extrapolate outside the measured range
    Table 2 projects LightRAG and MS-GraphRAG build costs to full corpus from fits; if the exponent changes beyond measured sizes, the construction-wall conclusions could shift.

pith-pipeline@v1.3.0-alltime-deepseek · 15162 in / 12870 out tokens · 136269 ms · 2026-08-03T01:39:49.030569+00:00 · methodology

0 comments
read the original abstract

Retrieval-augmented generation (RAG) spans lexical and dense retrieval, graph-based indexing, and agentic search, but these paradigms are usually evaluated on different benchmarks at one corpus size, leaving their accuracy-cost scaling unclear. To bridge this gap, we present a controlled study that varies corpus size along 28 strictly nested tiers spanning roughly 450-fold, while holding questions and a fixed bedrock of relevant and adversarial documents unchanged. Under one reader model and one judging protocol, we measure official accuracy, construction and query tokens, and latency. The results reveal a scale-dependent crossover rather than an unconditional winner. File-System Agent leads at the smallest shared tiers, but its sequential exploration costs 39 times more query tokens at the bedrock and becomes less effective as the search space grows. Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale. BM25 also anchors the low-cost end of the Pareto frontier without LLM-based construction. Dense retrieval remains efficient but less accurate, whereas graph-based RAG encounters construction walls before deployment scale and its scalable variants remain below BM25 at shared tiers. Overall, corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 32 linked inside Pith

  1. [4]

    SPLADE : Sparse lexical and expansion model for first stage ranking, 2021

    Thibault Formal, Benjamin Piwowarski, and St \'e phane Clinchant. SPLADE : Sparse lexical and expansion model for first stage ranking, 2021. https://arxiv.org/abs/2107.05720

    Pith/arXiv arXiv 2021

  2. [6]

    HippoRAG : Neurobiologically inspired long-term memory for large language models, 2025 a

    Bernal Jim \'e nez Guti \'e rrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. HippoRAG : Neurobiologically inspired long-term memory for large language models, 2025 a . https://arxiv.org/abs/2405.14831

    Pith/arXiv arXiv 2025

  3. [7]

    From RAG to memory: Non-parametric continual learning for large language models, 2025 b

    Bernal Jim \'e nez Guti \'e rrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From RAG to memory: Non-parametric continual learning for large language models, 2025 b . https://arxiv.org/abs/2502.14802

    Pith/arXiv arXiv 2025

  4. [9]

    Active retrieval augmented generation, 2023

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation, 2023. https://arxiv.org/abs/2305.06983

    Pith/arXiv arXiv 2023

  5. [12]

    Dense passage retrieval for open-domain question answering, 2020

    Vladimir Karpukhin, Barlas O g uz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering, 2020. https://arxiv.org/abs/2004.04906

    Pith/arXiv arXiv 2020

  6. [15]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks, 2021. https://arxiv.org/abs/2005.11401

    Pith/arXiv arXiv 2021

  7. [19]

    Graph retrieval-augmented generation: A survey, 2024

    Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph retrieval-augmented generation: A survey, 2024. https://arxiv.org/abs/2408.08921

    Pith/arXiv arXiv 2024

  8. [20]

    Measuring and narrowing the compositionality gap in language models, 2023

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023. https://arxiv.org/abs/2210.03350

    Pith/arXiv arXiv 2023

  9. [22]

    In-context retrieval-augmented language models, 2023

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmented language models, 2023. https://arxiv.org/abs/2302.00083

    Pith/arXiv arXiv 2023

  10. [23]

    The probabilistic relevance framework: BM25 and beyond

    Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3 0 (4): 0 333--389, 2009. doi:10.1561/1500000019

    doi:10.1561/1500000019 2009

  11. [25]

    Toolformer: Language models can teach themselves to use tools, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. https://arxiv.org/abs/2302.04761

    Pith/arXiv arXiv 2023

  12. [26]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. https://arxiv.org/abs/2303.11366

    Pith/arXiv arXiv 2023

  13. [28]

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions, 2023

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions, 2023. https://arxiv.org/abs/2212.10509

    Pith/arXiv arXiv 2023

  14. [35]

    2310.11511 , archivePrefix=

    Asai, Akari and Wu, Zeqiu and Wang, Yizhong and Sil, Avirup and Hajishirzi, Hannaneh , year=. 2310.11511 , archivePrefix=

    Pith/arXiv arXiv

  15. [36]

    From Local to Global: A

    Edge, Darren and Trinh, Ha and Cheng, Newman and Bradley, Joshua and Chao, Alex and Mody, Apurva and Truitt, Steven and Metropolitansky, Dasha and Ness, Robert Osazuwa and Larson, Jonathan , year=. From Local to Global: A. 2404.16130 , archivePrefix=

    Pith/arXiv arXiv

  16. [37]

    2021 , eprint=

    Formal, Thibault and Piwowarski, Benjamin and Clinchant, St. 2021 , eprint=

    2021

  17. [38]

    2023 , eprint=

    Retrieval-augmented generation for large language models: A survey , author=. 2023 , eprint=

    2023

  18. [39]

    2410.05779 , archivePrefix=

    Guo, Zirui and Xia, Lianghao and Yu, Yanhua and Ao, Tu and Huang, Chao , year=. 2410.05779 , archivePrefix=

    Pith/arXiv arXiv

  19. [40]

    Guti. From. 2025 , eprint=

    2025

  20. [41]

    2025 , eprint=

    Guti. 2025 , eprint=

    2025

  21. [42]

    2002.08909 , archivePrefix=

    Guu, Kelvin and Lee, Kenton and Tung, Zora and Pasupat, Panupong and Chang, Ming-Wei , year=. 2002.08909 , archivePrefix=

    Pith/arXiv arXiv 2002

  22. [43]

    Proceedings of the 11th international conference on World Wide Web , pages=

    Topic-sensitive pagerank , author=. Proceedings of the 11th international conference on World Wide Web , pages=

  23. [44]

    Advances in Neural Information Processing Systems , volume=

    G-retriever: Retrieval-augmented generation for textual graph understanding and question answering , author=. Advances in Neural Information Processing Systems , volume=

  24. [45]

    2021 , eprint=

    Unsupervised dense information retrieval with contrastive learning , author=. 2021 , eprint=

    2021

  25. [46]

    Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume , pages=

    Leveraging passage retrieval with generative models for open domain question answering , author=. Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume , pages=

  26. [47]

    2023 , eprint=

    Active Retrieval Augmented Generation , author=. 2023 , eprint=

    2023

  27. [48]

    2310.06770 , archivePrefix=

    Jimenez, Carlos E and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , year=. 2310.06770 , archivePrefix=

    Pith/arXiv arXiv

  28. [49]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Graph chain-of-thought: Augmenting large language models by reasoning on graphs , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  29. [50]

    2503.09516 , archivePrefix=

    Jin, Bowen and Zeng, Hansi and Yue, Zhenrui and Yoon, Jinsung and Arik, Sercan and Wang, Dong and Zamani, Hamed and Han, Jiawei , year=. 2503.09516 , archivePrefix=

    Pith/arXiv arXiv

  30. [51]

    2020 , eprint=

    Dense Passage Retrieval for Open-Domain Question Answering , author=. 2020 , eprint=

    2020

  31. [52]

    2004.12832 , archivePrefix=

    Khattab, Omar and Zaharia, Matei , year=. 2004.12832 , archivePrefix=

    Pith/arXiv arXiv 2004

  32. [53]

    and Zhang, Hao and Stoica, Ion , year=

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , year=. Efficient Memory Management for Large Language Model Serving with. 2309.06180 , archivePrefix=

    Pith/arXiv arXiv

  33. [54]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. 2021 , eprint=

    2021

  34. [55]

    Companion Proceedings of the ACM on Web Conference 2025 , pages=

    Kag: Boosting llms in professional domains via knowledge augmented generation , author=. Companion Proceedings of the ACM on Web Conference 2025 , pages=

    2025

  35. [56]

    Pyserini: An Easy-to-Use Python Toolkit to Support Replicable

    Lin, Jimmy and Ma, Xueguang and Lin, Sheng-Chieh and Yang, Jheng-Hong and Pradeep, Ronak and Nogueira, Rodrigo , year=. Pyserini: An Easy-to-Use Python Toolkit to Support Replicable. 2102.10073 , archivePrefix=

    Pith/arXiv arXiv

  36. [57]

    2405.20139 , archivePrefix=

    Mavromatis, Costas and Karypis, George , year=. 2405.20139 , archivePrefix=

    Pith/arXiv arXiv

  37. [58]

    2112.09332 , archivePrefix=

    Nakano, Reiichiro and Hilton, Jacob and Balaji, Suchir and Wu, Jeff and Ouyang, Long and Kim, Christina and Hesse, Christopher and Jain, Shantanu and Kosaraju, Vineet and Saunders, William and others , year=. 2112.09332 , archivePrefix=

    Pith/arXiv arXiv

  38. [59]

    2024 , eprint=

    Graph Retrieval-Augmented Generation: A Survey , author=. 2024 , eprint=

    2024

  39. [60]

    2023 , eprint=

    Measuring and Narrowing the Compositionality Gap in Language Models , author=. 2023 , eprint=

    2023

  40. [61]

    2307.16789 , archivePrefix=

    Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and others , year=. 2307.16789 , archivePrefix=

    Pith/arXiv arXiv

  41. [62]

    2023 , eprint=

    In-Context Retrieval-Augmented Language Models , author=. 2023 , eprint=

    2023

  42. [63]

    The Probabilistic Relevance Framework:

    Robertson, Stephen and Zaragoza, Hugo , journal=. The Probabilistic Relevance Framework:. 2009 , doi=

    2009

  43. [64]

    2023 , eprint=

    Toolformer: Language Models Can Teach Themselves to Use Tools , author=. 2023 , eprint=

    2023

  44. [65]

    2023 , eprint=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. 2023 , eprint=

    2023

  45. [66]

    2410.23875 , archivePrefix=

    Chen, Liyi and Tong, Panrong and Jin, Zhongming and Sun, Ying and Ye, Jieping and Xiong, Hui , year=. 2410.23875 , archivePrefix=

    Pith/arXiv arXiv

  46. [67]

    Butler , year=

    Yuhong Sun and Joachim Rahmfeld and Chris Weaver and Weijia Chen and Roshan Desai and Wenxi Huang and Mark H. Butler , year=. 2605.05253 , archivePrefix=

    Pith/arXiv arXiv

  47. [68]

    2510.10114 , archivePrefix=

    Luyao Zhuang and Shengyuan Chen and Yilin Xiao and Huachi Zhou and Yujing Zhang and Hao Chen and Qinggang Zhang and Xiao Huang , year=. 2510.10114 , archivePrefix=

    arXiv

  48. [69]

    Traag, V. A. and Waltman, L. and van Eck, N. J. , year=. From Louvain to Leiden: guaranteeing well-connected communities , volume=. Scientific Reports , publisher=. doi:10.1038/s41598-019-41695-z , number=

    doi:10.1038/s41598-019-41695-z

  49. [70]

    Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and others , year=

    Xingyao Wang and Boxuan Li and Yufan Song and Frank F. Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and others , year=. 2407.16741 , archivePrefix=

    Pith/arXiv arXiv

  50. [71]

    2406.04744 , archivePrefix=

    Xiao Yang and Kai Sun and Hao Xin and Yushi Sun and Nikita Bhalla and Xiangsen Chen and Sajal Choudhary and Rongze Daniel Gui and others , year=. 2406.04744 , archivePrefix=

    Pith/arXiv arXiv

  51. [72]

    2024 , eprint=

    MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models , author=. 2024 , eprint=

    2024

  52. [73]

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and others , year=. Judging. 2306.05685 , archivePrefix=

    Pith/arXiv arXiv

  53. [74]

    2210.03629 , archivePrefix=

    Shunyu Yao and Jeffrey Zhao and Dian Yu and Nan Du and Izhak Shafran and Karthik Narasimhan and Yuan Cao , year=. 2210.03629 , archivePrefix=

    Pith/arXiv arXiv

  54. [75]

    Manning , year=

    Parth Sarthi and Salman Abdullah and Aditi Tuli and Shubh Khanna and Anna Goldie and Christopher D. Manning , year=. 2401.18059 , archivePrefix=

    Pith/arXiv arXiv

  55. [76]

    2024 , eprint=

    Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph , author=. 2024 , eprint=

    2024

  56. [77]

    Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik Narasimhan and Ofir Press , year=

    John Yang and Carlos E. Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik Narasimhan and Ofir Press , year=. 2405.15793 , archivePrefix=

    Pith/arXiv arXiv

  57. [78]

    2023 , eprint=

    Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions , author=. 2023 , eprint=

    2023