Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A minimal dense-retrieval pipeline built on a compact, diverse 380-billion-word datastore consistently improves accuracy on reasoning-intensive benchmarks, with relative gains up to 33%.

desk verdict A useful, releasable datastore with a clear overclaim: the 'consistent gains' headline is contradicted by the paper's own 70B GPQA numbers, and the evaluation protocol needs variance and a random-context control. read the letter →

arxiv 2507.01297 v2 pith:NMNRHNFR submitted 2025-07-02 cs.CL cs.IR

Xinxi Lyu , Michael Duan , Rulin Shao , Pang Wei Koh , Sewon Min This is my paper
classification cs.CLcs.IR
keywords retrieval-augmentedgenerationdenseretrievaldatastoreconstructionweb-scalereasoningbenchmarksMMLUProGPQAMATH
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to overturn the common assumption that retrieval-augmented generation does not help reasoning-intensive benchmarks. It argues that prior minimal-RAG attempts were handicapped by the datastore: Wikipedia-based stores are too narrow, while trillion-token web stores are too large to deploy. It introduces COMPACT DS, a 380.5-billion-word datastore assembled from filtered web crawl plus math, academic, educational, and forum sources, and shows that simply retrieving and prepending the top-k passages improves accuracy on MMLU, MMLU Pro, AGI Eval, GPQA, and MATH. The reported relative gains over no retrieval reach 10% on MMLU, 33% on MMLU Pro, 14% on GPQA, and 19% on MATH, holding across models from 8B to 70B parameters. The point of the work is that a well-constructed, self-contained datastore can make minimal retrieval competitive with web search and complex agentic systems.

What carries the argument

The central object is COMPACT DS, a 380.5-billion-word datastore of 639 million documents chunked into 1.9 billion 256-word passages, composed of a heavily filtered Common Crawl subset together with Wikipedia, books, educational text, math corpora, academic papers, GitHub, Stack Exchange, and Reddit. The carrying mechanism is two-stage retrieval: an in-memory inverted-file product-quantization (IVFPQ) approximate nearest-neighbor index with the Contriever-MSMARCO encoder retrieves K candidate passages, and an on-disk exact inner-product search over GRIT LM-7B embeddings re-ranks them before the top k are prepended to the query. This split is what lets web-scale coverage run on a single 456GB-RAM node with subsecond latency while recovering the accuracy that approximate search alone would lose.

What would settle it

Re-run the pipeline on the full MMLU and MMLU Pro test sets with k chosen on a separate validation split and with a no-retrieval control that receives equally many neutral context tokens; if the accuracy advantage collapses toward zero or reverses, the central claim that COMPACT DS provides consistent retrieval gains is not supported.

Watch

Extended reading notes

Core claim

The central claim is that the missing ingredient for successful RAG on reasoning-heavy benchmarks is a web-scale datastore that matches the breadth of pretraining data and is practically usable. COMPACT DS achieves this by aggressively filtering low-quality web content and combining diverse sources, and a two-stage approximate-then-exact retrieval design keeps it searchable on a single node. Using only dense retrieval followed by concatenating the top passages to the prompt, the paper reports consistent accuracy gains across LLaMa 3.1 8B, Mistral 7B, Qwen3 8B, LLaMa 3.3 70B, and QwQ 32B, and shows that the datastore matches or beats Google Search on all benchmarks and matches or beats the agentic system Search-o1 on GPQA Diamond and MATH-500. The paper further claims that no single data source suffices, that diversity across sources is load-bearing, and that exact re-ranking with a more expressive encoder materially widens the gains over ANN-only retrieval.

Load-bearing premise

The load-bearing premise is that the paper's evaluation protocol represents real-world use: most benchmarks are measured on sampled subsets of 100 questions per category, the same benchmark sets are used to pick the number of retrieved passages and the datastore composition, and the no-retrieval baseline does not receive the extra context tokens, so the claimed gains stand only if those choices generalize to full and unseen benchmarks.

Editorial extensions

If this is right

  • Minimal dense retrieval—retrieve, prepend, generate—is sufficient to improve MMLU, MMLU Pro, AGI Eval, GPQA, and MATH across model families and sizes from 8B to 70B, with the largest relative gains on MMLU Pro at 33%.
  • An in-house datastore of 380.5B words can match or beat a commercial web search engine on these benchmarks, and can match or beat an agentic RAG system on GPQA Diamond and MATH-500, while remaining self-contained and reproducible.
  • Datastore diversity is load-bearing: no single source suffices, and removing even long-tail sources such as ArXiv, Books, GitHub, or Reddit degrades performance on at least some tasks.
  • Approximate-then-exact retrieval with different encoders adds accuracy over ANN alone, yet a four-times smaller index costs only about one percentage point on average, so memory and performance can be traded flexibly.
  • Oracle reranking lifts the average relative gain from 14.5% to 32.6%, showing that the retrieved passages contain substantially more signal than the tested generators currently exploit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if these datastore-driven gains generalize, prior negative results about retrieval for reasoning may reflect datastore narrowness or low quality rather than a fundamental incompatibility, and reasoning-benchmark leaderboards may understate what external text can contribute.
  • Editorial extension: the paper's own results show PDF-only web retrieval nearly matching web-plus-PDF on MMLU Pro and exceeding it on GPQA, so adding web-hosted lecture notes and problem-set PDFs to COMPACT DS is a concrete, testable way to extend the gains.
  • Editorial extension: merging COMPACT DS and search-engine results already gives small consistent gains, so a learned or rule-based fusion of a local datastore with a search engine is a promising next step for production retrieval pipelines.
  • Editorial extension: the 13-gram decontamination step moves MATH and GPQA in the opposite direction from MMLU, suggesting contamination interacts with reasoning tasks in task-specific ways; a controlled contamination sweep could map where retrieval gains are real versus contamination-driven.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces CompactDS, a 380.5B-word retrieval datastore built from web crawls, Wikipedia, books, educational text, math, academic papers, GitHub, and Q&A forums, and combines it with a two-stage retrieval pipeline: in-memory IVFPQ ANN search followed by exact inner-product reranking with a stronger encoder. The authors evaluate a minimal RAG pipeline on MMLU, MMLU Pro, AGI Eval, GPQA, and MATH across several instruction-tuned models from 7B to 70B, reporting consistent accuracy gains and relative improvements of 10--33% on several benchmarks. They also compare CompactDS with Google Search and with the agentic Search-o1 system on GPQA Diamond and MATH-500, and release the datastore and pipeline.

Significance. If the headline claims survive scrutiny, this would be a practically valuable contribution: a released, single-node-deployable web-scale datastore, a clean approximate-then-exact retrieval recipe, and a direct comparison showing that simple dense retrieval can compete with commercial search and agentic RAG on reasoning-heavy benchmarks. The paper also contains useful engineering ablations (index size, source composition, decontamination, reranking) and is transparent about many implementation details. However, the evaluation protocol has several load-bearing weaknesses: the central 'consistent improvements across all benchmarks and model sizes' claim is contradicted by the paper's own 70B GPQA results, there is no control for simply adding extra context tokens, and the reported numeric gains contain inconsistencies with the tables. These issues need to be addressed before the headline conclusions can be accepted.

major comments (5)
  1. [Abstract, §4.3, Table 5, §6] The abstract and conclusion claim 'consistent accuracy improvements across all benchmarks and model sizes' and 'consistent and significant gains across all datasets and model sizes,' but Table 5 shows the opposite for LLaMa 3.3 70B Instruct on GPQA with k=10: Physics drops from 64.2 to 62.0, Biology from 78.2 to 73.1, and Chemistry from 50.8 to 45.4. Section 4.3 itself concedes that 'GPQA is the one exception where COMPACT DS offers no improvement at 70B.' The central claim as written is therefore false and must be re-scoped to 'most benchmarks' or 'all benchmarks except GPQA at 70B,' with the exception stated in the abstract and conclusion.
  2. [Table 1 and Abstract] The 'Relative gains from No Retrieval' row in Table 1 is internally inconsistent with the table's own values. For example, GPQA Physics is reported as +36.3% but the table shows either 26.7→33.2 (+24.3%) or 26.7→29.4 (+10.1%) depending on which CompactDS row is used; MMLU Social Sciences is reported as +8.3% but 78.7→83.2 is only +5.7%; and AGI Eval is reported as +7.1% but 56.2→58.9 is +4.8%. The abstract's '14% on GPQA' likewise does not follow from the table: using the best per-category values in the two CompactDS rows gives roughly +9.8% micro-averaged or +12.4% category-averaged. These numbers need to be recomputed and the abstract/tables corrected, because the headline relative gains are the paper's main quantitative contribution.
  3. [§3.3, §4.2, Tables 1 and 5] There is no control for the effect of adding extra context tokens: the no-retrieval baseline does not receive retrieved passages, while the CompactDS condition prepends k passages. Any observed improvement could partly be an artifact of longer prompts, additional in-context examples, or a regularization effect. The authors should add a random-context control that inserts k passages of similar length from the same or an unrelated datastore, and ideally also a token-count-matched baseline. This control is necessary to attribute the gains to retrieval relevance rather than to context augmentation per se.
  4. [§4.1, §C.1, §C.2, §3.1] The evaluation protocol selects important hyperparameters and datastore composition on the same test benchmarks that are later used to report gains. Specifically, k is chosen using Figure 1 in §C.1 on these benchmarks; source composition is justified via leave-one-out results in Table 13 (§C.2); and the FineWeb-Edu threshold and decontamination Jaccard threshold in §3.1 appear to be chosen on the basis of the same evaluation sets. Without a held-out validation split or a pre-registered protocol, the reported improvements are optimistically biased. The authors should either provide a development/test split or report sensitivity of the main results across a plausible range of these choices.
  5. [§B, Table 1, Table 5] Most benchmarks are evaluated on a subsample of 100 questions per fine-grained category, and no confidence intervals or statistical tests are reported. On category sizes of roughly 100--200 questions, differences of 2--4 accuracy points (e.g., several MMLU subsets, and most GPQA rows in Table 5) are within sampling noise. The paper should report standard errors or binomial confidence intervals, or run the full benchmark, at least for the central comparisons used to support the 'consistent gains' claim.
minor comments (5)
  1. [§1, §4.1] There are typos that should be corrected: 'mininal' in the introduction, 'demends' in §4.1, and inconsistent rendering of CompactDS as 'COMPACT DS,' 'COMPACTDS,' and 'C OMPACT DS' throughout.
  2. [Figure 1, §C.1] The caption and axes of Figure 1 should be explicit: the x-axis is 'number of retrieved passages k,' and the curves should be labeled with the benchmark names and the metric used. The 'X' markers for the reordering method are described in the text but not defined in the caption.
  3. [Table 5 caption] The caption states 'Gains are consistent across different model sizes and families,' but the table itself shows large GPQA decreases for LLaMa 3.3 70B and Mistral 7B (GPQA Biology 44.9→38.5). The caption should be revised to match the data.
  4. [§5.1, §A.3] The cross-reference 'See §A for additional details' should point to §A.3, where the search engine aggregation strategies are actually described.
  5. [§2 and §4.2] The term 'web-scale datastore' is used loosely for a 380.5B-word corpus that is a heavily filtered subset of Common Crawl plus curated sources; given the 456GB RAM and large disk requirement, the 'single-node deployable' claim should state the full hardware configuration (RAM, disk, index size) that the claims refer to.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported gains are empirical measurements, not quantities defined by the construction; tuning on the evaluation benchmarks is a validity concern, not a by-construction reduction.

full rationale

No significant circularity found. The paper's core derivation—build a filtered datastore, retrieve with dense ANN plus exact search, prepend passages to a generator—is an empirical pipeline evaluated on external benchmarks; the gains in Tables 1, 5, 8, and 9 are measurements, not quantities defined by the construction. The choices of k (Figure 1, Section C.1), datastore composition (Table 13), and chunk size following [8] are tuned on the same benchmark sets used for reporting, which is an evaluation-leakage/overfitting concern rather than a circular reduction: the reported accuracies do not reduce to the tuning choices by definition, and the paper does not claim to predict held-out benchmarks. Self-citations to MASSIVE DS [8] and DPR [2] are used as baselines and as conventions for chunking and decontamination, not as load-bearing proofs of the central claim. The oracle reranking in Table 4 uses the ground-truth answer, but it is explicitly labeled an oracle upper bound, so it is not presented as a prediction. The one notable issue is an internal inconsistency: Section 4.3 concedes 'GPQA is the one exception where COMPACT DS offers no improvement at 70B' and Table 5 shows drops on GPQA Physics, Biology, and Chemistry for LLaMa 3.3 70B, while the abstract and conclusion claim 'consistent accuracy improvements across all benchmarks and model sizes.' This is a claim-scoping error, not a circularity. No equation in the paper defines a predicted quantity as equal to a fitted input, no uniqueness theorem is imported from the authors, and no ansatz is smuggled in via self-citation.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The paper's empirical claim relies on a set of choices that are not derived from first principles: datastore composition, filtering thresholds, retrieval hyperparameters, sampling, and decontamination are all selected or assumed. The ledger lists the most consequential of these.

free parameters (7)
  • k (number of retrieved passages) = 10 for main results
    Chosen from Figure 1 (Section C.1) as the point where gains saturate; gains depend strongly on k, especially for GPQA and MATH.
  • K (candidate pool for exact search) = 1000 in Table 2
    Set to 1000 for the two-stage retrieval; controls recall and latency trade-off.
  • FineWeb-Edu classifier threshold = 4.0
    Used to filter Common Crawl to the 'High-quality CC' source; threshold chosen manually and affects datastore composition.
  • Decontamination Jaccard threshold = 70% 13-gram Jaccard for CompactDS; any 13-gram overlap for search engine
    Stricter for search-engine results; affects measured performance (Table 6).
  • IVFPQ subquantizers = 256 (default), 64 for compressed variant
    Controls ANN accuracy and index size (125GB vs 456GB); Table 3 shows up to 1% average performance drop with compression.
  • Chunk size = 256 words for CompactDS, 512 words for search engine
    Passages are chunked into 256 words; for search results, c=512 chosen as cheapest with similar performance.
  • IVFPQ number of clusters and probes = Not fully specified
    FAISS hyperparameters affect ANN recall; the paper does not report exact values.
assumptions (5)
  • domain assumption Benchmark questions and answers are absent from CompactDS after decontamination.
    The paper filters paragraphs with >70% 13-gram Jaccard similarity, but residual near-duplicates or answer derivations could remain; the claim assumes decontamination is sufficient.
  • domain assumption Sampled 100 questions per category represents the full benchmark distribution.
    Most results use random subsamples (Section 4.1 and Appendix B); no confidence intervals or multiple seeds are reported.
  • domain assumption Retrieval gains are not just from added context.
    The no-retrieval baseline omits all retrieved passages; no control with random or irrelevant passages is included, so the improvement is attributed to retrieval content.
  • domain assumption Google Custom Search with static parsing is a fair web-search baseline.
    Search results are nondeterministic and depend on API costs and parsing; comparison may not reflect the best possible web retrieval.
  • domain assumption LLM and encoder choices are representative.
    The results depend on Contriever, GRIT-LM, and LLaMa/Mistral/Qwen models; other retrievers or generators may show different gains.
invented entities (1)
  • CompactDS independent evidence
    purpose: A curated 380.5B-word retrieval datastore for RAG evaluation
    The datastore itself is newly constructed but released for external use; its value beyond the five benchmarks is not yet independently demonstrated in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks." pith.science (2026). https://pith.science/paper/NMNRHNFR

@misc{pith2026250701297,
  author       = {Pith},
  title        = {Pith review of: Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMNRHNFR}},
  note         = {Machine review of arXiv:2507.01297}
}
read the original abstract

Retrieval-augmented Generation (RAG) has primarily been studied in limited settings, such as factoid question answering; more challenging, reasoning-intensive benchmarks have seen limited success from minimal RAG. In this work, we challenge this prevailing view on established, reasoning-intensive benchmarks: MMLU, MMLU Pro, AGI Eval, GPQA, and MATH. We identify a key missing component in prior work: a usable, web-scale datastore aligned with the breadth of pretraining data. To this end, we introduce CompactDS: a diverse, high-quality, web-scale datastore that achieves high retrieval accuracy and subsecond latency on a single-node. The key insights are (1) most web content can be filtered out without sacrificing coverage, and a compact, high-quality subset is sufficient; and (2) combining in-memory approximate nearest neighbor (ANN) retrieval and on-disk exact search balances speed and recall. Using CompactDS, we show that a minimal RAG pipeline achieves consistent accuracy improvements across all benchmarks and model sizes (8B--70B), with relative gains of 10% on MMLU, 33% on MMLU Pro, 14% on GPQA, and 19% on MATH. No single data source suffices alone, highlighting the importance of diversity of sources (web crawls, curated math, academic papers, textbooks). Finally, we show that our carefully designed in-house datastore matches or outperforms web search engines such as Google Search, as well as recently proposed, complex agent-based RAG systems--all while maintaining simplicity, reproducibility, and self-containment. We release CompactDS and our retrieval pipeline, supporting future research exploring retrieval-based AI systems.

Figures

Figures reproduced from arXiv: 2507.01297 by the authors.

Figure 1
Figure 1. Effect of number of retrieved pas￾sages with COMPACTDS. ‘X’ represents the points where we apply the reordering method from [49] at k = 100 [PITH_FULL_IMAGE:figures/full_fig_p020_1.png] view at source ↗
Figure 2
Figure 2. Effect of decontamination across eight single-source datastores. To quantify the effect of decontamination, we present the aggregated results from running RAG with eight single-source datastores before and after the decon￾tamination in [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation

    cs.IR 2026-06 unverdicted novelty 7.0 of 10

    PixelRAG shows that operating RAG entirely over web screenshots outperforms text-based retrieval on NQ, SimpleQA, MMSearch, LiveVQA, and MoNaCo, with up to 18.1% accuracy gains and 3x token savings via image compression.

  2. Procedural Knowledge at Scale Improves Reasoning

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    Reasoning Memory decomposes reasoning trajectories into 32 million subquestion-subroutine pairs and retrieves them via in-thought prompts to improve language model performance on math, science, and coding benchmarks b...

  3. Procedural Knowledge at Scale Improves Reasoning

    cs.CL 2026-04 conditional novelty 6.0 of 10

    Retrieving compact procedural hints from 32M subquestion–subroutine pairs improves reasoning-model accuracy on math, science, and coding benchmarks beyond compute-matched test-time scaling.

Reference graph

Works this paper leans on

70 extracted references · 35 canonical work pages · cited by 2 Pith papers

  1. [1]

    Retrieval augmented language model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning , pages 3929–3938. PMLR, 2020

    work page 2020

  2. [2]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP) , pages 6769–6781, Online, November 2020. Association for Computational Linguistics

    work page 2020

  3. [3]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. 11

    work page 2020

  4. [4]

    The web is your oyster-knowledge-intensive nlp against a very large web corpus

    Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas O ˘guz, Edouard Grave, Wen-tau Yih, et al. The web is your oyster-knowledge-intensive nlp against a very large web corpus. arXiv preprint arXiv:2112.09924, 2021

    arXiv 2021

  5. [5]

    Improving language models by retrieving from trillions of tokens

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR, 2022

    2022

  6. [6]

    Replug: Retrieval-augmented black-box language models.arXiv preprint arXiv:2301.12652, 2023

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models.arXiv preprint arXiv:2301.12652, 2023

    arXiv 2023

  7. [7]

    Retrieval-based language models and applications

    Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 6: Tutorial Abstracts), pages 41–46, 2023

    work page 2023

  8. [8]

    Scaling retrieval-based language models with a trillion-token datastore

    Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettle- moyer, and Pang Wei W Koh. Scaling retrieval-based language models with a trillion-token datastore. Advances in Neural Information Processing Systems , 37:91260–91299, 2024

    work page 2024

Show all 70 references
  1. [9]

    Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  2. [10]

    Latent retrieval for weakly supervised open domain question answering

    Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, 2019

    2019

  3. [11]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Annual Meeting of the Association for Computational Linguistics , pages 1601–1611, 2017

    2017

  4. [12]

    Kilt: a benchmark for knowledge intensive language tasks

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktaschel, and Sebastian Riedel. Kilt: a benchmark for knowledge intensive language tasks. In North American Chapter of the Assoc...

    2020

  5. [13]

    Can retriever-augmented language models reason? the blame game between the retriever and the language model

    Parishad BehnamGhader, Santiago Miret, and Siva Reddy. Can retriever-augmented language models reason? the blame game between the retriever and the language model. arXiv preprint arXiv:2212.09146, 2022

    2022 arXiv

  6. [14]

    Shangyi Geng, Wenting Zhao, and Alexander M. Rush. Great memory, shallow reasoning: Limits of knn-lms. ArXiv, abs/2408.11815, 2024

    2024 arXiv

  7. [15]

    Search-o1: Agentic search-enhanced large reasoning models

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025

    2025 arXiv

  8. [16]

    Agentic reasoning: Reasoning llms with tools for the deep research

    Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning llms with tools for the deep research. ArXiv, abs/2502.04644, 2025

    2025 arXiv

  9. [17]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  10. [18]

    Research: Learning to reason with search for llms via reinforcement learning

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. Research: Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470, 2025. 12

    2025 arXiv

  11. [19]

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025

    2025 arXiv

  12. [20]

    Zerosearch: Incentivize the search capability of llms without searching

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. Zerosearch: Incentivize the search capability of llms without searching. arXiv:2505.04588, 2025

    2025 arXiv

  13. [21]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  14. [22]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024

  15. [23]

    Agieval: A human-centric benchmark for evaluating foundation models, 2023

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023

    2023

  16. [24]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023

    2023

  17. [25]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021

    2021

  18. [26]

    Billion-scale similarity search with gpus

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547, 2019

    2019

  19. [27]

    Diskann: Fast accurate billion-point nearest neighbor search on a single node

    Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. Diskann: Fast accurate billion-point nearest neighbor search on a single node. In NeurIPS, 2019

    2019

  20. [28]

    Knn-lm does not improve open-ended text generation

    Shufan Wang, Yixiao Song, Andrew Drozdov, Aparna Garimella, Varun Manjunatha, and Mohit Iyyer. Knn-lm does not improve open-ended text generation. In Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  21. [29]

    Reasonir: Training retrievers for reasoning tasks

    Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. Reasonir: Training retrievers for reasoning tasks. arXiv preprint arXiv:2504.20595, 2025

    2025 arXiv

  22. [30]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  23. [31]

    Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Rus- sell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas ...

    2024

  24. [32]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processin...

    2024

  25. [33]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlí ˇcek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems , 37:30811–30849, 2024. 13

    2024

  26. [34]

    Ra-dit: Retrieval-augmented dual instruction tuning

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-augmented dual instruction tuning. In The Twelfth International Conference on Learning Representations , 2023

    2023

  27. [35]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019

    2019

  28. [36]

    Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan ...

    2025

  29. [37]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

    2019

  30. [38]

    Smith, Luke Zettlemoyer, Pang Wei Koh, Hannaneh Hajishirzi, Ali Farhadi, and Sewon Min

    Weijia Shi, Akshita Bhagia, Kevin Farhat, Niklas Muennighoff, Jacob Morrison, Pete Walsh, Dustin Schwenk, Shayne Longpre, Jake Poznanski, Allyson Ettinger, Daogao Liu, Margaret Li, Mike Lewis, Wen tau Yih, Dirk Groeneveld, Luca Soldaini, Kyle Lo, Noah A. Smith, Luke Zettlemoye...

    2025

  31. [39]

    Redpajama: An open source recipe to reproduce llama training dataset, April 2023

    Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, April 2023

    2023

  32. [40]

    Openwebmath: An open dataset of high-quality mathematical web text, 2023

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023

    2023

  33. [41]

    Naturalproofs: Mathematical theorem proving in natural language, 2021

    Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language, 2021

    2021

  34. [42]

    peS2o (Pretraining Efficiently on S2ORC) Dataset

    Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently on S2ORC) Dataset. Technical report, Allen Institute for AI, 2023. ODC-By, https://github.com/allenai/pes2o

    2023

  35. [43]

    Pubmed baseline 2023 repository

    National Library of Medicine. Pubmed baseline 2023 repository. https://lhncbc.nlm.nih. gov/ii/information/MBR.html, 2023

    2023

  36. [44]

    Product quantization for nearest neighbor search

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117–128, 2010

    2010

  37. [45]

    Unsupervised dense information retrieval with contrastive learning

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021

    2021 arXiv

  38. [46]

    Generative representational instruction tuning

    Niklas Muennighoff, SU Hongjin, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In ICLR 2024 Workshop: How Far Are We From AGI, 2024

    2024

  39. [47]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022

    2022

  40. [48]

    Is chatgpt good at search? investigating large language models as re-ranking agents

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.09542, 2023. 14

    2023 arXiv

  41. [49]

    Long-context llms meet rag: Overcoming challenges for long inputs in rag

    Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context llms meet rag: Overcoming challenges for long inputs in rag. arXiv preprint arXiv:2410.05983, 2024

    2024 arXiv

  42. [50]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

    2024

  43. [51]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, An- thony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Asto...

    2024 arXiv

  44. [52]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  45. [53]

    Mistral 7b

    AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  46. [54]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  47. [55]

    Qwq: Reflect deeply on the boundaries of the unknown, november 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, november 2024. URL https://qwenlm. github. io/blog/qwq-32b-preview, 2024

    2024

  48. [56]

    Webthinker: Empowering large reasoning models with deep research capability

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776, 2025

    2025 arXiv

  49. [57]

    Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl

    Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl. In Leif Azzopardi, Allan Hanbury, Gabriella Pasi, and Benjamin Piwowarski, editors, Advances in Information Retrieval. 40th European Confer...

    2018

  50. [58]

    Beautiful soup documentation

    Leonard Richardson. Beautiful soup documentation. April, 2007

    2007

  51. [59]

    olmocr: Unlocking trillions of tokens in pdfs with vision language models, 2025

    Jake Poznanski, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. olmocr: Unlocking trillions of tokens in pdfs with vision language models, 2025

    2025

  52. [60]

    Crawl4ai: Open-source llm friendly web crawler & scraper

    UncleCode. Crawl4ai: Open-source llm friendly web crawler & scraper. https://github. com/unclecode/crawl4ai, 2024

    2024

  53. [61]

    Unsupervised dense information retrieval with contrastive learning, 2022

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2022

    2022

  54. [62]

    cloudflare

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations, 2025. 17 A Method Details A.1 C OMPACT DS Statistics Table 10 reports the number of passages, number of words, and number of chunks for...

    2025

  55. [63]

    \\( 108 \\text{ cm}^3 \\) 3

    D Lesson 12-4 1. \\( 108 \\text{ cm}^3 \\) 3. \\( 26.95 \\text{ m}^3 \\) 5. \\( 206.4 \\text{ ft}^3 \\) 7. \\( 1025.4 \\text{ cm}^3 \\) 9. \\( 35.1 \\text{ cm} \\) 11. \\( 539 \\text{ m}^3 \\) 13. \\( 58.14 \\text{ ft}^3 \\) 15. \\( 1534.25 \\text{ in}^3 \\) 17. \\( 407.2 \\te...

  56. [64]

    \\( 521.5 \\text{ cm}^3 \\) 23

    \\( 2686.1 \\text{ mm}^3 \\) 21. \\( 521.5 \\text{ cm}^3 \\) 23. \\( 3934.9 \\text{ cm}^3 \\) 37. \\( 11\\frac{1}{4} \\text{ in} \\) 39. \\( 1100 \\text{ cm}^3 \\); Each triangular prism has a base area of \\( \\frac{1}{2}(8)(5.5) \\) or 22 cm\u00b2 and a height of 10 cm. 41a....

  57. [65]

    11.4 cm 59. 9.3 in. 61. 378 m\u00b2 Lesson 12-5 1. 75 in\u00b3 3. 62.4 m\u00b3 5. 51.3 in\u00b3 7. 28.1 mm\u00b3 9. 513.333 ft\u00b3 11. 105.8 mm\u00b3 13. 233.8 cm\u00b3 15. 35.6 cm\u00b3 17. 235.6 in\u00b3 19. 1473.1 cm\u00b3 21. 1072.3 in\u00b3 23. 234.6 cm\u00b3 25. 32.2 ft\u00b3

  58. [66]

    about 13,333 BTUs 31a

    3190.6 m\u00b3 29. about 13,333 BTUs 31a. The volume is doubled. 31b. The volume is multiplied by 2\u00b2 or 4. 31c. The volume is multiplied by 2\u00b3 or 8. 33. 14 in. 35a. Sample answer: 35b. The volumes are the same. The volume of a pyramid equals one third times the base ...

  59. [67]

    It gives a contribution to the angular momentum of the Dirac field

  60. [68]

    It gives a contribution to the four-momentum of the Dirac field

  61. [69]

    It generates all Poincar\u00e9 transformations of the Dirac field

  62. [70]

    Pauli matrices with two spacetime indices\

    It generates all Lorentz transformations of the Dirac field. Choices: (A) 1 and 4 (B) 2 and 4 (C) 2 and 3 (D) 1 and 3 Answer A COMPACT DS P_\\mu = i \\frac{\\partial}{\\partial x^\\mu}. \\tag{3.51} \\] From the relations derived it follows that \\(hM_k\\) is the orbital angula...

    1972

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.