Pith. sign in

REVIEW 4 major objections 6 minor 43 references

DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small language models gain up to 27.7% from teacher-generated evidence

desk verdict Teacher-generated evidence in the appendix visibly names the correct answer, so the headline accuracy gains are uninterpretable until a leakage-control experiment rules out oracle copying. read the letter →

arxiv 2506.01954 v1 pith:SVYDKV72 submitted 2025-06-02 cs.CL cs.AIcs.LG

Jennifer Chen , Aidar Myrzakhan , Yaxin Luo , Hassaan Muhammad Khan , Sondos Mahmoud Bsharat , Zhiqiang Shen This is my paper
classification cs.CLcs.AIcs.LG
keywords RAGdistillationsmalllanguagemodelsknowledgegraphevidencerankinghallucinationmitigationmultiple-choicequestionansweringprivacy-preservingretrievalzero-shotinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a small language model (SLM) can gain retrieval-augmented capability from a large language model (LLM) without any fine-tuning, purely by conditioning at inference on evidence and knowledge-graph triples generated by a larger teacher. If the claim holds, resource-constrained deployments could get near-teacher factual accuracy while keeping a small model's efficiency. The authors report consistent double-digit accuracy gains over strong RAG baselines such as MiniRAG on MedMCQA, MMLU, GPQA, and ARC-Challenge, with up to 27.7% on ARC-Challenge, and lower hallucination on a fact-verification benchmark.

What carries the argument

The central object is the four-stage DRAG pipeline. First, the teacher generates N textual evidences for a question. Second, evidences are ranked by an equal-weight combination of cosine similarity (via sentence embeddings) and an LLM relevance score, keeping the top K. Third, the filtered evidences are converted into a knowledge graph of entity-relationship triples, with duplicate edges aggregated into a simple graph. Fourth, the small student is prompted with the ranked evidences and/or the top K triples to produce the answer. The graph representation exists to cut inference tokens (roughly 18% shorter than raw evidence in the paper's statistics) while preserving the relational facts the student needs.

What would settle it

Take a random sample of ARC-Challenge and MedMCQA questions, run DRAG's evidence generation, and ask independent annotators (or a second LLM) to pick the correct option using only the evidence and graph triples, without showing the question's option labels; if accuracy on this evidence-only guessing test is well above chance, the reported gains are contaminated by answer leakage and the distillation claim fails as stated.

Watch

Extended reading notes

Core claim

DRAG's central discovery claim is that retrieval-augmented reasoning can be distilled as evidence rather than as answers: a teacher LLM generates candidate factual snippets and entity-relationship triples, ranks them, and the small model answers using only that context. On the paper's benchmarks, this lifts small models close to the teacher's own accuracy (e.g., Gemma-2-9B reaches 94.73% on ARC-Challenge and 77.80% on MMLU with GPT-4o evidence) while the student stays small and runs zero-shot. The authors also claim reduced hallucination on the AVERITEC fact-verification task and a privacy-preserving variant that strips personally identifiable information from queries before sending them to the cloud.

Load-bearing premise

The whole benchmark story assumes the teacher's evidence and graph triples do not reveal the correct answer, so that improved multiple-choice scores reflect the student reasoning over neutral facts rather than just copying the answer from the context.

Editorial extensions

If this is right

  • If DRAG's claims hold, a 2B-3B model can reach accuracy within a few points of GPT-4o on ARC-Challenge and MedMCQA, making high-accuracy QA feasible on edge devices.
  • Evidence-only context is the strongest configuration in most experiments, so the simplest variant of the method is also its best-performing one.
  • The graph representation lowers token cost by about 18% with only small accuracy differences, which matters for latency- and cost-sensitive deployments.
  • The privacy use case suggests a general recipe: send de-identified queries to a cloud teacher and keep answer generation local, which the authors validate by removing 95.7% of injected PII while retaining most of the accuracy gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct consequence the authors do not draw: if leakage-free evidence is achievable, the same pipeline could be applied to open-domain generation, not just multiple-choice QA, wherever a teacher can produce neutral factual context.
  • The paper's own limitation note admits a potential risk of answer leakage, and the appendix examples show evidence that states the correct option's content; a leakage-controlled re-evaluation is the natural next test of the method.
  • Because graph triples are shorter and more abstract than raw evidence, they may leak less of the exact answer wording; comparing leakage rates between the evidence-only and graph-only settings would isolate which representation truly transfers reasoning.
  • The privacy benchmark's PII removal is done by the SLM itself; testing whether the DRAG pipeline still works when PII removal is imperfect (e.g., rare names or indirect identifiers) would extend the claim to realistic settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. DRAG proposes a fine-tuning-free framework that distills RAG capabilities from large language models into small language models by generating ranked textual evidence and knowledge-graph triples with a teacher model and conditioning the student on this context at inference time. The paper claims large accuracy gains on multiple-choice benchmarks (up to +27.7% over MiniRAG on ARC-Challenge), reduced hallucination on fact verification, and a privacy-preserving use case with a new PII-injection benchmark. The method is evaluated across several teacher and student model combinations, with ablations over the number of evidence pieces and graph relationships.

Significance. If the central claim were sound, DRAG would be a practical contribution to deploying SLMs in resource-constrained settings without fine-tuning, and the privacy-protection extension is an interesting direction. The paper has strengths: it releases code, evaluates multiple teacher and student model pairs, and includes ablations over K and N. However, the main benchmark evidence is compromised by answer leakage in the teacher-generated evidence, which is acknowledged in the Limitations but not controlled for. The reported gains therefore cannot be attributed to knowledge transfer or hallucination mitigation, and the inconsistency between Section 4.4 and Table 2 further undermines confidence in the numbers. The significance rests on a verification that the paper does not provide.

major comments (4)
  1. [Section 4.4 / Appendix I / Limitations] The teacher-generated evidence and graph triples leak the correct multiple-choice option, making the headline improvements uninterpretable as knowledge transfer. In Appendix I.1, the gluconeogenesis question has correct option (a) "Leucine and lysine"; the generated evidence states "Leucine and lysine are the only exclusively ketogenic amino acids, meaning they cannot serve as substrates for gluconeogenesis," and the filtered graph triples are "Leucine cannot be used as a substrate for gluconeogenesis" and "Lysine cannot be used as a substrate for gluconeogenesis," both directly naming the answer. The evidence-generation prompt in Appendix F instructs the teacher "Do not give the answer away directly," but the teacher is given the full question with options, so it can tailor evidence to the correct choice. The Limitations section concedes "there is still a potential risk of unintended leakage." No control experiment is reported to measure how often the evidence alone identifies the correct option. Without such a leakage-controlled baseline (e.g., evidence-only classification by an independent model, or answer-option permutation), the claimed gains over MiniRAG and original SLMs can be explained by oracle answer leakage rather than by the student's improved reasoning or knowledge transfer.
  2. [Section 4.4 vs Table 2] The baseline numbers for Gemma-2-9b-it in Section 4.4 are inconsistent with Table 2. Section 4.4 states that Gemma-2-9b-it improves on ARC-C "from 53.71% ... to 94.73%," but Table 2 reports the Original ARC-C score for gemma-2-9b-it as 63.27%, while 53.71% is the Original score for gemma-2-2b-it. The sentence conflates the two student models, and if 53.71% is meant to be Gemma-2-9b-it's baseline, it contradicts Table 2. This inconsistency affects the paper's representative example of knowledge transfer and must be corrected and re-evaluated.
  3. [Tables 1-4, 6, 8-17] All accuracy results are reported as single-run numbers without error bars, confidence intervals, or significance tests. Several comparisons in the ablation tables differ by less than half a percentage point (e.g., Table 2, Phi-3.5 Evidence Only at 15 vs 20 is 94.01 vs 94.10; Table 11, gemma-2-9b-it Evidence 15 vs 20 is 77.15 vs 77.80). This makes claims such as "~15 evidence pieces generally provide the optimal cost-accuracy trade-off" and the teacher-model ranking (GPT-4o > Claude > DeepSeek > Llama > Gemini) unsupported by statistical evidence. The authors should provide repeated-run variability or at least claim only the qualitative pattern.
  4. [Table 1 and Section 4.2] The comparison against MiniRAG is not controlled for the source of retrieved context. MiniRAG retrieves from a corpus, while DRAG's evidence is generated by GPT-4o after seeing the full question and answer options. The reported gains of up to +27.7% on ARC-C therefore conflate the DRAG pipeline with the effect of supplying the student with oracle-derived evidence. To substantiate the claim that DRAG outperforms MiniRAG "using the same models," the baseline would need comparable per-query evidence access, or the comparison should be framed as a teacher-oracle upper bound rather than a method comparison.
minor comments (6)
  1. [Abstract] The phrase "hallucinated content from Humans" is unclear and likely should be "hallucinated content from human-authored sources" or similar; as written it reads as a non-sequitur.
  2. [Section 3.2, Algorithm 1] The notation in Algorithm 1 is inconsistent: Step 2 computes s_i = score_i + rank_LLM(d_i), but the selection of the top K is described as discarding the lowest-scoring X = N-K evidences; the text does not state whether the LLM ranking score is normalized to the same scale as the cosine similarity before summing, which matters for the combined score.
  3. [Section 4.1] The evaluation setup says "4xRTX 4090 GPUs" but does not specify how many runs or seeds were used; please state the number of evaluation runs and the variance if any.
  4. [Table 3] The column headers for Table 3 list teacher models (GPT-4o, Llama 3.3 70b, etc.) but the meaning of "Graph," "Evide.," and "Comb." under each teacher is not explained in the caption; add a footnote describing the three configurations.
  5. [Appendix G] The privacy benchmark construction says 15,090 PIIs were injected, but Section 4.7 says only 649 remain after filtering; the paper does not report the accuracy drop caused by the redaction step separately from the DRAG gains, which would help isolate the privacy-utility trade-off.
  6. [References] The reference list omits a citation for the Open-LLM-Leaderboard dataset (Myrzakhan et al., 2024) despite it being used in Section 4.5; please check that all datasets and tools cited in the text appear in the bibliography.

Circularity Check

1 steps flagged · score 6.0 of 10

Teacher-generated evidence leaks the correct option (Appendix I), and the Limitations concede the risk, so reported accuracy gains partly reduce to answer leakage rather than knowledge transfer.

  1. fitted input called prediction [Limitations; Appendix I, Example 1; Appendix F evidence generation prompt]
    "Leucine and lysine are the only exclusively ketogenic amino acids, meaning they cannot serve as substrates for gluconeogenesis. ... In DRAG, when generating evidence, we aim to prevent data/answer leakage by instructing the model explicitly in the prompt with “do not give the answer away directly”. However, despite this precaution, there is still a potential risk of unintended leakage."

    The teacher receives the full multiple-choice question including option letters and is asked to generate evidence for it; the appendix example shows the generated evidence names the correct option (“Leucine and lysine”), and the filtered graph triple states “Leucine cannot be used as a substrate for gluconeogenesis.” The student is then scored on whether it selects that option after reading this evidence. Therefore the reported accuracy gain measures how well the teacher can write answer-bearing evidence, not whether the student acquired transferable RAG knowledge.

full rationale

The central derivation is otherwise self-contained: the method is not fitted to test data, no uniqueness theorem is imported, and the external benchmarks (ARC-C, MedMCQA, MMLU, GPQA) provide independent support. However, the main evaluation protocol is circular in the leakage sense: teacher evidence is generated from the same question that defines the correct answer, and the appendix shows this leakage occurring. The Limitations section explicitly acknowledges the risk, so this is not an external artifact. This prevents the headline gains from being read as knowledge transfer, although it does not prove that all gains are leakage; hence partial circularity score 6. The Open-LLM-Leaderboard citation is self-citational (Myrzakhan, Bsharat, and Shen are co-authors) but is not load-bearing for the central claim, since external benchmarks carry the argument.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical result rests on three unproven assumptions about teacher-generated evidence (relevance, non-leakage, ranking quality) plus two tuned hyperparameters (N and K). No new entities are postulated.

free parameters (3)
  • Number of generated evidences N = up to 20, optimal around 15
    Ablation in Section 4.4 and Appendix D selects about 15 as the cost-accuracy optimum; performance peaks there.
  • Top-K relationships and evidences kept = 5 to 20, best around 15
    Filtering threshold chosen by ablation on ARC-C and MedMCQA; not derived from theory.
  • Evidence ranking combination weights = equal weights (0.5/0.5)
    Equation 4 uses an equally weighted sum of cosine similarity and LLM rank; no justification or fitting beyond convention.
assumptions (3)
  • domain assumption Teacher LLMs generate factually relevant evidence from parametric knowledge without needing external retrieval
    Section 3.2 Step 1 replaces document retrieval with GPT-4o generation; if the teacher's parametric knowledge is wrong or incomplete, the evidence is wrong.
  • domain assumption The instruction 'do not give the answer away directly' prevents answer leakage into evidence
    Invoked in the prompt in Appendix F and in Limitations; the paper's own examples show the assumption is violated.
  • domain assumption Semantic similarity plus LLM rank identifies the most useful evidence for the student
    Equation 4 defines the combined score as an unweighted sum; no theoretical or empirical justification that this ranking maximizes student accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation." pith.science (2026). https://pith.science/paper/SVYDKV72

@misc{pith2026250601954,
  author       = {Pith},
  title        = {Pith review of: DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVYDKV72}},
  note         = {Machine review of arXiv:2506.01954}
}
abstract

Retrieval-Augmented Generation (RAG) methods have proven highly effective for tasks requiring factual consistency and robust knowledge retrieval. However, large-scale RAG systems consume significant computational resources and are prone to generating hallucinated content from Humans. In this work, we introduce $\texttt{DRAG}$, a novel framework for distilling RAG knowledge from large-scale Language Models (LLMs) into small LMs (SLMs). Our approach leverages evidence- and knowledge graph-based distillation, ensuring that the distilled model retains critical factual knowledge while significantly reducing model size and computational cost. By aligning the smaller model's predictions with a structured knowledge graph and ranked evidence, $\texttt{DRAG}$ effectively mitigates hallucinations and improves factual accuracy. We further present a case demonstrating how our framework mitigates user privacy risks and introduce a corresponding benchmark. Experimental evaluations on multiple benchmarks demonstrate that our method outperforms the prior competitive RAG methods like MiniRAG for SLMs by up to 27.7% using the same models, preserving high-level efficiency and reliability. With $\texttt{DRAG}$, we provide a practical and resource-efficient roadmap to deploying enhanced retrieval and generation capabilities in small-sized LLMs.

Figures

Figures reproduced from arXiv: 2506.01954 by the authors.

Figure 1
Figure 1. Framework Overview of Our Evidence- and Graph-based RAG Distillation. Given a user query (top-left), the approach first retrieves and filters evidence by collecting relevant text snippets. Then, these references are fed into relationship filtering and ranking using an LLM and cosine similarity to yield high-quality ordered references. The resulting multigraph RAG structure is then converted into a simplified RAG gra… view at source ↗
Figure 2
Figure 2. Effect of retrieved graph-based and evidence-based RAG on multiple-choice QA tasks. We evaluate different retrieval strategies: Graph Only, Evidence Only, and the Original LLM across four benchmarks (ARC-C, GPQA, MedMCQA, and MMLU) using various backbone models. The x-axis represents the number of retrieved items, while the y-axis denotes accuracy (%). Open-LLM Leaderboard Original DRAGG DRAGE DRAGC Phi-3.5-mini-ins… view at source ↗
Figure 3
Figure 3. More results on Retrieval Strategies. We evaluate different retrieval strategies: Graph Only, Evidence Only, and the Original LLM across four benchmarks (ARC-C, GPQA, MedMCQA, and MMLU) extended on Llama3.2-3B-it and Gemma-2-9B-it benchmarks. In this figure, the x-axis represents the number of retrieved items, while the y-axis denotes accuracy (%). Target SLM Original Graph Only Evidence Only Graph and Evidence Comb… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Raviteja Anantha, Tharun Bethi, Danil Vodianik, and Srinivas Chappidi. 2023. Context tuning for retrieval augmented generation. arXiv preprint arXiv:2312.05708

    arXiv 2023

  3. [3]

    Anthropic . 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

    work page 2024

  4. [4]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511

    arXiv 2023

  5. [5]

    Yuri Fa c anha Bezerra and Li Weigang. 2025. Llmquoter: Enhancing rag capabilities through efficient quote extraction from large contexts. arXiv preprint arXiv:2501.05554

    work page Pith review arXiv 2025

  6. [6]

    Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning, 11(23-581):81

    work page 2010

  7. [7]

    Qinyuan Cheng, Xiaonan Li, Shimin Li, Qin Zhu, Zhangyue Yin, Yunfan Shao, Linyang Li, Tianxiang Sun, Hang Yan, and Xipeng Qiu. 2024. Unified active retrieval for retrieval augmented generation

    work page 2024

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

Show all 43 references
  1. [9]

    Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval

    2009

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130

    2024 arXiv

  4. [12]

    Tianyu Fan, Jingyuan Wang, Xubin Ren, and Chao Huang. 2025. Minirag: Towards extremely simple retrieval-augmented generation. arXiv preprint arXiv:2501.06713

    2025 arXiv

  5. [13]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  6. [14]

    Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630

    2024 arXiv

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  8. [16]

    Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. 2024. Grag: Graph retrieval-augmented generation. arXiv preprint arXiv:2405.16506

    2024 arXiv

  9. [17]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  10. [18]

    Gautier Izacard and Edouard Grave. 2020. Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:2012.04584

    2020 arXiv

  11. [19]

    Pengyue Jia, Derong Xu, Xiaopeng Li, Zhaocheng Du, Xiangyang Li, Xiangyu Zhao, Yichao Wang, Yuhao Wang, Huifeng Guo, and Ruiming Tang. 2024. Bridging relevance and reasoning: Rationale distillation in retrieval-augmented generation. arXiv preprint arXiv:2412.08519

    2024

  12. [20]

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  13. [21]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Adva...

    2020

  14. [22]

    Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In ACL

    2018

  15. [23]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  16. [24]

    Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Cehao Yang, Jiaxin Mao, and Jian Guo. 2024. Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. arXiv preprint arXiv:2407.10805

    2024 arXiv

  17. [25]

    Qiheng Mao, Zemin Liu, Chenghao Liu, Zhuo Li, and Jianling Sun. 2024. Advancing graph representation learning with large language models: A comprehensive survey of techniques. arXiv preprint arXiv:2402.05952

    2024 arXiv

  18. [26]

    Costas Mavromatis and George Karypis. 2024. Gnn-rag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139

    2024 arXiv

  19. [27]

    Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. 2024. Open-llm-leaderboard: From multi-choice to open-style questions for llms evaluation, benchmark, and arena. arXiv preprint arXiv:2406.07545

    2024 arXiv

  20. [28]

    OpenAI. 2024. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o mini: Advancing cost-efficient intelligence . Accessed: 2025-02-14

    2024

  21. [29]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning. PMLR

    2022

  22. [30]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP-IJCNLP, pages 3982--3992

    2019

  23. [31]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  24. [32]

    Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. Averitec: A dataset for real-world claim verification with evidence from the web. Advances in Neural Information Processing Systems

    2023

  25. [33]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024 a . Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  26. [34]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024 b . Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  27. [35]

    Xintao Wang, Qianwen Yang, Yongting Qiu, Jiaqing Liang, Qianyu He, Zhouhong Gu, Yanghua Xiao, and Wei Wang. 2023. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv:2308.11761

    2023 arXiv

  28. [36]

    Wolfram Alpha . 2025. https://mathworld.wolfram.com/SimpleGraph.html Simple Graph . Accessed: 2025-02-14

    2025

  29. [37]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025

    2023 arXiv

  30. [38]

    Ran Xu, Hui Liu, Sreyashi Nag, Zhenwei Dai, Yaochen Xie, Xianfeng Tang, Chen Luo, Yang Li, Joyce C Ho, Carl Yang, et al. 2024. Simrag: Self-improving retrieval-augmented generation for adapting large language models to specialized domains. arXiv preprint arXiv:2410.17952

    2024 arXiv

  31. [40]

    Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024 b . Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884

    2024 arXiv

  32. [41]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  33. [42]

    Yun Zhu et al. 2024. Longrag: Enhancing retrieval-augmented generation with long-context llms. arXiv preprint arXiv:2406.15319

    2024 arXiv

  34. [43]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  35. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.