Pith. sign in

REVIEW 3 major objections 6 minor 64 references

Retrieved official guidance lets small language models match or beat larger ones on public-health questions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 00:42 UTC pith:Q2ZNBYXZ

load-bearing objection Careful public-health RAG ablations: hybrid retrieval and short ranked context let smaller open models match larger closed-book ones; single-target labels and MCQA cues soften the causal claim but the practical takeaway holds. the 3 major comments →

arxiv 2607.06641 v1 pith:Q2ZNBYXZ submitted 2026-07-07 cs.CL cs.AIcs.LG

Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

Felix Feldman , Joshua Harris , Timothy Laurence , Leo Loman , Ollie Higgins , Fan Grayson , Poonam Soma , Bethany Pace-Bonello
show 2 more authors
Michael Borowitz Toby Nonnenmacher
This is my paper
classification cs.CL cs.AIcs.LG
keywords retrieval-augmented generationpublic health QAhybrid retrievalPubHealthBenchLLM-as-judgeofficial guidancechunkingfaithfulness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether grounding large language models in a maintained corpus of UK government public-health guidance can make their answers reliable enough for real use. It extends an existing multiple-choice and free-form benchmark of nearly eight thousand questions drawn from that guidance, and systematically tests how dense, sparse, and hybrid retrieval, chunk design, and context size affect both retrieval quality and downstream answers. The central result is that high-quality retrieved context is the main performance lever: it lifts multiple-choice accuracy so that smaller open-weight models match or exceed larger models used without retrieval, and the same retrieval ranking also shapes how faithful free-form answers stay to the source material. A rubric-based automated judge is introduced and checked against dual human raters; agreement is usable for faithfulness and completeness but weaker for factual consistency and clarity. The practical claim is that retrieval configuration, not model size alone, is the primary route to keeping public-health QA systems current and accurate as official guidance changes.

Core claim

When models are given retrieved chunks from the same official UK public-health guidance used to build the questions, multiple-choice accuracy rises substantially across many LLMs, allowing smaller open-weight models to match or surpass larger models run without retrieval; those gains are driven mainly by retrieval quality and careful selection of a short context window rather than by model scale alone. Hybrid dense-plus-sparse retrieval consistently improves recall and ranking, and free-form faithfulness tracks how highly the relevant chunk is ranked.

What carries the argument

Hybrid retrieval via weighted Reciprocal Rank Fusion of dense embeddings and sparse (TF-IDF/BM25) ranks over a header-chunked guidance corpus, with the top-k chunks supplied as context to the generator and free-form answers scored by a four-criterion LLM-as-judge (faithfulness, completeness, factual consistency, clarity) validated against dual human annotations.

Load-bearing premise

Every question is treated as having exactly one labelled relevant chunk, so both retrieval scores and faithfulness judgments are measured against that single target even when other passages could also answer the question.

What would settle it

Re-label a substantial sample of questions with all passages that independently support a correct answer; if hybrid retrieval and MCQA/free-form gains collapse once multi-evidence support is credited, or if faithfulness scores no longer track single-target rank, the single-target claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper extends PubHealthBench (7,929 MCQA items from UK public health guidance) into a retrieval-augmented setting. It systematically compares dense, sparse, and hybrid retrieval across embedding models and corpus variants (full, summary-only, reduced), then measures end-to-end MCQA gains for 11 LLMs and free-form quality via a four-criterion LLM-as-judge validated against dual human annotations. The central claims are that hybrid retrieval consistently improves recall and ranking, that retrieved official guidance substantially raises MCQA accuracy so smaller open-weight models can match or beat larger closed-book models, and that free-form faithfulness/completeness are the most reliable automated dimensions while factual consistency is not.

Significance. If the results hold, the paper supplies concrete, domain-grounded guidance for building RAG systems over official public health corpora: hybrid RRF fusion, chunk-length effects, k budgets, and rank sensitivity. The dual-annotated judge validation with Cohen’s κ and bootstrap CIs is a genuine methodological contribution for free-form public health QA evaluation. The finding that retrieval quality can substitute for parameter count is practically important for agencies with compute, hosting, and update-cadence constraints. Strengths include systematic α sweeps, multiple corpus variants, topic and length breakdowns, and an honest Limitations section. These are falsifiable empirical measurements against a fixed corpus rather than tautological claims.

major comments (3)
  1. Abstract and §4.2 claim gains are “primarily driven by retrieval quality,” yet Figure 4a shows high MCQA accuracy even when the labelled target is not retrieved, and §4.2 notes strong k=1 performance despite Precision@1 ≤0.80. Related Work already cites Singh et al. on MCQA option cues (6.7 ppt above chance with stem masked). The single-target axiom (§3.5.1) treats all non-target passages as noise, so the causal attribution overstates pure retrieval-of-gold-chunk effects relative to option elimination and multi-evidence support. Please either (i) add a multi-relevance or distractor-ablation analysis, or (ii) soften the abstract/§4.2 causal language to match what Figure 4 actually supports, and quantify residual accuracy without the target.
  2. Table 5 and Figure 3 evaluate Llama-3.3-70B, starred as the model used to generate the benchmark (§3.1; Table 1). Reporting 0.995 total accuracy for the generator on its own LLM-generated items is a load-bearing contamination risk for the headline “smaller models match larger models” comparison. Either exclude the generator from primary MCQA tables, report a held-out human-authored subset, or add a clear contamination analysis and demote that row from the main claim.
  3. Free-form faithfulness is scored only against the single reference chunk (§3.5.2; Limitations §5), while free-form prompts use k=15 and up to 10,000 tokens. Figure 5’s ~0.5–0.64 faithfulness pass rates therefore mix true hallucination with legitimate multi-passage synthesis. The paper correctly de-emphasizes factual consistency (Table 6: judge–human κ near zero), but still presents free-form results as generalizing MCQA gains (§4.2.2; §6.3). Restrict primary free-form claims to faithfulness/completeness with the single-evidence definition made explicit in figures and abstract, and avoid treating low faithfulness as pure model failure without a multi-evidence control.
minor comments (6)
  1. Repeated typo “free-from” for “free-form” appears in the abstract, §1 Q3, §2.3 (Singh et al. paraphrase), and §3.4.3. Standardize throughout.
  2. Equation (1): state the search grid and selection criterion for α more clearly in the main text (currently only in §3.4.1); report whether α was tuned on the full query set or a held-out split to avoid optimistic hybrid metrics in Tables 3–4.
  3. Figure 2 caption and ANOVA η² discussion: clarify whether topic effects are tested after multiple-comparison correction and whether chunk-length confounds topic (Blood Safety vs TB/travel may differ systematically in length).
  4. Table 5 topic abbreviations (HCAI/AMR, VPDs) are not expanded in the table caption; add a short key for readability.
  5. §3.2 item 4–5: confirm that summaries are never passed to the generator (only used for retrieval indexing); one sentence in the generation prompt description would prevent misreading.
  6. References and arXiv IDs are fine; fix minor formatting (e.g., “uisng” in Conclusion; “V olume” spacing in AfriMed-QA citation).

Circularity Check

1 steps flagged

Empirical RAG study with no circular derivation; only minor self-citation of prior closed-book baselines from overlapping authors.

specific steps
  1. self citation load bearing [Abstract; §4.2 MCQA with Retrieval; Table 5 footnote; citation [1]]
    "Several models also meet or surpass the best performing model without retrieved context (GPT-4.5, 92.5%) [1]. ... ∗Model used to generate the benchmark."

    The headline comparison that smaller open-weight RAG models match or beat a larger closed model without retrieval partly imports the GPT-4.5 closed-book figure and human baseline from the authors’ prior PubHealthBench paper [1], rather than re-measuring that baseline under identical conditions here. This is mild and non-load-bearing: within-paper with-vs-without retrieval and across-retriever contrasts remain independent empirical measurements, and high absolute MCQA accuracies are reported directly.

full rationale

This paper is an empirical systems evaluation of retrieval configurations and RAG generation on PubHealthBench, not a first-principles derivation. Hybrid vs dense/sparse metrics, MCQA accuracy with vs without retrieved context, rank-sensitivity plots, and free-form rubric scores are measured quantities, not quantities forced by definition or by a fitted parameter renamed as a prediction. α is selected for best retrieval on the same query set (standard optimistic hyperparameter reporting), but the paper does not present those tuned setups as independent predictions of a held-out phenomenon. Self-citation of Harris et al. [1] supplies the benchmark construction, the 88% human baseline, and the 92.5% GPT-4.5 closed-book number used for comparison; that is normal follow-up work and is not load-bearing for the within-paper contrast that retrieval raises accuracy. Evaluating Llama-3.3-70B (the benchmark generator) is a mild contamination risk, not a circular reduction of the central claim, which holds across many non-generator models. Single-target chunk labels and MCQA option cues are validity/limitation issues the paper itself flags; they do not make the reported lifts tautological. No self-definitional loop, uniqueness theorem, or ansatz smuggled via citation is present. Score 1 only for the minor self-cited closed-book baseline comparison.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claims rest on standard IR/RAG machinery plus domain evaluation choices: single-target relevance labels inherited from how questions were built, official guidance as ground truth, RRF fusion with a conventional constant, hand-chosen context budgets, and a rubric whose factual-consistency axis is weakly human-aligned. No new physical entities are postulated; free parameters are configuration knobs fitted or selected for best retrieval/QA metrics.

free parameters (4)
  • RRF fusion weight α
    Chosen per model from {0.50…0.95} to maximize retrieval metrics; best α is model-dependent and reported as part of the winning configuration.
  • RRF constant c
    Fixed at 60 following foundational RRF work; affects rank fusion scores.
  • context size k and free-form token budget
    k∈{1,3,5,10} for MCQA; k=15 and 10,000-token context for free-form—hand-chosen operational settings that shape reported accuracy and faithfulness.
  • 512-token reduced-corpus cutoff
    Chunks longer than 512 tokens (E5 tokenizer) replaced by summaries for CR; aligns with Multilingual-E5 context capacity and affects which corpus wins.
axioms (4)
  • ad hoc to paper Exactly one target chunk is the relevant document for each query for Recall/MRR/nDCG/Precision@1.
    Stated in §3.5.1; enables simple metrics but is acknowledged as incomplete for multi-passage public-health questions.
  • domain assumption Official UK government public health guidance is the authoritative ground truth for answers.
    Core of the RAG setup and faithfulness criterion; appropriate for the deployment goal but excludes multi-source clinical judgment.
  • standard math Hybrid RRF of dense cosine ranks and sparse TF-IDF/BM25 ranks is a valid way to combine signals.
    Standard IR practice (Cormack et al. RRF); used as method, not proved here.
  • domain assumption LLM-as-judge scores on the four rubric criteria approximate human expert quality judgments at scale.
    Partially validated in §4.2.1; holds better for faithfulness/completeness than factual consistency/clarity.
invented entities (1)
  • Four-criterion free-form public-health rubric (faithfulness, completeness, factual consistency, clarity) as operationalized for GPT-OSS-120B judging no independent evidence
    purpose: Automate free-form evaluation beyond MCQA and ROUGE-style metrics.
    Rubric is adapted from HealthBench-style practice but defined specifically for this benchmark; independent_evidence is partial via dual human annotation on 100 samples, not external clinical trials.

pith-pipeline@v1.1.0-grok45 · 21753 in / 3299 out tokens · 38684 ms · 2026-07-11T00:42:55.098471+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) achieve promising results on medical question answering benchmarks, yet their use in public health is constrained by hallucinations and the rapid evolution of official guidance. Retrieval-Augmented Generation (RAG) mitigates these risks by grounding responses in an explicitly maintained corpus, but end-to-end performance depends critically on retrieval configuration and on evaluation beyond multiple-choice formats. We extend PubHealthBench, a question answering (QA) benchmark of 7,929 questions derived from UK Government public health guidance, into a retrieval-augmented setting and systematically evaluate retrieval and generation choices. We compare dense, sparse, and hybrid retrieval across multiple embedding models and corpus variants, and show that hybrid retrieval consistently improves recall and ranking quality, with chunk length and topic interacting with ranking performance. Providing retrieved context substantially increases multiple-choice accuracy across a diverse set of LLMs, enabling smaller open-weight models to match or outperform larger models used without retrieval, with gains primarily driven by retrieval quality and careful context selection. To assess realistic free-form answering, we introduce a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency, and validate it against dual human annotations. Judge-human agreement is strongest for faithfulness and completeness, while factual consistency and clarity are less reliably reproduced, motivating caution when interpreting those dimensions at scale. Overall, our results highlight retrieval as a primary lever for reliable public health QA and provide practical guidance for building and evaluating RAG systems grounded in official guidance.

Figures

Figures reproduced from arXiv: 2607.06641 by Bethany Pace-Bonello, Fan Grayson, Felix Feldman, Joshua Harris, Leo Loman, Michael Borowitz, Ollie Higgins, Poonam Soma, Timothy Laurence, Toby Nonnenmacher.

Figure 1
Figure 1. Figure 1: Retrieval by target length Retrieval metrics for various setups broken down by word count of the target chunk [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Topic-wise deviations in ranking quality. Bars show the mean difference (percentage points) between each guidance topic area’s nDCG@10 and the corresponding model’s overall mean nDCG@10 (dashed line at 0) 8 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: MCQA accuracy by number of context chunks The MCQA scores of generation models by the number of context chunks used in the prompt. Scores are shown for the 3 different retrieval runs used. *Model used to generate benchmark 4.2.1 LLM-as-Judge Validation [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effects of target chunk retrieval and rank on MCQA accuracy (a) The differences in MCQA accuracy when target chunk retrieved vs not retrieved and retrieved at rank r = 1 vs r ≥ 2, for k ∈ {3, 5, 10}. (b) The difference in MCQA accuracy when target chunk is retrieved at rank r, or not retrieved, compared to r = k, for k ∈ {3, 5, 10}. conclusions, while recognising that developing reliable evaluation methods… view at source ↗
Figure 5
Figure 5. Figure 5: Judge Rubric scores per model Proportion of free-form answers that meet each rubric criteria for all LLMs according to the LLM Judge (GPT-OSS-120B) underlying text, limiting causal interpretation. Third, while we observe topic-level variability in ranking quality, our experiments do not isolate the underlying drivers (e.g., topic-specific language, formatting conventions, or document structure). Fourth, th… view at source ↗
Figure 6
Figure 6. Figure 6: Effect of target chunk rank on criteria scores Proportion of free-form answers that meet each judge criteria when the target chunk is retrieved in rank ranges r = 1, 2 ≤ r ≤ 4, 5 ≤ r ≤ 7, 8 ≤ r ≤ 15. 6 Discussion 6.1 Retrieval The strong performance of NV-Embed-v2 across all metrics aligns with external evaluations where NV-Embed models top the MTEB leaderboard, reflecting deliberate optimisation for embed… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 64 canonical work pages · 25 internal anchors

  1. [1]

    Healthy llms? benchmarking llm knowledge of uk government public health information.arXiv preprint arXiv:2505.06046, 2025

    Joshua Harris, Fan Grayson, Felix Feldman, Timothy Laurence, Toby Nonnenmacher, Oliver Higgins, Leo Loman, Selina Patel, Thomas Finnie, Samuel Collins, and Michael Borowitz. Healthy llms? benchmarking llm knowledge of uk government public health information.arXiv preprint arXiv:2505.06046, 2025

    work page arXiv 2025

  2. [2]

    How we are pioneering artificial intelligence applications in public health , Mar 2025

    Blog Editor. How we are pioneering artificial intelligence applications in public health , Mar 2025

    work page 2025

  3. [3]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senevi- ratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Aguera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj...

    work page 2022

  4. [4]

    Sutton, David Pincock, Daniel C

    Reed T. Sutton, David Pincock, Daniel C. Baumgart, Daniel C. Sadowski, Richard N. Fedorak, and Karen I. Kroeker. An overview of clinical decision support systems: benefits, risks, and strategies for success.NPJ Digital Medicine, 3:17, February 2020

    work page 2020

  5. [5]

    Evidence-based methodologies for public health: How to assess the best available evidence when time is limited and there is lack of sound evidence

    European Centre for Disease Prevention and Control. Evidence-based methodologies for public health: How to assess the best available evidence when time is limited and there is lack of sound evidence. Technical report, European Centre for Disease Prevention and Control (ECDC), Stockholm, September 2011

    work page 2011

  6. [6]

    The living approach to who normative products and country implementation: Member state briefing

    World Health Organization. The living approach to who normative products and country implementation: Member state briefing. Member State Briefing (PDF slides), October 2022. 14

    work page 2022

  7. [7]

    Disinformation and public health

    World Health Organization. Disinformation and public health. Questions and answers, February

    work page

  8. [8]

    Almeida, Natasha Azzopardi-Muscat, Marisa A

    Israel Júnior Borges do Nascimento, Ana Beatriz Pizarro, Jussara M. Almeida, Natasha Azzopardi-Muscat, Marisa A. Gonçalves, Mattias Björklund, and David Novillo-Ortiz. Info- demics and health misinformation: a systematic review of reviews.Bulletin of the World Health Organization, 100(9):544–561, September 2022

    work page 2022

  9. [9]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems (NeurIPS), 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems (NeurIPS), 2020

    work page 2020

  10. [10]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, et al. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  11. [11]

    Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers

    Chaitanya Sharma. Retrieval-augmented generation: A comprehensive survey of architectures, enhancements, and robustness frontiers.arXiv preprint arXiv:2506.00054, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey

    Bo Ni, Zheyuan Liu, Leyao Wang, Yongjia Lei, Yuying Zhao, Xueqi Cheng, Qingkai Zeng, Luna Dong, Yinglong Xia, Krishnaram Kenthapadi, Ryan Rossi, Franck Dernoncourt, Md Mehrab Tanjim, Nesreen Ahmed, Xiaorui Liu, Wenqi Fan, Erik Blasch, Yu Wang, Meng Jiang, and Tyler Derr. Towards trustworthy retrieval augmented generation for large language models: A surve...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering

    Yucheng Shi, Shaochen Xu, Tianze Yang, Zhengliang Liu, Tianming Liu, Quanzheng Li, Xiang Li, and Ninghao Liu. Mkrag: Medical knowledge retrieval augmented generation for medical question answering.arXiv preprint arXiv:2309.16035, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Evaluating retrieval-augmented gen- eration–enhanced large language models for question answering on german neurovascular guidelines.Clinical Neuroradiology, 2025

    Marius Vach, Michael Gliem, Daniel Weiss, Vivien Lorena Ivan, Frederik Hauke, Christian Boschenriedter, Christian Rubbert, and Julian Caspers. Evaluating retrieval-augmented gen- eration–enhanced large language models for question answering on german neurovascular guidelines.Clinical Neuroradiology, 2025. (Online ahead of print)

    work page 2025

  15. [15]

    Pichel, and David E

    Marcos Fernández-Pichel, Juan C. Pichel, and David E. Losada. Evaluating search engines and large language models for answering health questions.npj Digital Medicine, 8(1):153, 2025

    work page 2025

  16. [16]

    Lawrence K. Q. Yan, Qian Niu, Ming Li, Yichao Zhang, Caitlyn H. Yin, Cheng Fei, Benji Peng, Ziqian Bi, Pohsun Feng, et al. Large language model benchmarks in medical tasks.arXiv preprint arXiv:2410.21348, 2024

    work page arXiv 2024

  17. [17]

    HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models

    Andrew Maranhão Ventura D’Addario. Healthqa-br: A system-wide benchmark reveals critical knowledge gaps in large language models.arXiv preprint arXiv:2506.21578, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero- Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health, 2025

    work page 2025

  19. [19]

    Evaluating large language models for public health classification and extraction tasks, 2025

    Joshua Harris, Timothy Laurence, Leo Loman, Fan Grayson, Toby Nonnenmacher, Harry Long, Loes WalsGriffith, Amy Douglas, Holly Fountain, Stelios Georgiou, Jo Hardstaff, Kathryn Hopkins, Y-Ling Chi, Galena Kuyumdzhieva, Lesley Larkin, Samuel Collins, Hamish Mo- hammed, Thomas Finnie, Luke Hounsome, Michael Borowitz, and Steven Riley. Evaluating large langua...

    work page 2025

  20. [20]

    Improving language models by retrieving from trillions of tokens

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Ori...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  21. [21]

    Scaling Retrieval-Based Language Models with a Trillion-Token Datastore

    Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettle- moyer, and Pang Wei Koh. Scaling retrieval-based language models with a trillion-token datastore.arXiv preprint arXiv:2407.12854, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    A Survey on Knowledge-Oriented Retrieval-Augmented Generation

    Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, and Daoyu Wang. A survey on knowledge-oriented retrieval-augmented generation.arXiv preprint arXiv:2503.10677, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    Scaling laws for dense retrieval, 2024

    Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. Scaling laws for dense retrieval, 2024

    work page 2024

  24. [24]

    An analysis of fusion functions for hybrid retrieval

    Sebastian Bruch, Siyu Gai, and Amir Ingber. An analysis of fusion functions for hybrid retrieval. ACM Transactions on Information Systems, 42(1):1–35, 2023

    work page 2023

  25. [25]

    Gte-moderncolbert, 2025

    Antoine Chaffin. Gte-moderncolbert, 2025

    work page 2025

  26. [26]

    Medical graph rag: Towards safe medical large language model via graph retrieval- augmented generation, 2024

    Junde Wu, Jiayuan Zhu, Yunli Qi, Jingkun Chen, Min Xu, Filippo Menolascina, and Vicente Grau. Medical graph rag: Towards safe medical large language model via graph retrieval- augmented generation, 2024

    work page 2024

  27. [27]

    Walk&Retrieve: Simple Yet Effective Zero-shot Retrieval-Augmented Generation via Knowledge Graph Walks

    Martin Böckling, Heiko Paulheim, and Andreea Iana. Walk&retrieve: Simple yet effec- tive zero-shot retrieval-augmented generation via knowledge graph walks.arXiv preprint arXiv:2505.16849, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  28. [28]

    Estimating Optimal Context Length for Hybrid Retrieval-augmented Multi-document Summarization

    Adithya Pratapa and Teruko Mitamura. Estimating optimal context length for hybrid retrieval- augmented multi-document summarization.arXiv preprint arXiv:2504.12972, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O. Arik. Long-context llms meet rag: Overcoming challenges for long inputs in rag. InThe Thirteenth International Conference on Learning Representations (ICLR), 2025

    work page 2025

  30. [30]

    Enhancing retrieval- augmented generation: A study of best practices

    Siran Li, Linus Stenzel, Carsten Eickhoff, and Seyed Ali Bahrainian. Enhancing retrieval- augmented generation: A study of best practices. InProceedings of the 31st International Conference on Computational Linguistics (COLING ’25), 2025

    work page 2025

  31. [31]

    Towards Adaptive Memory-Based Optimization for Enhanced Retrieval-Augmented Generation

    Qitao Qin, Yucong Luo, Yihang Lu, Zhibo Chu, and Xianwei Meng. Towards adaptive memory-based optimization for enhanced retrieval-augmented generation.arXiv preprint arXiv:2504.05312, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann, editors,Proceedings of the Conference on Health, Inference, and Learning, volume 174 ofProceedings of Machine Learni...

    work page 2022

  33. [33]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Weihuong Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    work page 2021

  34. [34]

    It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education

    Shrutika Singh, Anton Alyakin, Daniel A. Alber, Jaden Stryker, Miguel Hernandez-Rovira, Ki Yun Park, Eric K. Oermann, et al. It is too many options: Pitfalls of multiple-choice questions in generative ai and medical education.arXiv preprint arXiv:2503.13508, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    A Benchmark for Long-Form Medical Question Answering

    Pedram Hosseini, Jessica M. Sin, Bing Ren, Bryceton G. Thomas, Elnaz Nouri, Ali Farahanchi, and Saeed Hassanpour. A benchmark for long-form medical question answering.arXiv preprint arXiv:2411.09834, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    K-qa: A real-world medical q&a benchmark

    Itay Manes, Naama Ronn, David Cohen, Ran Ilan Ber, Zehavi Horowitz-Kugler, and Gabriel Stanovsky. K-qa: A real-world medical q&a benchmark. InProceedings of the 23rd Workshop on Biomedical NLP (BioNLP 2023), pages 277–294, 2023. 16

    work page 2023

  37. [37]

    Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, Bonaventure F

    Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, Chidi Asuzu Md, Naome A Etori, Aïmérou Ndiaye, Ifeoma Okoh, Evans Doe Ocansey...

    work page 1948

  38. [38]

    An empirical study of evaluating long-form question answering

    Ning Xian, Yixing Fan, Ruqing Zhang, Maarten de Rijke, and Jiafeng Guo. An empirical study of evaluating long-form question answering. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2025

    work page 2025

  39. [39]

    A critical evaluation of evaluations for long-form question answering

    Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. A critical evaluation of evaluations for long-form question answering. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pages 3225–3245, 2023

    work page 2023

  40. [40]

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models.arXiv preprint arXiv:2310.08491, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  41. [41]

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models.arXiv preprint arXiv:2405.01535, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  43. [43]

    A Survey on LLM-as-a-Judge

    Jiahui Gu et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [44]

    Automating evaluation of ai text generation in healthcare using medical llm-as-a-judge.medRxiv, 2025

    Euan Croxford et al. Automating evaluation of ai text generation in healthcare using medical llm-as-a-judge.medRxiv, 2025. Version v2; PubMed PMID: 40313300

    work page 2025

  45. [45]

    NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models.arXiv preprint arXiv:2405.17428, 2024. Includes NV-Embed-v1 and NV-Embed-v2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    Introducing embeddinggemma: The best-in-class open model for on-device embeddings

    Min Choi, Sahil Dua, Alice Lisak, and Google DeepMind. Introducing embeddinggemma: The best-in-class open model for on-device embeddings. Google Developer Blog, September 4 2025

    work page 2025

  47. [47]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024

    work page 2024

  48. [48]

    Sfr-embedding-mistral: Enhance text retrieval with transfer learning

    Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding-mistral: Enhance text retrieval with transfer learning. Salesforce AI Research Blog / Hugging Face model card, October 2024

    work page 2024

  49. [49]

    Multilingual E5 Text Embeddings: A Technical Report

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual e5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672, 2024. Includes themultilingual-E5-largemodel

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  50. [50]

    text-embedding-3-large: Openai’s next-generation large embedding model

    OpenAI. text-embedding-3-large: Openai’s next-generation large embedding model. OpenAI API / Documentation, January 2024. Creates embeddings with up to 3072 dimensions. 17

    work page 2024

  51. [51]

    Cormack, Charles L

    Gordon V . Cormack, Charles L. A. Clarke, and Stefan Büttcher. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. InProceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’09), pages 758–759. ACM, 2009

    work page 2009

  52. [52]

    Dangi, Beni B

    Deven B. Dangi, Beni B. Dangi, and Oliver Steinbock. Evaluation of gpt-4o and gpt-4o-mini’s vision capabilities for compositional analysis from dried solution drops, 2025

    work page 2025

  53. [53]

    The Llama 3 Herd of Models

    A. Grattafiori et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. Describes the Llama 3 family, including the 70B instruction-tuned model

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  54. [54]

    Phi-4 Technical Report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu,...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  55. [55]

    Gemma 3 Technical Report

    Gemma Team, Google DeepMind. Gemma 3: A multimodal, multilingual, long-context open model family.arXiv preprint arXiv:2503.19786, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  56. [56]

    MedGemma Technical Report

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma: A multimodal generative model family for medical text and image comprehension.arXiv preprint arXiv:2507.05201, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  57. [57]

    Cohere labs command-r (c4ai-command-r-v01)

    Cohere Labs / CohereForAI. Cohere labs command-r (c4ai-command-r-v01). Hugging Face model card, March 2024. 35B parameter generative model; 128K token context

    work page 2024

  58. [58]

    BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  59. [59]

    On (normalised) discounted cumulative gain as an off-policy evaluation metric for top-n recommendation

    Olivier Jeunen, Ivan Potapov, and Aleksei Ustimenko. On (normalised) discounted cumulative gain as an off-policy evaluation metric for top-n recommendation. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24). Association for Computing Machinery, 2024

    work page 2024

  60. [60]

    BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives

    Xiaoyue Wang, Jianyou Wang, Weili Cao, Kaicheng Wang, Ramamohan Paturi, and Leon Bergen. Birco: A benchmark of information retrieval tasks with complex objectives.arXiv preprint arXiv:2402.14151, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  61. [61]

    Implementing large language models in health care: Clinician-focused review with interactive guideline.Journal of Medical Internet Research, 27(1):e71916, 2025

    HongYi Li, Jun-Fen Fu, and Andre Python. Implementing large language models in health care: Clinician-focused review with interactive guideline.Journal of Medical Internet Research, 27(1):e71916, 2025

    work page 2025

  62. [62]

    Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of ai models

    European Data Protection Board. Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of ai models. EDPB Opinion 28/2024, European Data Protection Board, Brussels, Belgium, December 2024. Adopted on 17 December 2024

    work page 2024

  63. [63]

    A guide to good practice for digital and data-driven health technologies

    Department of Health and Social Care and Government Digital Service. A guide to good practice for digital and data-driven health technologies. https://www.gov.uk/data-ethic s-guidance/a-guide-to-good-practice-for-digital-and-data-driven-healt h-technologies, 2025. Published 27 January 2025; accessed 9 December 2025

    work page 2025

  64. [64]

    Frangi, Antonio R

    Karim Lekadir, Alejandro F. Frangi, Antonio R. Porras, Ben Glocker, Celia Cintas, Curtis P. Langlotz, Eva Weicken, Folkert W. Asselbergs, Fred Prior, Gary S. Collins, Georgios Kaissis, Gianna Tsakou, Irène Buvat, Jayashree Kalpathy-Cramer, John Mongan, Julia A. Schnabel, Kaisar Kushibar, Katrine Riklund, Kostas Marias, Lameck M. Amugongo, Lauren A. Fromon...

    work page 2025