Pith. sign in

REVIEW 4 major objections 6 minor 41 references

EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EvidenceMap claims that fine-tuning a 66M-parameter encoder to emit support, correlation, and summary vectors as soft prompts lets a frozen 3B generator beat 8B retrieval-augmented generation on biomedical questions.

desk verdict A promising evidence-analysis wrapper for small LMs, undermined by a likely strawman 8B RAG baseline that needn't sink the method, only the headline. read the letter →

arxiv 2501.12746 v4 pith:6ENWXVYT submitted 2025-01-22 cs.CL cs.AI

Chang Zong , Jian Wan , Siliang Tang , Lei Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords biomedicalquestionansweringevidenceanalysissmalllanguagemodelssoftpromptingretrieval-augmentedgenerationmapparameter-efficientfine-tuninghallucinationmitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EvidenceMap asks whether a 66M-parameter language model can learn how to weigh, relate, and condense evidence, then hand that analysis to a frozen 3B generator as a soft prompt. The paper reports that this learning step is enough to beat retrieval-augmented generation with an 8B model by 19.9% on reference-based quality and 5.7% on accuracy across BioASQ and PubMedQA. The selling point is resource efficiency: the only trained parameters are the tiny encoder's, so the approach fits on a single GPU while outperforming far larger systems. A sympathetic reading is that explicit, structured evidence analysis can substitute for raw parameter scale in knowledge-grounded question answering.

What carries the argument

The central object is the evidence map, a structure that pairs each question with its evidence plus three analysis vectors: support evaluation $R_{\mathrm{eval}}$ for each evidence-question pair, logical correlation $R_{\mathrm{cor}}$ for each evidence-evidence pair, and a summarization $E_{\mathrm{sum}}$ of all evidence. The mechanism that carries the argument is prompt-conditioned hidden-state extraction with a tiny pre-trained language model: the model is given separate prompts for evaluation, correlation, and summarization, and its last hidden states are treated as the analysis vectors, concatenated, projected by an MLP, and prepended to the frozen generator's input as a soft prompt. This separates analysis from generation and gives the generator an explicit relational summary of the evidence that plain retrieval-augmented prompting omits.

What would settle it

Probe the three hidden-state vectors with labeled tasks: classify whether $R_{\mathrm{eval}}$ predicts evidence-support judgments, whether $R_{\mathrm{cor}}$ predicts pairwise agreement or conflict, and whether $E_{\mathrm{sum}}$ reconstructs extractive summaries; if these probes fail while the end-to-end scores stay high, the components are not doing the claimed analysis. Alternatively, swap one component vector for another at generation time; if performance barely changes, the components are interchangeable.

Watch

Extended reading notes

Core claim

The paper claims that explicitly modeling three facets of evidence analysis—supportive evaluation (does each evidence item support the question), logical correlation (how pairs of evidence relate), and content summarization (the aggregate of all evidence)—as representations produced by a small pre-trained encoder, and projecting those representations into the generator's embedding space, yields better answers than implicit methods like RAG, CoT, RAT, prefix tuning, and LoRA. The three facets are organized in an evidence map $\mathcal{M} = \{Q, E_1,\ldots,E_m, R_{\mathrm{eval}}, R_{\mathrm{cor}}, E_{\mathrm{sum}}\}$; each is computed by prompting DistilBERT with task-specific prompts and taking hidden states, then an MLP projects them into the generator. The generator (Llama-3.2-3B) is frozen; only the 66M encoder is updated by the token-generation loss of the reference answer. Results on BioASQ and PubMedQA show that EvidenceMap with a 3B generator exceeds Llama3.1-RAG 8B, GPT-4o-mini-RAG, and OpenBioLLM-8B, with ablations attributing the gain mostly to the correlation component and to the evidence-analysis input during generation.

Load-bearing premise

The method assumes that maximizing the chance of producing the reference answer actually teaches the small model to evaluate support, relate evidence, and summarize, rather than just learning one generic prompt vector.

Editorial extensions

If this is right

  • A 66M encoder trained only on answer-generation loss can substitute for billions of parameters in evidence-grounded QA, cutting training and deployment cost.
  • Explicitly isolating analysis components outperforms implicit tuning methods on the same generator, including LoRA, prefix tuning, and RAG-LoRA.
  • Removing the evidence-analysis input drops BERT-S by about 15.6% on BioASQ, nearly as much as removing the textual evidence itself, so the analysis vectors are load-bearing for answer quality.
  • Performance rises as more paper evidence and LLM evidence are supplied, suggesting the framework scales with richer evidence rather than being overwhelmed by it.
  • The method transfers across small generators (Llama-3.2-3B, Qwen-2.5-3B, Phi-3.5-mini) and across small encoders, with larger encoders giving modest gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the three vectors encode distinct semantics, the same 66M encoder could be lifted out and reused with other frozen generators or for evidence reranking without retraining the decoder; the paper does not test this.
  • Editorial inference: the reported ablation order, with correlation mattering most and summarization least, suggests that when evidence is noisy and conflicting, relationship modeling deserves more capacity; a testable extension is to weight or oversample pair-correlation features.
  • Editorial inference: because the training objective is only answer likelihood, a natural next experiment, not run here, is to add auxiliary losses or labels for support, correlation, and summarization; if those improve scores, it would confirm the components are meaningful rather than emergent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces EvidenceMap, a framework for biomedical question answering in which a tiny pre-trained model (DistilBERT, 66M parameters) is fine-tuned to produce latent, analysis-conditioned embeddings—intended to encode supportive evaluation of evidence, pairwise logical correlation, and content summarization—that are projected and prepended as soft prompts to a frozen small generative model (Llama-3.2-3B). The framework is evaluated on BioASQ and PubMedQA with ROUGE-L, BERTScore, and a GPT-4o-based accuracy score, and the authors report that EvidenceMap with a 3B generator exceeds an 8B Llama-3.1 RAG baseline by 19.9% in BERTScore and 5.7% in LLM accuracy, while also outperforming CoT, RAT, prefix-tuning, LoRA, and RAG-LoRA baselines. Ablation studies show that each of the three analysis components, the textual evidence input, and the evidence-analysis prompt all contribute to the reported performance, and additional experiments vary the pre-trained model, the generative model, and the amount of evidence.

Significance. If the results hold, EvidenceMap would be a practically interesting result: it achieves competitive or better performance than several 3B-8B RAG and tuning baselines while updating only a 66M model, which is attractive for low-resource deployment. The paper has genuine strengths: the method is clearly described, the ablations are systematic, the case studies illustrate qualitative differences, and the comparison spans several baseline families. However, the headline quantitative claim rests on a single, implausibly weak 8B baseline, and no significance testing or error bars are reported, which is especially concerning given the small PubMedQA test set. The central interpretive claim—that the three latent vectors genuinely encode supportive evaluation, logical correlation, and summarization—is also not directly evidenced, since the only training signal is the final answer-generation loss. The contribution is more accurately characterized as an evidence-conditioned soft-prompt method with a plausible but unverified analysis-inspired decomposition.

major comments (4)
  1. [Table 1 and Appendix B] The 8B baseline used for the abstract's headline claim is implausibly weak and not sufficiently specified. In Table 1, Llama3.1-RAG 8B scores ROUGE-L 0.1936 and BERT-S 0.6077 on BioASQ, while Llama3.2-RAG 3B scores 0.2956 and 0.6225 on the same metrics. A larger model with the same evidence and prompt should not be so much worse, which suggests differences in prompt template, evidence truncation, decoding settings, or retrieval setup. Appendix B states only that the same evidence is provided and does not report the prompt or hyperparameters for the RAG baselines. Since the 19.9% and 5.7% margins in the abstract are computed against this specific baseline, the authors must either (a) provide the exact baseline setup and rerun the comparison with a stronger, more representative 8B baseline (e.g., Llama-3.1-8B-Instruct with the same controlled prompt and evidence), or (b) remove or substantially qualify the headline claim.
  2. [Tables 1-3 and Appendix C] No statistical support is provided for the central comparisons. The PubMedQA test set has only 47 samples (Appendix C, Table 5), yet the paper reports single-run scores without error bars, confidence intervals, or significance tests. Differences such as EvidenceMap 3B versus LoRA-Tuning on PubMedQA (ROUGE-L 0.2931 vs 0.2517, BERT-S 0.6783 vs 0.6319) could easily be within noise on 47 samples. Similarly, the ablation differences in Table 2 are small (e.g., a 0.61% LLM-ACC drop on BioASQ when summarization is removed) and may not be reliable. The authors should report multiple seeds, paired significance tests (e.g., bootstrap or approximate randomization) for the main results and key ablations, and, for the GPT-4o-based LLM-ACC metric, variance over multiple judge calls.
  3. [Section 3.4, Eq. (14)] The claim that the 66M model explicitly learns supportive evaluation, logical correlation, and content summarization is not supported by the training objective. Equation (14) is a standard autoregressive answer-token likelihood over the final generated answer; there are no labels, probes, or manual checks for the three analysis components. The embeddings R_eval, R_cor, and E_sum are produced by prompt-conditioned hidden states and trained only to improve the final answer. Consequently, the three components could collapse into a generic evidence-conditioned prompt vector, in which case the method reduces to soft-prompt tuning and the evidence-analysis story is unsupported, even if the performance numbers still hold. The ablations in Table 2 show that removing each component hurts performance, but they do not establish that the remaining vectors encode the intended semantics. The authors should either provide probing results (e.g., linear classifiers on the three embedding types), a control experiment with shuffled or randomized analysis vectors that preserves the same parameter count, or temper the claims to describe the vectors as analysis-inspired latent prompts.
  4. [Abstract and Table 1] The abstract's quantitative claim is dataset- and metric-specific but is presented as a general result. The 19.9% improvement is the relative BERTScore gain on BioASQ when EvidenceMap 3B is compared with Llama3.1-RAG 8B, and the 5.7% is the corresponding LLM-ACC gain on the same dataset. On PubMedQA, the analogous relative gains are much smaller (BERT-S: 0.6783 vs 0.6431, about 5.5%; LLM-ACC: 0.6610 vs 0.6440, about 2.6%), and they are not accompanied by uncertainty estimates. The abstract and conclusions should state which dataset and metric support the headline numbers, and the authors should avoid implying that the 8B comparison is uniformly favorable across settings.
minor comments (6)
  1. [Section 4.6] The phrase 'DistilBERT retrains most of the ability of BERT' should be 'retains' rather than 'retrains.'
  2. [Section 3.3, Eq. (12)] There is a typo: 'formualted' should be 'formulated.'
  3. [Section 4.6, Table 4] The arrows in Table 4 are used inconsistently: for example, BERT-Base improves BERT-S on BioASQ but the LLM-ACC column shows a downward arrow, and the table does not state the direction of improvement for each metric. Please clarify the notation.
  4. [Section 4.7, Figure 4] Figure 4 lacks axis labels, a legend, and clearly marked dataset panels; the text refers to 'Left for BioASQ and right for PubMedQA' but the figure itself is not readable. Please add labels and a caption that defines PN and the LLM-evidence condition.
  5. [Section 4.1 and Appendix B] The paper says 'the 1B and 3B versions of Llama-3.2' are used, but it does not specify whether these are the base or instruct variants. Baseline reproducibility would benefit from exact model identifiers, including for OpenBioLLM and GPT-4o-mini.
  6. [Appendix B] The RAG-LoRA baseline is described as an 'instruction-tuning strategy', but the method name suggests retrieval-augmented LoRA; please clarify the prompt and training setup so that the boundary between RAG-LoRA and LoRA-Tuning is clear.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EvidenceMap's headline result is an empirical comparison on held-out test sets, and the unvalidated evidence-analysis interpretation is a construct-validity limitation, not a self-referential derivation.

full rationale

The paper's central claim is an empirical improvement measured on held-out test splits (BioASQ 340 test questions and PubMedQA 47 test questions, Appendix C) after training on separate train data. The EvidenceMap vectors in Eqs. 7-10 and 12 are trained end-to-end through the answer-likelihood objective in Eq. 14; the test-set numbers are not recomputed from fit parameters, so the reported margins are not forced by construction. The three named analysis components come from prompt templates in Appendix A.1 and are never probed against external labels, making the interpretive claim that the 66M PLM has genuinely learned supportive evaluation, logical correlation, and summarization unsupported; however, this is a construct-validity and interpretability limitation, explicitly acknowledged in the Limitations section ('we will continue to conduct in-depth research on the interpretability and stability of model inference'), not circular reasoning. The implausible Table 1 inversion (Llama3.1-8B RAG at ROUGE-L 0.1936 versus Llama3.2-3B RAG at 0.2956) is a possible baseline-fairness or setup artifact and belongs in correctness risk, not in circularity analysis. There is no load-bearing self-citation: the paper's citations to PromptBERT, PromptEOL, RAG, CoT, RAT, and related work are external and do not define the target result. No fitted parameter is renamed as a prediction, no equation reduces to its own input, and no uniqueness theorem is imported from the authors' prior work.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central claim rests on several assumptions about evaluation validity, evidence quality, and the sufficiency of end-to-end supervision. There are no invented physical entities; the EvidenceMap is a data structure, not a postulated entity. Trainable parameters are the expected free parameters of an ML method, but the absence of direct supervision for the three analysis components means the semantic content of those vectors is not independently verified.

free parameters (4)
  • DistilBERT fine-tuned parameters (theta) = 66M parameters, values not reported
    All PLM weights are updated by Eq. 14 answer-generation loss; these carry the claimed evidence-analysis ability but are never supervised on labeled evidence analysis.
  • Evidence summarization MLP weights (W_s, b_s) = not reported
    Aggregates PLM evidence features into E_sum; trained end-to-end with no direct summarization objective.
  • Projector MLP weights (W_hat, b_hat) = not reported
    Projects [E_sum, Reval, Rcor] into the GLM embedding space; trained end-to-end.
  • Evidence cap and training hyperparameters = max papers=5, lr=5e-4, batch=4, epochs=10
    Chosen to standardize experimental settings and reduce overhead; the evidence cap directly controls the number of pairwise correlation embeddings.
assumptions (5)
  • ad hoc to paper Eq. 14 answer-token likelihood is sufficient supervision to induce the three evidence-analysis skills.
    The three analysis components are never directly labeled; the paper assumes end-to-end generative loss instills them.
  • domain assumption BioASQ and PubMedQA reference answers and pre-annotated snippets are correct and relevant evidence.
    The paper uses the datasets as gold-standard question-answer pairs and treats their human-annotated snippets as evidence.
  • domain assumption ROUGE-L, BERTScore, and GPT-4o LLM-ACC validly measure answer quality and accuracy.
    The main conclusions depend on these metrics; LLM-ACC is an automated judge rather than a human clinical evaluation.
  • domain assumption A frozen GLM can be effectively steered by projected soft prompts from a different encoder.
    The whole framework relies on the MLP projection in Eq. 12 aligning PLM hidden states with the GLM embedding space.
  • domain assumption GPT-4o-generated evidence is a valid evidence source and does not leak the reference answer.
    GPT-4o is prompted only with the question, but the same model is also used to score the generated answers with the reference answer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering." pith.science (2026). https://pith.science/paper/6ENWXVYT

@misc{pith2026250112746,
  author       = {Pith},
  title        = {Pith review of: EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ENWXVYT}},
  note         = {Machine review of arXiv:2501.12746}
}
read the original abstract

When addressing professional questions in the biomedical domain, humans typically acquire multiple pieces of information as evidence and engage in multifaceted analysis to provide high-quality answers. Current LLM-based question answering methods lack a detailed definition and learning process for evidence analysis, leading to the risk of error propagation and hallucinations while using evidence. Although increasing the parameter size of LLMs can alleviate these issues, it also presents challenges in training and deployment with limited resources. In this study, we propose EvidenceMap, which aims to enable a tiny pre-trained language model to explicitly learn multiple aspects of biomedical evidence, including supportive evaluation, logical correlation and content summarization, thereby latently guiding a small generative model (around 3B parameters) to provide textual responses. Experimental results demonstrate that our method, learning evidence analysis by fine-tuning a model with only 66M parameters, exceeds the RAG method with an 8B LLM by 19.9% and 5.7% in reference-based quality and accuracy, respectively.

Figures

Figures reproduced from arXiv: 2501.12746 by the authors.

Figure 1
Figure 1. A diagram describing the analysis with various [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Existing methods (left) suffer from erroneous [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The overview of our EvidenceMap framework. An evidence-augmented question is modeled as an evidence map, in which the representations of evidence evaluation, correlation and summarization are computed by a pre-trained language model (PLM) accordingly, in order to facilitate answering by a generative model (GLM). 3 Framework Our EvidenceMap framework is illustrated in Fig￾ure 3, in which a pre-trained language model … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The variation of BERT-S and LLM-ACC with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Two cases with distinct difficulties demonstrate that [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Results on BioASQ with various small genera [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511

    arXiv 2023

  5. [5]

    Shuwei Chen, Jun Liu, and Yang Xu. 2021. A logical reasoning based decision making method for handling qualitative knowledge. International Journal of Approximate Reasoning, 129:49--63

    work page 2021

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. https://aclanthology.org/2024.eacl-demo.16/ RAGA s: Automated evaluation of retrieval augmented generation . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150--158, St. Julians, Malta. Ass...

    work page 2024

Show all 41 references
  1. [9]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  2. [10]

    Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630

    2024 arXiv

  3. [11]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  4. [12]

    Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.785 Calibrating long-form generations from large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024,...

    2024 doi

  5. [13]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  6. [14]

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.181 Scaling sentence embeddings with large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3182--31...

    2024 doi

  7. [15]

    Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.603 P rompt BERT : Improving BERT sentence embeddings with prompts . In Proceedings of the 2022 C...

    2022 doi

  8. [16]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  9. [17]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. https://doi.org/10.18653/v1/D19-1259 P ub M ed QA : A dataset for biomedical research question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing an...

    2019 doi

  10. [18]

    Tian Kang, Yingcheng Sun, Jae Hyun Kim, Casey Ta, Adler Perotte, Kayla Schiffer, Mutong Wu, Yang Zhao, Nour Moustafa-Fahmy, Yifan Peng, et al. 2023. Evidencemap: a three-level knowledge representation for medical evidence computation and comprehension. Journal of the American ...

    2023

  11. [19]

    Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170

    2023

  12. [20]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.243 The power of scale for parameter-efficient prompt tuning . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045--3059, Online a...

    2021 doi

  13. [21]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  14. [22]

    Xiang Lisa Li and Percy Liang. 2021. https://doi.org/10.18653/v1/2021.acl-long.353 Prefix-tuning: Optimizing continuous prompts for generation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    2021 doi

  15. [23]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

    2023 arXiv

  16. [24]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  17. [25]

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352

    2023 arXiv

  18. [26]

    Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22266--22275

    2024

  19. [27]

    Malaikannan Sankarasubbu Ankit Pal and Malaikannan Sankarasubbu. 2024. Openbiollms: Advancing open-source large language models for healthcare and life sciences

    2024

  20. [28]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  21. [29]

    V Sanh. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108

    2019 arXiv

  22. [30]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172--180

    2023

  23. [31]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1--8

    2025

  24. [32]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.754 Self-instruct: Aligning language models with self-generated instructions . In Proceedings of the 61st Annual Mee...

    2023 doi

  25. [33]

    Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. 2024. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. arXiv preprint arXiv:2403.05313

    2024 arXiv

  26. [34]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavi \'e , Orion Weller, Oskar Hallstr \"o m, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and lo...

    2024 arXiv

  27. [35]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  28. [36]

    Qianqian Xie, Jennifer Amy Bishop, Prayag Tiwari, and Sophia Ananiadou. 2022. Pre-trained language models with domain knowledge for biomedical extractive summarization. Knowledge-Based Systems, 252:109460

    2022

  29. [37]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  30. [38]

    Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. 2023. https://doi.org/10.18653/v1/2023.acl-long.25 Tailor: A soft-prompt-based approach to attribute-based controlled text generation . In Proceedings of the 61st Annual Meeting of th...

    2023 doi

  31. [39]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  32. [40]

    Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina Idnay, Yiming Luo, Elizabeth Park, Jordan G Nestor, Matthew E Spotnitz, Ali Soroush, et al. 2024. Closing the gap between open source and commercial large language models for medical evidence summarization. npj Digital Me...

    2024

  33. [41]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.