Pith. sign in

REVIEW 3 major objections 4 minor 18 references

Zero-shot LLM ensembles, paired with focused retrieval, can outperform individual models and rival domain-tuned systems on biomedical yes/no question answering; longer contexts can hurt accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An ensemble of zero-shot LLMs with BM25 retrieval and semantic reranking ranked first in one BioASQ 13 yes/no batch, with longer contexts observed to hurt answer quality.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A useful BioASQ system report with one strong external result and two claims (the ensemble advantage and the context-length effect) that the tables do not actually support. the 3 major comments →

arxiv 2509.08596 v1 pith:X56ET22P submitted 2025-09-10 cs.CL

LLM Ensemble for RAG: Role of Context Length in Zero-Shot Question Answering for BioASQ Challenge

Dima Galat , Diego Molla-Aliod This is my paper
classification cs.CL
keywords large language modelszero-shot question answeringretrieval-augmented generationbiomedical QAensemble methodscontext lengthBioASQinformation retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a zero-shot question-answering system, built by retrieving a small set of relevant biomedical documents and combining answers from several large language models, can match or beat systems that are fine-tuned on domain data. The claim is evaluated on the BioASQ biomedical QA challenge, where the approach reached rank 1 on yes/no questions in one end-to-end batch and in three of the four Phase B batches. A second claim is that context length matters in a non-monotonic way: feeding full article abstracts can hurt accuracy compared with focused snippets, which the authors attribute to information dilution and model disorientation. If correct, the practical consequence is that careful retrieval plus ensemble synthesis can substitute for expensive domain fine-tuning, and that RAG design should treat context selection as a key accuracy lever.

Core claim

The central claim is that an ensemble of zero-shot LLMs, when fed by a multi-stage IR pipeline, can achieve state-of-the-art performance on biomedical yes/no QA without any fine-tuning or labeled training data. The system generates search queries with an LLM, runs them through lexical BM25 retrieval over PubMed titles and abstracts, reranks results semantically, and keeps the top 300 documents as context. Candidate answers are produced by several LLM variants and then merged by a synthesis model that also assigns a confidence score; low-confidence answers are regenerated more deterministically. On BioASQ 13 the approach took first place on yes/no questions in Phase A+ batch 4 (accuracy 0.92)

What carries the argument

Two mechanisms carry the argument. The first is a multi-stage IR pipeline: an LLM generates a structured boolean query, BM25 retrieves candidate documents, a semantic reranker selects the top 300, and an automated fallback widens sources when evidence is thin. The second is a zero-shot ensemble: several LLMs answer the same question, and a synthesis step resolves contradictions, merges answers, and returns a confidence score; scores below a threshold trigger a lower-temperature rerun. The context-length investigation compares these systems under three input regimes—IR-derived documents, curated snippets, and full abstracts—to isolate how much evidence the QA step needs.

Load-bearing premise

The load-bearing premise is that the ranking differences and the longer-context decline stem from the described pipeline rather than from chance fluctuations in small test batches, prompt wording, or specific commercial model versions.

What would settle it

Run the pipeline on a fixed set of several hundred yes/no and list questions with frozen model versions and prompts, permute batch assignments, and compare the ensemble to its best single model and to short-snippet versus full-abstract contexts. If the accuracy gaps shrink to noise, or if longer context does not consistently lower accuracy, the architecture claims fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Zero-shot ensembles can replace domain fine-tuning for binary biomedical QA, eliminating the need for labeled data.
  • Retrieval precision, not model choice, is the main bottleneck in RAG: a good reranker plus focused context can outperform larger or tuned models.
  • Longer contexts are not always better; supplying full abstracts can lower accuracy, so RAG systems should tune context granularity.
  • LLM-generated queries can overcome the lexical gap of sparse retrieval, keeping BM25 competitive without dense passage retrieval.
  • Confidence-gated answer synthesis can make ensemble outputs stable enough for exact-answer evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's two explanations for longer-context degradation—information dilution versus formatting difficulty for structured outputs—are not cleanly separated, since the failures concentrate in List and Factoid questions; decoupling generation from formatting is a natural next experiment.
  • The success may be specific to binary questions: with two classes, majority-style synthesis is forgiving, whereas structured answers leave little room for aggregation, so ensembles may help least exactly where formatting is strictest.
  • A direct extension would test whether a 'just enough context' budget, chosen per question type, generalizes to other RAG domains or whether the dilution effect is specific to biomedical abstracts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes a zero-shot RAG system for the BioASQ 13 challenge. The IR pipeline combines LLM-generated Elasticsearch/BM25 queries, semantic reranking, and a top-300 context selection. The QA stage uses zero-shot prompting of Gemini 2.0 Flash, Gemini 2.5 Flash Preview, and Claude 3.7 Sonnet, followed by a synthesis step that consolidates candidate answers and assigns a confidence score. The paper reports rank-1 Yes/No results on Phase A+ batch 4 (accuracy 0.92) and on Phase B batches 1–3, and claims that ensembles outperform individual LLMs and that longer contexts degrade performance via 'information dilution and model disorientation.' The contribution is framed as a scalable, fine-tuning-free alternative to domain-tuned systems.

Significance. If fully supported, the results would be meaningful: externally evaluated competition rankings show that a zero-shot ensemble plus RAG can be competitive with specialized biomedical QA systems, and a controlled study of context length would inform RAG design. The paper's strengths include its use of an external benchmark, the absence of fine-tuning or labeled data, and a reasonably detailed description of the multi-stage retrieval pipeline. However, as written, the two headline claims are not established by the reported data: no table compares the ensemble output against its member models, and the context-length effect is confounded with model choice and context-selection strategy. The paper's current value therefore rests on system-level rank reports rather than on validated mechanisms.

major comments (3)
  1. [Abstract; §3.2; Tables 1–4] The central claim that ensembles outperform individual LLMs is unsupported by the reported comparisons. The rows in Tables 1–4 are context/model variants ('Simple truncation' is identified as Gemini 2.0 Flash, 'Extractive' as Claude 3.7 Sonnet); no row corresponds to the ensemble/synthesis condition. §4.2 says rank-1 Yes/No results were obtained 'using an ensemble of Gemini 2.0 Flash and Gemini 2.5 Flash Preview', but Table 3 does not label that condition. Please add a per-batch ablation on the same questions comparing each member model, a majority-vote baseline, and the synthesis output, reporting both accuracy and rank.
  2. [Abstract; §4.1; §4.2] The title's context-length claim is not isolated. The comparisons vary model and context selection simultaneously: 'Simple truncation' uses Gemini 2.0 Flash, 'Extractive' uses Claude 3.7 Sonnet, and 'Kmeans' is another context-selection variant. No condition holds the model fixed and varies only context length/truncation. Furthermore, the abstract's 'information dilution and model disorientation' is at odds with §4.2's 'difficulties in generating well-structured outputs'. Please report controlled experiments (same LLM, same questions, different context lengths/selection) and align the mechanistic explanation with the observed data.
  3. [Tables 1–4] The reported statistics are insufficient to support rank comparisons. BioASQ batch sizes are small; for example, in Table 3 batch 1, 1.00 vs 0.94 can be a difference of one or two questions, and batch 3 ties three systems at 0.95. Please report the number of Yes/No questions per batch, confidence intervals or exact binomial tests, and consistent system names. Without this, the rank-1 claims and cross-batch general statements are hard to interpret or compare.
minor comments (4)
  1. [§3.2] The synthesis confidence threshold of 0.5 is described as 'determined empirically', but no sensitivity analysis or data are provided. Clarify whether this threshold affects the reported results and whether the 're-run' condition was applied in the submitted runs.
  2. [Tables 1–4] The system variants 'Simple truncation', 'Extractive', 'Kmeans', and '(last)' are not defined in the methodology or table captions; they are introduced only in §4.1 prose and are inconsistently capitalized ('KMeans' in Table 4). Add a notation table or caption definitions.
  3. [Introduction] The author email field in the arXiv title page is corrupted ('/envel⌢pe-⌢pendima.galat...'), and §4.1 contains a typo: 'It's important note that'.
  4. [§4.2] The statement that 'Gemini consistently outperformed Claude' is contradicted by Table 3 batch 3, where the Claude-based Extractive run ties at 0.95. Qualify the claim or provide aggregate statistics that justify it.

Circularity Check

0 steps flagged

No significant circularity: results are external benchmark evaluations with no fitted-input predictions or self-citation chain.

full rationale

This paper reports system outputs on the external BioASQ 13 benchmark and compares them against official competition rankings. The central claims (ensemble-based zero-shot QA can achieve SOTA on Yes/No questions; longer contexts can hurt answer quality) are empirical observations from those external evaluations, not predictions derived from fitted parameters. The only explicitly fitted knob is the confidence threshold of 0.5 used to decide whether to re-run the synthesis step with lower temperature; this is a minor implementation detail, not a quantity used to generate the reported benchmark numbers in a way that would force the outcome. No load-bearing self-citation appears: the paper's references are to prior BioASQ overviews, query-expansion work, and general RAG/reranking methods, none of which are by the present authors or invoked to justify the main result. The paper does contain an evidentiary gap: Tables 1-4 label single-model/context variants (e.g., 'Extractive', 'Simple truncation', 'Kmeans') but do not include a row explicitly labeled 'Ensemble' or 'Synthesis', so the claim that ensembles outperform individual LLMs is inferred from system-level ranks rather than demonstrated by a controlled within-paper comparison. That is a missing ablation, not a circularity; the external benchmark results remain independent evidence of the system's performance. Similarly, the context-length finding is a reported observation, and the abstract's 'information dilution and model disorientation' explanation differs somewhat from Section 4.2's 'difficulties in generating well-structured outputs,' but this internal inconsistency concerns interpretation, not circular derivation. Overall, the derivation chain is self-contained against an external benchmark, and no step reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces no new theoretical entities. Its load-bearing inputs are commercial API behaviors, hand-set pipeline hyperparameters (10k retrieved, 300 reranked, threshold 0.5), and the external competition data. These dominate the reported results.

free parameters (3)
  • synthesis confidence threshold = 0.5
    Section 3.2: threshold below which synthesis is re-run with reduced temperature; 'determined empirically'.
  • BM25 retrieval count = up to 10,000
    Section 3.1: documents retrieved before reranking; hand-set design choice.
  • top reranked document count = 300
    Section 3.1: top 300 documents selected for RAG; hand-set, no sensitivity analysis.
axioms (3)
  • domain assumption BioASQ competition results are accurate and comparable across teams
    The paper builds its SOTA claims entirely on externally reported rankings (Tables 1-4), assuming run conditions are fair and stable.
  • domain assumption Commercial LLM API outputs are sufficiently deterministic for the reported comparisons
    Single-pass zero-shot results with temperature above 0 vary across runs; the paper does not report multiple seeds or variance.
  • domain assumption Accuracy on small per-batch question sets is a stable estimator of system quality
    Rank claims for batches where 'last' runs differ drastically (e.g., 0.29 vs 1.00 in batch 1 Phase B) suggest instability; the paper still generalizes to system-level conclusions.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Ensemble for RAG: Role of Context Length in Zero-Shot Question Answering for BioASQ Challenge." pith.science (2026). https://pith.science/paper/X56ET22P

@misc{pith2026250908596,
  author       = {Pith},
  title        = {Pith review of: LLM Ensemble for RAG: Role of Context Length in Zero-Shot Question Answering for BioASQ Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X56ET22P}},
  note         = {Machine review of arXiv:2509.08596}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Biomedical question answering (QA) poses significant challenges due to the need for precise interpretation of specialized knowledge drawn from a vast, complex, and rapidly evolving corpus. In this work, we explore how large language models (LLMs) can be used for information retrieval (IR), and an ensemble of zero-shot models can accomplish state-of-the-art performance on a domain-specific Yes/No QA task. Evaluating our approach on the BioASQ challenge tasks, we show that ensembles can outperform individual LLMs and in some cases rival or surpass domain-tuned systems - all while preserving generalizability and avoiding the need for costly fine-tuning or labeled data. Our method aggregates outputs from multiple LLM variants, including models from Anthropic and Google, to synthesize more accurate and robust answers. Moreover, our investigation highlights a relationship between context length and performance: while expanded contexts are meant to provide valuable evidence, they simultaneously risk information dilution and model disorientation. These findings emphasize IR as a critical foundation in Retrieval-Augmented Generation (RAG) approaches for biomedical QA systems. Precise, focused retrieval remains essential for ensuring LLMs operate within relevant information boundaries when generating answers from retrieved documents. Our results establish that ensemble-based zero-shot approaches, when paired with effective RAG pipelines, constitute a practical and scalable alternative to domain-tuned systems for biomedical question answering.

Figures

Figures reproduced from arXiv: 2509.08596 by Diego Molla-Aliod, Dima Galat.

Figure 1
Figure 1. Figure 1: IR process Healthline, and Wikipedia. This ensures that we have an answer candidate for all questions in Phase A+ test sets. 3.2. Question Answering Pipeline We adopt a unified, zero-shot QA framework for both Phase A+ and Phase B of the challenge. While the core QA procedure remains consistent across phases, Phase A+ incorporates an additional IR step to verify the presence of candidate answers within rel… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

  1. [1]

    Tsatsaronis, G

    G. Tsatsaronis, G. Balikas, P. Malakasiotis, I. Partalas, M. Zschunke, M. R. Alvers, D. Weissenborn, A. Krithara, S. Petridis, D. Polychronopoulos, Y. Almirantis, J. Pavlopoulos, N. Baskiotis, P. Gallinari, T. Artiéres, A. C. N. Ngomo, N. Heino, E. Gaussier, L. Barrio-Alvers, M. Schroeder, I. Androutsopou- los, G. Paliouras, An overview of the BioASQ larg...

    doi:10.1186/s12859-015-0564-6 2015

  2. [2]

    Nentidis, A

    A. Nentidis, A. Krithara, G. Paliouras, M. Krallinger, L. G. Sanchez, S. Lima, E. Farre, N. Loukachevitch, V. Davydova, E. Tutubalina, BioASQ at CLEF2024: The Twelfth Edition of the Large-Scale Biomedical Semantic Indexing and Question Answering Challenge, 2024, pp. 490–497. doi:10.1007/978-3-031-56069-9_67

    work page doi:10.1007/978-3-031-56069-9_67 2024

  3. [3]

    Jagerman, H

    R. Jagerman, H. Zhuang, Z. Qin, X. Wang, M. Bendersky, Query expansion by prompting large language models, arXiv preprint arXiv:2305.03653 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    L. Wang, N. Yang, F. Wei, Query2doc: Query expansion with large language models, arXiv preprint arXiv:2303.07678 (2023)

    Pith/arXiv arXiv 2023

  5. [5]

    Alaofi, L

    M. Alaofi, L. Gallagher, M. Sanderson, F. Scholer, P. Thomas, Can generative LLMs create query variants for test collections? an exploratory study, in: Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, 2023, pp. 1869–1873

    2023

  6. [6]

    W. Peng, G. Li, Y. Jiang, Z. Wang, D. Ou, X. Zeng, D. Xu, T. Xu, E. Chen, Large language model based long-tail query rewriting in taobao search, in: Companion Proceedings of the ACM Web Conference 2024, 2024, pp. 20–28

    2024

  7. [7]

    X. Ma, Y. Gong, P. He, H. Zhao, N. Duan, Query rewriting in retrieval-augmented large language models, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 5303–5315

    2023

  8. [8]

    Reimers, I

    N. Reimers, I. Gurevych, Sentence-BERT: Sentence embeddings using Siamese BERT-networks, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hon...

    doi:10.18653/v1/d19-1410 2019

  9. [9]

    D. M. Bikel, K. B. Hall, ReFr: An open-source reranker framework, in: Interspeech 2013, 2013, pp. 756–758

    2013

  10. [10]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., Retrieval-augmented generation for knowledge-intensive NLP tasks, in: Advances in Neural Information Processing Systems, volume 33, 2020, pp. 9459–9474

    2020

  11. [11]

    Accessed: 2025

    Google Cloud, Ranking and re-ranking search results, https://cloud.google.com/ generative-ai-app-builder/docs/ranking, 2024. Accessed: 2025

    2024

  12. [12]

    Zaheer, G

    M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, A. Ahmed, Big bird: Transformers for longer sequences, in: Advances in Neural Information Processing Systems, volume 2020-December, 2020

    2020

  13. [13]

    Beltagy, M

    I. Beltagy, M. E. Peters, A. Cohan, Longformer: The long-document transformer (2020)

    2020

  14. [14]

    Karpukhin, B

    V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, W.-t. Yih, Dense passage retrieval for open-domain question answering, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020, pp. 6769–6781

    2020

  15. [15]

    S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, H. Hajishirzi, FActScore: Fine-grained atomic evaluation of factual precision in long form text generation, in: EMNLP, 2023. URL: https://arxiv.org/abs/2305.14251

    Pith/arXiv arXiv 2023

  16. [16]

    Saad-Falcon, O

    J. Saad-Falcon, O. Khattab, C. Potts, M. Zaharia, ARES: An automated evaluation framework for retrieval-augmented generation systems, in: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024, pp. 4392–4408

    2024

  17. [17]

    C. Qu, L. Yang, M. Qiu, W. B. Croft, Y. Zhang, M. Iyyer, BERT with history answer embedding for conversational question answering, in: Proceedings of the 43rd International ACM SIGIR Conference, 2020, pp. 1133–1136

    2020

  18. [18]

    Zhang, Y

    W. Zhang, Y. Liu, H. Chen, X. Wang, To retrieve or not to retrieve? uncertainty detection for dynamic retrieval-augmented generation, arXiv preprint arXiv:2501.09292 (2025). URL: https: //arxiv.org/abs/2501.09292

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.