Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach

Alexandros C. Dimopoulos; Dimitra Panou; Manolis Koubarakis; Martin Reczko

REVIEW 3 major objections 5 minor 17 references

Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Combining thirteen open-source LLMs with majority voting for yes/no and answer union for list/factoid questions consistently outperforms individual models on biomedical QA, as shown by top placements in the 2025 BioASQ Synergy track.

desk verdict Real BioASQ results, but the 'consistently superior' combination claim is unsupported because the same four rounds were used for selection and evaluation. read the letter →

arxiv 2508.01480 v1 pith:PJ3XDEDF submitted 2025-08-02 cs.CL

Dimitra Panou , Alexandros C. Dimopoulos , Manolis Koubarakis , Martin Reczko This is my paper

classification cs.CL

keywords biomedicalquestionansweringBioASQlargelanguagemodelsensemblemethodsmajorityvotinganswerunionretrieval-augmentedgenerationopen-sourceLLMs

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports on a biomedical question-answering system built from thirteen open-source large language models and submitted to the 2025 BioASQ challenge. The authors' central claim is that combining model outputs by question type—majority voting for yes/no questions and answer union for list and factoid questions—and choosing the best-performing model subset for each type consistently produces superior results over any single model. In the Synergy track they achieved first place for ideal answers and second for exact answers in round 2, and shared first place for exact answers in rounds 3 and 4. The result matters because it shows that transparent, locally runnable open-source models, assembled with a simple aggregation rule, can compete at the top level of a demanding biomedical retrieval-and-generation benchmark.

What carries the argument

The central mechanism is the committee-of-models aggregation with per-type voting rules. Majority voting takes the most common yes/no answer among the selected models as the final answer, while union aggregation collects every distinct answer item proposed by any selected model, maximizing recall for lists and factoids. The selection of which models enter the committee was made by exhaustively searching all non-empty subsets of the thirteen models on the challenge's development and test rounds, so the pipeline is tailored to each question type. The retrieval stage, BM25 with RM3 pseudo-relevance feedback followed by snippet re-ranking, supplies the evidence passages that condition all generators.

What would settle it

Take the same thirteen-model committee and the same aggregation rules, but replace the retrieval stage with randomly chosen document snippets; if the ensemble still answers a substantial share of BioASQ questions correctly, the retrieval-completeness assumption is not load-bearing. Alternatively, run the identical pipeline with the single best model from the paper's evaluation in place of the committee; if the single model matches the ensemble's accuracy on fresh biomedical questions, the claimed superiority of the multi-model aggregation would be refuted.

Watch

Extended reading notes

Core claim

The paper's core discovery is that a carefully selected ensemble of open-source LLMs outperforms each constituent model on biomedical question answering, provided the aggregation rule matches the question type and the model combination is chosen by exhaustive evaluation on the challenge's own test rounds. For yes/no questions, majority voting over the models' answers yields the final response; for list and factoid questions, the final answer is the union of the answers produced by the selected models. The authors evaluated all possible combinations of the thirteen models and found that different question types favor different subsets, which they interpret as evidence that no single open-source model is sufficient but a tailored committee is. Their reported BioASQ 2025 Synergy results—first place for ideal answers in round 2, plus shared first places for exact answers in rounds 3 and 4—are presented as evidence of the approach's effectiveness.

Load-bearing premise

The system's answers are only as good as the retrieved evidence: if BM25 plus RM3 re-ranked snippets miss the documents that contain the answer, no aggregation of language models can recover the correct response.

Editorial extensions

If this is right

If the central claim holds, teams without access to closed commercial models can build state-of-the-art biomedical QA systems using only open-source LLMs and a simple aggregation rule.
Question-type-specific aggregation becomes a practical design principle: yes/no questions call for majority vote, while list and factoid questions call for answer union.
Exhaustive search over model subsets, though computationally expensive, yields concrete best-committee configurations for each question type that future systems can adopt directly.
The retrieval pipeline remains the floor for system performance; the paper's results indicate that generation-side gains from ensembling are real but cannot compensate for missing evidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The exhaustive combination search was performed on BioASQ's own rounds, so the selected per-type committees may overfit to the specific question distribution of the 2025 benchmark; a fairer test would hold out future BioASQ rounds or an independent biomedical QA set.
The union strategy for list and factoid questions likely raises recall but may admit false positives; the paper does not report precision separately, so a testable extension is to measure precision-recall trade-offs per question type.
The same voting/union ensemble idea could transfer to other expert domains, such as legal, financial, or clinical decision support, where open-source models and evidence retrieval are used, provided a per-question-type evaluation is repeated.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper is a legitimate BioASQ 2025 system description with real external evaluations, not a toy experiment. The authors deployed 13 open-source LLMs, used majority voting for yes/no questions and answer union for list/factoid questions, and report top placements in the Synergy task (1st ideal in round 2, shared 1st exact in rounds 3 and 4). That is useful, concrete evidence for practitioners. The retrieval pipeline (BM25+RM3, snippet re-ranking) is clearly described, and the paper is honest about what it did. Credit where it is due: the system works, and the write-up is readable.

The soft spot is the central analytical claim. The abstract says they explored 'all possible model combinations' and found combinations that 'consistently produce superior results' per question type. With 13 models, that is 8191 subsets per type. If the selection and the reported scores come from the same four BioASQ rounds, the best combination is likely to win by chance on tens of questions per type. The paper reports no significance test, no confidence intervals, and no held-out split (e.g., selecting on BioASQ 2024 and testing on 2025). So the word 'consistently' is not backed by the evidence. The stress-test note is right: even with perfect retrieval, the generalization claim remains unproven unless selection is independent of evaluation. The retrieval-completeness issue is real but secondary; the paper never claims to solve retrieval, and its focus is generation.

That said, this is a shared-task system paper, and the bar for such papers is usually lower. The authors are not hiding anything; the method is transparent. The challenge results themselves are valuable, and the per-question-type ensemble mapping is a practical recipe others can copy. But the overclaim should be fixed. The authors should either soften 'consistently' to something like 'on the 2025 BioASQ rounds' or add an out-of-sample check, even a simple per-round leave-one-out analysis, and compare against the single best model. With tens of questions, even a small table of per-round wins/losses would help.

Who is this for? Anyone building RAG pipelines for biomedical QA, BioASQ participants, and people curious about whether open-source LLM ensembles beat single models on real benchmark data. It is an incremental engineering contribution, not a new capability. I would cite it as a data point in a related work section, but not as proof of ensemble superiority.

My recommendation: send it to peer review, but ask for revision. The paper deserves referee time because the BioASQ placements are real and the method is clearly specified. A serious referee should push for significance testing or a softened claim, but this is not a reject. It is a conditional accept with the stated caveats.

Referee Report

3 major / 5 minor

Summary. This manuscript reports the authors' participation in BioASQ 2025 (Task 13b and the Synergy task), proposing a retrieval-augmented question-answering system that combines answers from multiple open-source LLMs. For yes/no questions, a majority vote determines the final answer; for list and factoid questions, the union of answers is used. The authors state that they evaluated 13 open-source LLMs and explored all possible model combinations, obtaining 'tailored LLM pipelines' per question type. They report notable BioASQ placements, including 1st place in ideal answers in Synergy round 2 and shared 1st places in exact answers in rounds 3 and 4, and claim that certain combinations 'consistently produce superior results' for specific question types.

Significance. If the central claim of consistent, superior model combinations were properly supported, the paper would provide practically valuable guidance for ensemble-based biomedical QA systems, particularly in resource-constrained settings that rely on open-source models. The reported BioASQ results are real external evaluations, and the challenge setting lends ecological validity to the system-level comparison. However, the paper as written does not establish the generalization claim: the selection of the best combination appears to be performed on the same evaluation data used to report the results, and no statistical testing or held-out validation is provided. The manuscript also lacks the experimental detail needed to reproduce or assess the approach. Consequently, the current contribution is more a challenge report than a validated scientific study, and the significance of the findings is not yet demonstrated.

major comments (3)

[Abstract] The central claim that certain LLM combinations 'consistently produce superior results' is not supported by the evidence presented. The authors state that they explored 'all possible model combinations' (Abstract), which for 13 models means 2^13-1 = 8191 subsets per question type. If the selection was performed on the same BioASQ rounds used to report the final scores, then the reported performance is in-sample and vulnerable to selection bias; with only tens of questions per type per round, the best combination can outperform alternatives by chance. The manuscript provides no description of a held-out development set, no time-based split, no confidence intervals, and no significance tests. To support the generalization claim, the authors must either select combinations on a separate development set (e.g., previous BioASQ rounds) and evaluate on 2025 rounds, or apply multiple-testing corrections and report measures of uncertainty.
[Section 2.2 (and overall structure)] The manuscript is substantially incomplete. The full text ends at Section 2.2 with a description of snippet prediction, and there is no results section, no list of the 13 evaluated models, no specification of the majority-voting or union-aggregation procedures, no description of how the 'tailored LLM pipelines' were derived per question type, and no quantitative comparison of the selected combinations against single models or other baselines. Without these components, the reader cannot assess the validity of the central claim or the reproducibility of the approach. The paper needs to be expanded with a complete methodology, results, and analysis before it can be evaluated as a scientific contribution.
[Section 2.2] The retrieval stage is described as BM25+RM3 followed by snippet-based re-ranking of the top 50 candidate documents. The paper does not address the risk that the relevant document is not retrieved at all. Since all answer generation is conditioned on the retrieved snippets, the system's accuracy is bounded by retrieval completeness. If the retrieval stage misses the relevant information, no ensemble of generators can produce the correct answer. The authors should report retrieval performance (e.g., MAP or recall at k) for the same rounds and discuss how retrieval errors propagate to the final answers. This concern is not the load-bearing issue with the central claim, but it is a limitation that should be acknowledged and quantified.

minor comments (5)

[Abstract] There is a typographical error: 'the union of their answers in used' should read 'the union of their answers is used'.
[Section 2.1] The manuscript states that the Synergy task used 'the same methods as for the final submissions of the BioASQ12 competition' with the addition of a DeepSeek-R1 variant, but it does not specify which variant or how it was integrated. Please provide the exact model and the prompting or reasoning configuration.
[Abstract / Section 2] The phrase 'tailored LLM pipelines for each question type' is vague. The authors should specify, for each question type, which models and which aggregation rule (majority vote, union, or other) were selected, and how the selection was performed.
[Throughout] References [7] and [8] are cited for critical components, but the present manuscript does not summarize the essential details from those works. A reader of this paper alone cannot understand the snippet prediction method or the DeepSeek-R1 setup. Please make the paper self-contained to the extent possible.
[Overall] There are no tables or figures in the manuscript. Given that the paper reports rankings and system performance, a table summarizing results for each round and question type, and another table listing the evaluated models and combinations, would greatly improve readability and verifiability.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'tailored LLM pipelines' are selected by exhaustive search over the same BioASQ rounds used as evidence, so the central generalization claim reduces to an in-sample optimum.

fitted input called prediction [Abstract]
"We evaluated 13 state-of-the-art open source LLMs, exploring all possible model combinations to contribute to the final answer, resulting in tailored LLM pipelines for each question type. Our findings provide valuable insight into which combinations of LLMs consistently produce superior results for specific question types."

The tailored pipelines are not independently derived predictions: they are the output of an exhaustive search over all non-empty model subsets per question type, and the only reported evidence of 'consistently superior' performance is the score of the selected combination on the same four BioASQ 2025 rounds. If selection and evaluation share the same question set, then by construction the chosen combination has maximal (or near-maximal) score on that set; reporting that score as evidence of consistent superiority is reporting the objective value of the fitted selection, not an out-of-sample validation.

full rationale

The BioASQ leaderboard placements are genuine external system outputs and the majority-vote/union aggregation rules are stated independently; those parts are not circular. The circular step is confined to the paper's central generalization claim that the selected combinations 'consistently produce superior results'. Since the combination selector appears to be fitted on the same four rounds used for the reported scores, the improvement of the chosen pipeline over other combinations is an in-sample fact, not a prediction. A score of 6 reflects this partial, load-bearing circularity; the exact-answer placements retain independent value because they are externally evaluated, but they do not validate the tailored-combination insight without a selection/evaluation split.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The system introduces no new entities or theoretical constructs. Its empirical contribution rests on two domain assumptions: retrieval completeness and model output diversity. The main fitted quantity is the per-question-type model combination, chosen by exhaustive search.

free parameters (1)

best_llm_combination_per_question_type = not disclosed; chosen by exhaustive search among 13 models
The tailored pipeline for each question type is selected by evaluating all combinations on challenge data; this is a fitted selection, not a theoretically derived one.

assumptions (2)

domain assumption BM25+RM3 retrieval with snippet re-ranking retrieves all evidence needed to answer the questions.
Invoked in Section 2.2 for Phase A; if retrieval misses relevant documents, downstream ensemble accuracy is capped regardless of generator choice.
domain assumption Open-source LLM outputs are diverse enough that majority voting and union improve answer quality over single models.
Core mechanism of the ensemble in Sections 2.1 and the abstract; no diversity or correlation analysis is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach." pith.science (2026). https://pith.science/paper/PJ3XDEDF

@misc{pith2026250801480,
  author       = {Pith},
  title        = {Pith review of: Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJ3XDEDF}},
  note         = {Machine review of arXiv:2508.01480}
}

read the original abstract

Biomedical text mining and question-answering are essential yet highly demanding tasks, particularly in the face of the exponential growth of biomedical literature. In this work, we present our participation in the 13th edition of the BioASQ challenge, which involves biomedical semantic question-answering for Task 13b and biomedical question-answering for developing topics for the Synergy task. We deploy a selection of open-source large language models (LLMs) as retrieval-augmented generators to answer biomedical questions. Various models are used to process the questions. A majority voting system combines their output to determine the final answer for Yes/No questions, while for list and factoid type questions, the union of their answers in used. We evaluated 13 state-of-the-art open source LLMs, exploring all possible model combinations to contribute to the final answer, resulting in tailored LLM pipelines for each question type. Our findings provide valuable insight into which combinations of LLMs consistently produce superior results for specific question types. In the four rounds of the 2025 BioASQ challenge, our system achieved notable results: in the Synergy task, we secured 1st place for ideal answers and 2nd place for exact answers in round 2, as well as two shared 1st places for exact answers in round 3 and 4.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...
[2]

Achiam , author S

author OpenAI , author J. Achiam , author S. Adler , author S. Agarwal , author L. Ahmad , author I. Akkaya , author F. L. Aleman , author D. Almeida , author J. Altenschmidt , author S. Altman , author et al. (281 authors) , title Gpt-4 technical report , year 2024 . https://arxiv.org/abs/2303.08774

arXiv 2024
[3]

Mandikal , author R

author P. Mandikal , author R. Mooney , title Sparse meets dense: A hybrid approach to enhance scientific document retrieval , year 2024 . https://arxiv.org/abs/2401.04055

arXiv 2024
[4]

Lee , author W

author J. Lee , author W. Yoon , author S. Kim , author D. Kim , author S. Kim , author C. H. So , author J. Kang , title Biobert: a pre-trained biomedical language representation model for biomedical text mining , journal Bioinformatics volume 36 ( year 2019 ) pages 1234--1240 . https://doi.org/10.1093/bioinformatics/btz682

doi:10.1093/bioinformatics/btz682 2019
[5]

Gu , author R

author Y. Gu , author R. Tinn , author H. Cheng , author M. Lucas , author N. Usuyama , author X. Liu , author T. Naumann , author J. Gao , author H. Poon , title Domain-specific language model pretraining for biomedical natural language processing , year 2020 . http://arxiv.org/abs/arXiv:2007.15779 arXiv:arXiv:2007.15779

arXiv 2020
[6]

Reczko , title Electrolbert: Combining replaced token detection and sentence order prediction

author M. Reczko , title Electrolbert: Combining replaced token detection and sentence order prediction. , in: booktitle CLEF (Working Notes) , year 2022 , pp. pages 335--340

work page 2022
[7]

author D. N. Panou , author M. Reczko , title Semi-supervised training for biomedical question answering. , in: booktitle CLEF (Working Notes) , year 2023 , pp. pages 152--158

work page 2023
[8]

Panou , author A

author D. Panou , author A. Dimopoulos , author M. Reczko , title Farming open llms for biomedical question answering , journal CLEF Working Notes ( year 2024 ). https://doi.org/10.5281/zenodo.13683433

work page doi:10.5281/zenodo.13683433 2024

Show all 17 references

[9]

https://huggingface.co/second-state/DeepSeek-R1-Distill-Llama-70B-GGUF

author DeepSeek-R1-Distill-Llama-70B-GGUF , title Huggingface , year 2025 . https://huggingface.co/second-state/DeepSeek-R1-Distill-Llama-70B-GGUF

2025
[10]

Nentidis , author G

author A. Nentidis , author G. Katsimpras , author A. Krithara , author S. Lima-L\' o pez , author E. Farr\' e -Maduell , author M. Krallinger , author N. Loukachevitch , author V. Davydova , author E. Tutubalina , author G. Paliouras , title Overview of bioasq 2024: The twelf...

2024
[11]

Nentidis , author G

author A. Nentidis , author G. Katsimpras , author A. Krithara , author M. Krallinger , author M. R. Ortega , author N. Loukachevitch , author A. Sakhovskiy , author E. Tutubalina , author G. Tsoumakas , author G. Giannakoulas , et al., title Bioasq at clef2025: The thirteenth...

2025
[12]

Robertson , author H

author S. Robertson , author H. Zaragoza , title The probabilistic relevance framework: Bm25 and beyond , journal Found. Trends Inf. Retr. volume 3 ( year 2009 ) pages 333–389 . https://doi.org/10.1561/1500000019

2009 doi
[13]

Robertson , author S

author S. Robertson , author S. Walker , author M. Beaulieu , title Experimentation as a way of life: Okapi at trec , journal Information Processing & Management volume 36 ( year 2000 ) pages 95--108 . https://www.sciencedirect.com/science/article/pii/S0306457399000461

2000
[14]

Lavrenko , author W

author V. Lavrenko , author W. B. Croft , title Relevance based language models , in: booktitle Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '01, publisher Association for Computing Machinery , a...

2001
[15]

https://github.com/ollama/ollama

author Ollama , title github , year 2024 . https://github.com/ollama/ollama

2024
[16]

AI , title Lightning studio (lm studio) , howpublished https://lightning.ai/docs/studio/ , year 2023

author L. AI , title Lightning studio (lm studio) , howpublished https://lightning.ai/docs/studio/ , year 2023 . note Software

2023
[17]

Malakasiotis , author I

author P. Malakasiotis , author I. Pavlopoulos , author I. Androutsopoulos , author A. Nentidis , title Evaluation measures for task b , journal BioASQ-EvalMeasures-taskB ( year 2018 )

2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

[2] [2]

Achiam , author S

author OpenAI , author J. Achiam , author S. Adler , author S. Agarwal , author L. Ahmad , author I. Akkaya , author F. L. Aleman , author D. Almeida , author J. Altenschmidt , author S. Altman , author et al. (281 authors) , title Gpt-4 technical report , year 2024 . https://arxiv.org/abs/2303.08774

arXiv 2024

[3] [3]

Mandikal , author R

author P. Mandikal , author R. Mooney , title Sparse meets dense: A hybrid approach to enhance scientific document retrieval , year 2024 . https://arxiv.org/abs/2401.04055

arXiv 2024

[4] [4]

Lee , author W

author J. Lee , author W. Yoon , author S. Kim , author D. Kim , author S. Kim , author C. H. So , author J. Kang , title Biobert: a pre-trained biomedical language representation model for biomedical text mining , journal Bioinformatics volume 36 ( year 2019 ) pages 1234--1240 . https://doi.org/10.1093/bioinformatics/btz682

doi:10.1093/bioinformatics/btz682 2019

[5] [5]

Gu , author R

author Y. Gu , author R. Tinn , author H. Cheng , author M. Lucas , author N. Usuyama , author X. Liu , author T. Naumann , author J. Gao , author H. Poon , title Domain-specific language model pretraining for biomedical natural language processing , year 2020 . http://arxiv.org/abs/arXiv:2007.15779 arXiv:arXiv:2007.15779

arXiv 2020

[6] [6]

Reczko , title Electrolbert: Combining replaced token detection and sentence order prediction

author M. Reczko , title Electrolbert: Combining replaced token detection and sentence order prediction. , in: booktitle CLEF (Working Notes) , year 2022 , pp. pages 335--340

work page 2022

[7] [7]

author D. N. Panou , author M. Reczko , title Semi-supervised training for biomedical question answering. , in: booktitle CLEF (Working Notes) , year 2023 , pp. pages 152--158

work page 2023

[8] [8]

Panou , author A

author D. Panou , author A. Dimopoulos , author M. Reczko , title Farming open llms for biomedical question answering , journal CLEF Working Notes ( year 2024 ). https://doi.org/10.5281/zenodo.13683433

work page doi:10.5281/zenodo.13683433 2024