pith. sign in

arxiv: 2606.19345 · v1 · pith:SXHAN5LDnew · submitted 2026-04-24 · 💻 cs.CL · cs.AI

Ensembles of Large Language Models for Identifying EQ-5D Studies in PubMed Based on Their Abstracts

Zhyar Rzgar K. Rostam , M\'arta P\'entek , J\'anos Tibor Czere , Zsombor Zrubka , L\'aszl\'o Gul\'acsi , G\'abor Kert\'esz This is my paper

Pith reviewed 2026-07-04 16:11 UTC · model glm-5.2

classification 💻 cs.CL cs.AI
keywords modelseq-5dpubmedresultsstudiesabstractsapproachautomating
0
0 comments X

The pith

Ensemble of three LLMs hits 0.74 F1 for EQ-5D screening

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tests whether combining multiple large language models can automate a tedious step in biomedical research: identifying which published studies report results from the EQ-5D health-related quality-of-life instrument. Nine Google Gemini and Gemma models were individually evaluated on 200 PubMed abstracts that two experts had manually labeled for EQ-5D reporting. The top three performers were then combined using a weighted ensemble that scales each model's vote by its individual accuracy and self-reported confidence. The ensemble achieved a weighted F1-score of 0.74 and 0.74 accuracy, marginally exceeding the best single model's 0.71 F1 and 0.73 accuracy. A second combination method, soft stacking, used logistic regression on the models' probability and confidence outputs, achieving 0.72 F1. The paper argues that these ensemble approaches improve the balance between precision and recall over individual models and provide a scalable, cost-effective path toward automating literature screening in health research.

Core claim

The central finding is that a weighted ensemble of gemini-2.5-pro, gemma-3-12b, and gemma-3-27b, where each model's prediction is scaled by its individual F1-score and confidence value, achieves 0.74 weighted F1 on EQ-5D abstract classification, outperforming each model individually. The improvement over the best single model is +0.01 in accuracy and +0.03 in weighted F1. Feature analysis from the soft stacking meta-classifier shows that the models' soft probability outputs carry substantially more discriminative signal than their raw confidence scores.

What carries the argument

The weighted ensemble framework combines three components: (1) few-shot prompting with 40 labeled examples to guide each LLM, (2) per-model weights derived from individual F1-scores, and (3) confidence-weighted voting where each model's binary prediction is multiplied by its normalized confidence score and accumulated across positive and negative classes. The soft stacking framework adds a logistic regression meta-classifier operating on six features: three soft positive-class probabilities and three raw confidence values from the base models.

If this is right

  • If the ensemble approach generalizes, research teams conducting systematic literature reviews could screen thousands of abstracts for specific instrument reporting at a cost of fractions of a cent per abstract, reducing manual screening effort substantially.
  • The finding that soft probabilities dominate raw confidence scores as meta-features suggests that LLM confidence calibration may be less important than the directional probability signal when combining models for classification tasks.
  • The cost-performance tradeoff data indicates that lighter, cheaper models could be deployed in resource-constrained settings while still achieving acceptable screening accuracy, broadening access to automated literature review tools.
  • If ensemble gains hold on larger datasets, the approach could be extended to other clinical instruments or reporting criteria beyond EQ-5D, creating a generalizable pipeline for automating inclusion-criteria screening in systematic reviews.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The +0.03 F1 improvement from ensembling is small enough that it could reflect sampling noise on a 200-publication dataset, particularly since the weighted ensemble appears to be evaluated on the same data used to select the top three models. A held-out test set or bootstrap significance test would be needed to confirm the gain is real.
  • The class imbalance (121 positive vs. 79 negative) and the use of weighted F1 means the ensemble's improvement may be driven by better recall on the majority class rather than genuinely better discrimination, which matters for screening applications where false negatives are costly.
  • If the approach were applied to the full 15,547 EQ-5D publications in the original dataset, the 0.74 F1 would imply a substantial number of misclassifications, raising the question of whether ensemble-based screening is better used as a triage tool with human verification rather than a replacement for manual review.

Load-bearing premise

The paper assumes that a 200-publication dataset, with the weighted ensemble evaluated on the same samples used to select the top three models and no separate held-out test set, is sufficient to demonstrate that the ensemble reliably exceeds individual model performance and is scalable to large-scale biomedical screening.

What would settle it

If the weighted ensemble were evaluated on a separate held-out test set not used for model selection, its F1 improvement over the best single model would likely shrink to within noise, or the ensemble might not exceed the single best model at all.

read the original abstract

The rapid increase in scientific publications leads to the fact that manual study screening in systematic literature reviews (SLRs) is increasingly resource consuming, inefficient, and inconsistent. Classifying studies that clearly report health-related quality-of-life results, such as EQ-5D data, requires a high level of clinical interpretation and poses challenges for human reviewers. This study investigates the use of Google's Gemini and Gemma large language models (LLMs) in automating EQ-5D detection in the PubMed biomedical database based only on published abstracts. A multi-phase framework is proposed that integrates few-shot prompting, weight ensembling aggregation, and a soft stacking meta-classifier. Nine LLMs are evaluated on a dataset of PubMed studies manually labeled by two experts regarding EQ-5D reporting. The weighted ensemble of gemini-2.5-pro, gemma-3-12b, and gemma-3-27b obtained a 0.74 weighted F1-score and 0.74 accuracy, exceeding individually attained results. The ensembling of top-performing models improved the balance between precision and recall compared to individual models, while the soft stacking approach provided greater reliability and interpretability. Feature analysis shows that the probability results from the models are important in guiding the final predictions. The findings suggest that an ensemble-based LLM setup is a reliable and scalable approach for automating screening in biomedical research.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. This paper proposes a multi-phase ensemble framework combining Google Gemini and Gemma LLMs to automatically detect EQ-5D reporting in PubMed abstracts. Nine models are individually evaluated with few-shot prompting on 200 expert-labeled studies; the top three are combined via a weighted ensemble (using F1-scores as weights and confidence voting) and a soft stacking meta-classifier (logistic regression with 5-fold CV). The weighted ensemble achieves 0.74 weighted F1 and 0.74 accuracy, marginally exceeding the best single model (gemini-2.5-pro, F1=0.71, accuracy=0.73). The soft stacking approach achieves F1=0.72 and accuracy=0.73. A cost analysis is provided.

Significance. The application domain—automating EQ-5D study identification for systematic literature reviews—is practically motivated and relevant to health informatics. The dataset is expert-labeled with clear inclusion criteria, and the code is publicly available (GitHub link in Section I), which supports reproducibility. The soft stacking framework with interpretable feature coefficients (Table VI) is a reasonable contribution. However, the central claim that the ensemble 'exceeds individually attained results' rests on a +0.01 accuracy and +0.03 F1 improvement on 200 samples without a held-out test set or significance testing, which substantially limits the significance of the improvement claim.

major comments (3)
  1. Section III.C.4, Eqs. (3)-(5); Section IV.B, Table IV: The weighted ensemble uses each model's individual F1-score (computed on the same 200-sample set) as its weight, and the ensemble is then evaluated on the same 200 samples. This constitutes evaluation on the training data for the ensemble weights. The reported improvement of +0.01 accuracy and +0.03 F1 over the best single model corresponds to approximately 2 additional correct predictions out of 200, which is well within sampling noise. No significance test (e.g., McNemar's test or bootstrap CI) is reported. The claim that the ensemble 'exceeds individually attained results' is not supported without either a held-out test set or a statistical test.
  2. Section III.C.3: The top-3 model selection is based on performance on the same 200-sample evaluation set. This selection step, combined with the weighted ensemble evaluation on the same set, creates a compound overfitting risk: the models are selected and weighted based on their performance on the evaluation data, then evaluated on that same data. The soft stacking approach (Phase 2b) at least uses 5-fold cross-validation, but the best-performing weighted ensemble does not. The paper should either (a) split the data into model-selection/weight-tuning and final-evaluation sets, or (b) apply cross-validation to the weighted ensemble as well.
  3. Abstract and Conclusion: The claims that the approach is 'reliable and scalable' are not substantiated by the experimental design. With 200 samples, single-dataset evaluation, no external validation, and modest absolute performance (0.74 F1 on a binary task), 'reliable' overstates the evidence. The cost analysis (Table VII) addresses scalability in a limited sense but does not address generalization. These claims should be tempered to match the evidence.
minor comments (7)
  1. Table I: Parameter sizes for Gemini models are listed as '-' while Gemma sizes are provided. If the Gemini sizes are not publicly disclosed, this should be noted explicitly rather than left blank.
  2. Section III.B: The 40 few-shot examples are drawn from the same 200-sample dataset. It is unclear whether these 40 examples are excluded from the evaluation set or included. If included, the models see their own few-shot demonstrations during evaluation, which would inflate performance. Please clarify.
  3. Table III: The caption states 'all metrics weighted except Accuracy,' but the Precision and Recall columns appear to be macro-averaged or per-class values rather than weighted. Please verify and clarify which averaging method is used for each column.
  4. Section IV.C, Table V: The soft stacking results show class-specific F1-scores (0.60 for negative, 0.80 for positive) but the table format differs from Table III. Standardizing the table formats would improve readability.
  5. Section III.A: The statement 'These publications were labeled negative and treated as a minor source of noise' regarding non-English full texts is unclear. If the abstracts were in English but full texts were not, please explain how this introduces noise and how it was handled.
  6. Reference [19] (Cao et al.) is cited in Section II.A but the in-text citation does not include author names in the standard format used elsewhere. Please verify the citation style is consistent.
  7. The GitHub repository URL (https://github.com/ZhyarUoS/) should be verified to ensure the repository is publicly accessible and contains the code and data examples referenced.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three interconnected concerns: (1) the weighted ensemble is evaluated on the same data used to compute model weights, creating an evaluation-on-training-data problem; (2) model selection and weight-tuning both use the same 200-sample set, compounding overfitting risk; and (3) the claims of 'reliable and scalable' are overstated given the sample size, single dataset, and modest performance. We agree with all three points and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: Section III.C.4, Eqs. (3)-(5); Section IV.B, Table IV: The weighted ensemble uses each model's individual F1-score (computed on the same 200-sample set) as its weight, and the ensemble is then evaluated on the same 200 samples. This constitutes evaluation on the training data for the ensemble weights. The reported improvement of +0.01 accuracy and +0.03 F1 over the best single model corresponds to approximately 2 additional correct predictions out of 200, which is well within sampling noise. No significance test (e.g., McNemar's test or bootstrap CI) is reported. The claim that the ensemble 'exceeds individually attained results' is not supported without either a held-out test set or a statistical test.

    Authors: The referee is correct on all counts. The weighted ensemble does use F1-scores computed on the same 200-sample set as weights, and the ensemble is then evaluated on those same 200 samples. This is a methodological weakness: the improvement of +0.01 accuracy and +0.03 F1 corresponds to approximately 2 additional correct predictions, which is within sampling noise. We will address this in the revision by: (1) applying 5-fold cross-validation to the weighted ensemble, mirroring the approach already used for the soft stacking method, so that weight computation and evaluation are performed on disjoint folds; (2) reporting McNemar's test between the weighted ensemble and the best single model to assess whether the difference is statistically significant; and (3) revising the language from 'exceeding individually attained results' to a more cautious formulation such as 'showing a modest improvement over the best single model in cross-validated evaluation.' If the cross-validated results do not show a statistically significant improvement, we will state this transparently rather than claiming superiority. revision: yes

  2. Referee: Section III.C.3: The top-3 model selection is based on performance on the same 200-sample evaluation set. This selection step, combined with the weighted ensemble evaluation on the same set, creates a compound overfitting risk: the models are selected and weighted based on their performance on the evaluation data, then evaluated on that same data. The soft stacking approach (Phase 2b) at least uses 5-fold cross-validation, but the best-performing weighted ensemble does not. The paper should either (a) split the data into model-selection/weight-tuning and final-evaluation sets, or (b) apply cross-validation to the weighted ensemble as well.

    Authors: We agree that the compound overfitting risk is real. Model selection, weight computation, and final evaluation all currently operate on the same 200 samples. In the revision, we will apply cross-validation to the entire pipeline: within each fold, model selection and weight tuning will be performed on the training portion only, and evaluation will be performed on the held-out fold. This nested approach ensures that model selection, weight assignment, and performance estimation are properly separated. We will also add an explicit discussion of this overfitting risk in the methodology section and describe the cross-validation procedure for the weighted ensemble in detail. We acknowledge that with only 200 samples, even cross-validated estimates will have wide confidence intervals, and we will state this limitation clearly. revision: yes

  3. Referee: Abstract and Conclusion: The claims that the approach is 'reliable and scalable' are not substantiated by the experimental design. With 200 samples, single-dataset evaluation, no external validation, and modest absolute performance (0.74 F1 on a binary task), 'reliable' overstates the evidence. The cost analysis (Table VII) addresses scalability in a limited sense but does not address generalization. These claims should be tempered to match the evidence.

    Authors: The referee is right that 'reliable and scalable' overstates what the evidence supports. With 200 samples, a single dataset, no external validation, and 0.74 F1 on a binary task, the claim of reliability is not warranted. The cost analysis addresses computational and financial feasibility but not generalization. In the revision, we will: (1) remove 'reliable and scalable' from the abstract and conclusion; (2) replace it with a measured statement such as 'a feasible and cost-effective approach warranting further evaluation on larger datasets'; (3) add to the limitations section that the study lacks external validation and that generalization to other databases or clinical domains has not been tested; and (4) reframe the conclusion to emphasize that the work is a proof-of-concept demonstrating the potential of ensemble LLM approaches for EQ-5D screening, rather than a validated production-ready system. revision: yes

Circularity Check

0 steps flagged

No significant circularity; the weighted ensemble's improvement is empirical, not tautological, though weights are fit on the evaluation set

full rationale

The paper's central claim — that the weighted ensemble (F1=0.74) exceeds the best individual model (F1=0.71) — is not circular by construction. The ensemble weights (w_m = each model's individual F1-score, Eq. 3-5) are indeed computed on the same 200-sample set used for final evaluation, which creates an overfitting risk. However, the ensemble's F1-score is computed from its actual per-sample predictions (combining individual predictions and confidences via Eqs. 3-5), not from a weighted average of the input F1-scores. The ensemble could have performed worse than the best single model; the +0.03 F1 improvement is an empirical outcome, not a definitional identity. The self-citations (Ref [3] for the dataset, Refs [11,18] for related PLM work) are by overlapping authors but are not load-bearing for the ensemble's performance claim — the dataset is independently expert-labeled, and the cited works provide context rather than theoretical constraints that force the present result. The soft stacking approach (Phase 2b) uses 5-fold cross-validation, a standard non-circular methodology. The absence of a held-out test set for the weighted ensemble is a generalization/correctness concern, not a circularity concern. Score: 1 — one minor self-citation for the dataset that is not load-bearing for the central claim.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 0 invented entities

The paper does not invent new entities. It uses existing LLMs and standard machine learning techniques. The free parameters are the ensemble weights and logistic regression coefficients, which are fitted to the 200-sample dataset.

free parameters (2)
  • Model weights (w_m) = 0.71, 0.65, 0.65 (F1-scores)
    The weights for the weighted ensemble are set to the individual F1-scores achieved on the 200-sample dataset, which are fitted to the evaluation data.
  • Logistic regression coefficients (w, b) = Not explicitly listed, but feature coefficients in Table VI
    The meta-classifier coefficients are learned from the 5-fold cross-validation on the 200 samples.
axioms (2)
  • domain assumption The 200 manually labeled studies are a representative sample of PubMed EQ-5D literature.
    The paper acknowledges in Section VII that the 200-study dataset size may limit generalization to broader corpora.
  • domain assumption LLM confidence scores (0-100) are calibrated proxies for prediction probability.
    The weighted ensemble and soft stacking frameworks use the confidence values directly as probabilities or weights, assuming they carry discriminative information, which the feature analysis partially contradicts for the stacking model.

pith-pipeline@v1.1.0-glm · 11750 in / 2133 out tokens · 156536 ms · 2026-07-04T16:11:41.865403+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages

  1. [1]

    Novel deep learning approach for sci- entific literature classification,

    M. M. Ahanger and M. A. Wani, “Novel deep learning approach for sci- entific literature classification,” in2022 9th International Conference on Computing for Sustainable Global Development (INDIACom), pp. 249– 254, IEEE, 2022

    work page 2022

  2. [2]

    Multilabel text classification with label-dependent representation,

    R. Alfaro, H. Allende-Cid, and H. Allende, “Multilabel text classification with label-dependent representation,”Applied Sciences, vol. 13, no. 6, 2023

    work page 2023

  3. [3]

    Towards automating the selection of articles reporting eq-5d data for system- atic literature reviews using large language models

    G. Kert ´esz, J. T. Czere, Z. Zrubka, L. Gul ´acsi, and M. P ´entek, “Towards automating the selection of articles reporting eq-5d data for system- atic literature reviews using large language models.” https://ssrn.com/ abstract=4876024, 2024. Available at SSRN: https://ssrn.com/abstract= 4876024 or http://dx.doi.org/10.2139/ssrn.4876024

    work page doi:10.2139/ssrn.4876024 2024

  4. [4]

    Paying for digital health interventions–what evidence is needed?,

    V . Zah, A. Burrell, C. Asche, and Z. Zrubka, “Paying for digital health interventions–what evidence is needed?,”Acta Polytechnica Hungarica, vol. 19, no. 9, pp. 179–199, 2022

    work page 2022

  5. [5]

    Euroqol - a new facility for the measurement of health-related quality of life,

    “Euroqol - a new facility for the measurement of health-related quality of life,”Health Policy, vol. 16, no. 3, pp. 199–208, 1990

    work page 1990

  6. [6]

    Szende, B

    A. Szende, B. Janssen, and J. Cabases, eds.,Self-Reported Population Health: An International Perspective based on EQ-5D. Dordrecht, Netherlands: Springer Nature, 2014. https://www.ncbi.nlm.nih.gov/ pubmed/29787044, PMID: 29787044

    work page arXiv 2014

  7. [7]

    Survey of pre-trained models for natural language processing,

    J. Peng and K. Han, “Survey of pre-trained models for natural language processing,” in2021 International Conference on Electronic Communi- cations, Internet of Things and Big Data (ICEIB), pp. 277–280, IEEE, 2021

    work page 2021

  8. [8]

    Benchmarking for biomedical natural language processing tasks with a domain specific albert,

    U. Naseem, A. G. Dunn, M. Khushi, and J. Kim, “Benchmarking for biomedical natural language processing tasks with a domain specific albert,”BMC bioinformatics, vol. 23, no. 1, p. 144, 2022

    work page 2022

  9. [9]

    A brief survey of text classification methods,

    Q. Jiao, “A brief survey of text classification methods,” in2023 IEEE 3rd International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA), vol. 3, pp. 1384–1389, IEEE, 2023

    work page 2023

  10. [10]

    Cross-domain limi- tations of neural models on biomedical relation classification,

    I. Alimova, E. Tutubalina, and S. I. Nikolenko, “Cross-domain limi- tations of neural models on biomedical relation classification,”IEEE Access, vol. 10, pp. 1432–1439, 2021

    work page 2021

  11. [11]

    Advances in pre-trained language models for domain-specific text classification: A systematic review,

    Z. R. K. Rostam and G. Kert ´esz, “Advances in pre-trained language models for domain-specific text classification: A systematic review,” ACM Trans. Intell. Syst. Technol., vol. 16, Oct. 2025

    work page 2025

  12. [12]

    Artificial intelligence in medicine: A systematic review of guidelines for the reporting and interpretation of studies,

    Z. Zrubka, L. Kov ´acs, H. Motahari Nezhad, J. Czere, L. Gul ´acsi, and M. P ´entek, “Artificial intelligence in medicine: A systematic review of guidelines for the reporting and interpretation of studies,”Acta Polytechnica Hungarica, vol. 22, no. 10, pp. 125–143, 2025

    work page 2025

  13. [13]

    Scibert: A pretrained language model for scientific text,

    I. Beltagy, K. Lo, and A. Cohan, “Scibert: A pretrained language model for scientific text,”arXiv preprint arXiv:1903.10676, 2019

    work page arXiv 1903

  14. [14]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining,

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,”Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020

    work page 2020

  15. [15]

    Transfer learning in biomedical natural lan- guage processing: An evaluation of bert and elmo on ten benchmarking datasets,

    Y . Peng, S. Yan, and Z. Lu, “Transfer learning in biomedical natural lan- guage processing: An evaluation of bert and elmo on ten benchmarking datasets,” inProceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pp. 58–65, 2019

    work page 2019

  16. [16]

    Finbert: A pre-trained financial language representation model for financial text mining,

    Z. Liu, D. Huang, K. Huang, Z. Li, and J. Zhao, “Finbert: A pre-trained financial language representation model for financial text mining,” in Proceedings of the twenty-ninth international conference on interna- tional joint conferences on artificial intelligence, pp. 4513–4519, 2021

    work page 2021

  17. [17]

    Medibiodeberta: Biomedical lan- guage model with continuous learning and intermediate fine-tuning,

    E. Kim, Y . Jeong, and M.-s. Choi, “Medibiodeberta: Biomedical lan- guage model with continuous learning and intermediate fine-tuning,” IEEE Access, 2023

    work page 2023

  18. [18]

    Fine-tuning large language models for scientific text classification: A comparative study,

    Z. R. K. Rostam and G. Kert ´esz, “Fine-tuning large language models for scientific text classification: A comparative study,” in2024 IEEE 6th International Symposium on Logistics and Industrial Informatics (LINDI), pp. 000233–000238, 2024

    work page 2024

  19. [19]

    Development of prompt templates for large language model–driven screening in systematic reviews,

    “Development of prompt templates for large language model–driven screening in systematic reviews,”Annals of Internal Medicine, vol. 178, no. 3, pp. 389–401, 2025. PMID: 39993313

    work page 2025

  20. [20]

    Generative ai text classification using ensemble llm approaches,

    H. Abburi, M. Suesserman, N. Pudota, B. Veeramani, E. Bowen, and S. Bhattacharya, “Generative ai text classification using ensemble llm approaches,”arXiv preprint arXiv:2309.07755, 2023

    work page arXiv 2023

  21. [21]

    Enstack: An ensemble stacking framework of large language models for enhanced vulnerability detection in source code,

    S. Z. Ridoy, M. Shazzad Hossain Shaon, A. Cuzzocrea, and M. S. Akter, “Enstack: An ensemble stacking framework of large language models for enhanced vulnerability detection in source code,” in2024 IEEE International Conference on Big Data (BigData), pp. 6356–6364, 2024

    work page 2024

  22. [22]

    Pubmed 2.0,

    J. White, “Pubmed 2.0,”Medical reference services quarterly, vol. 39, no. 4, pp. 382–387, 2020

    work page 2020

  23. [23]

    Gemini API pricing

    Google, “Gemini API pricing.” https://ai.google.dev/gemini- api/docs/pricing, 2026. Accessed: Feb. 11, 2026

    work page 2026

  24. [24]

    Google API pricing

    PricePerToken, “Google API pricing.” https://pricepertoken.com/pricing- page/provider/google, 2026. Accessed: Feb. 11, 2026

    work page 2026