Pith. sign in

REVIEW 3 major objections 6 minor 51 references

Metric structure, not model strength, decides when an LLM judge is enough and when an aggregator can beat it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A metric-aware split between LLM-as-judge and deterministic synonym-union aggregation improves BioASQ list recall and yields first-place combined-exact results on three preliminary leaderboards.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Useful metric-structure lens and a careful ablation, but the resolver recall gain hinges on an unverified scorer-semantics assumption. the 3 major comments →

arxiv 2607.13551 v1 pith:RNIM5U42 submitted 2026-07-15 cs.CL

Cost-Pragmatic Quality Gating and Selection-Fusion Multi-Model Combiners for BioASQ Phases A+ and B

Dima Galat , Marian-Andrei Rizoiu This is my paper
classification cs.CL
keywords BioASQbiomedical question answeringretrieval-augmented generationLLM-as-judgesynonym unionmulti-model ensemblingselection-fusion decompositioncost-pragmatic re-retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the right way to combine multiple language-model answers in the BioASQ biomedical question-answering challenge is determined by the scoring metric's structure, not by the toolkit. It proves a simple lemma: any combiner that must return one candidate verbatim (an LLM-as-judge) is capped by the per-question best single head, while a deterministic aggregator that unions synonym sets can exceed that ceiling on metrics that reward larger candidate sets. The paper shows this split empirically: a synonym-union resolver wins list recall on Task 13B 2025, while an LLM judge is sufficient for yes/no and summary metrics. The practical contribution is a cost-pragmatic retrieval policy that re-retrieves only weakly-supported questions and saves 12% cost while improving list F1, plus a live-leaderboard showing the system placing first on three of eight combined-exact boards in Task 14B 2026.

Core claim

The central claim is a selection-fusion decomposition: any selector (a judge that returns one head's answer verbatim) is bounded above by the per-question oracle, while an aggregator (a deterministic rule constructing a new candidate, like synonym-union) can exceed the oracle on metrics that reward candidate-set enlargement. The paper predicts before any experiment that LLM-as-judge wins on selection-dominated metrics (yes/no accuracy, multi-reference ROUGE) but is structurally insufficient on the recall component of fusion-friendly metrics (factoid rank-1, list recall). The empirical confirmation is that the synonym-union resolver wins list recall on Task 13B 2025 by +0.025 over the best si

What carries the argument

The central object is the selection-fusion decomposition with Lemma 3 (the 'judge ceiling'): for any selector J that returns a candidate verbatim, the expected score is bounded by the selection oracle S(Omega*), while an aggregator Phi can achieve positive fusion lift on metrics where unioning candidates improves the score. The always-pick floor diagnostic (Observation 4) is a cheap test: any judge scoring below the best constant-head selector is destroying value. The synonym-union resolver is the implemented aggregator for list and factoid tracks.

Load-bearing premise

The synonym-union resolver's recall gain assumes the official BioASQ list evaluator matches any normalized synonym form in the gold reference set; if the evaluator is stricter, the union's within-item synonym gains vanish and the predicted fusion lift does not follow.

What would settle it

Take a set of BioASQ list questions, run the same three answer heads, then replace the resolver's within-item synonym union with a verbatim union of items per head (no synonym merging). If list recall does not drop by approximately the reported +0.025 margin, the synonym-matching mechanism is not the cause, and the metric-structure principle would need re-examination.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the metric-structure principle is correct, teams can decide a priori whether to invest in a judge or an aggregator based on the target metric, avoiding wasted effort on LLM judge tuning for recall-oriented tracks.
  • The synonym-union resolver's within-item synonym merging is precision-preserving under BioASQ's gold-reference-set matching, so any multi-head system can safely adopt it for list recall gains without hurting precision.
  • The always-pick floor offers a trivial deployment diagnostic: any deployed judge that scores below the best constant head must be replaced, and the paper shows this catches real regressions on factoid and summary metrics.
  • The cost-pragmatic re-retrieval policy shows that selective re-retrieval (Tier 1 plus rescue) beats indiscriminate re-retrieval on list F1 at 12% lower cost, suggesting that quality-gated retrieval is a scalable strategy for expensive agent loops.
  • If the val-to-test-gold gap of +0.132 list F1 is retrieval-recoverable, then retrieval-side improvements (denser indexes, citation expansion) are the highest-leverage remaining gain for list questions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The judge-ceiling lemma generalizes beyond BioASQ: any multi-LLM ensembling task that evaluates with set-enlargement-friendly metrics (e.g., recall-oriented QA, entity linking, or retrieval) should prefer deterministic union-based aggregators over pick-one judges, while precision-dominated tasks should stay with judges.
  • The paper's own data suggests a constructive judge that performs synonym-union and item-level union when heads agree could extend LLM judging to fusion-friendly metrics, but it is not implemented here; that would be a testable next step.
  • The b4 weakness (list F1 gap to competitor families) likely stems from the 3-stage team's Validator normalizing away synonym variants; the paper suggests ablating that pass, which is a concrete, low-cost experiment to recover recall.
  • The reported live leaderboard standings are self-reported and preliminary; the real test will be the final official ranking, especially for the b3 ideal #1 which is only auto-ROUGE so far.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper describes a BioASQ Task 14B 2026 system with two main design contributions: a cost-pragmatic re-retrieval policy gated by a BGE cross-encoder, and a selection–fusion decomposition of multi-model ensemble combiner choice. The selection–fusion framework proves a judge ceiling (Lemma 3) and argues that LLM-as-judge is sufficient for selection-dominated metrics (yes/no, multi-reference ROUGE) but structurally insufficient for fusion-friendly metrics such as factoid rank-1 and list recall. The paper reports that a deterministic synonym-union resolver significantly improves list recall on Task 13B 2025 over all single heads, and that the system places first on three of eight preliminary Task 14B 2026 combined-exact leaderboards. The paper also reports a controlled ablation of re-retrieval policies on Task 12B 2024 validation, showing a cost-pragmatic policy wins list F1 and precision at lower cost.

Significance. If the central claims hold, the paper makes a useful methodological contribution: it gives a clean, formal distinction between selection and aggregation in multi-model biomedical QA, and it provides empirical evidence on the under-explored role of synonym-union aggregation for recall-oriented metrics. The paper is commendable for using paired-bootstrap confidence intervals, for reporting non-inferiority margins, and for stating several limitations honestly (e.g., small per-type n, val-to-test question-set differences, internal ROUGE scorer differences). The Task 13B list-recall result and the leaderboard placements, if verifiable, would be valuable evidence for the metric-structure principle. However, the central empirical explanation depends on an externally specified property of the BioASQ scorer that is not yet documented or verified in the paper, and the headline re-retrieval significance is computed on the same validation set used to select the policy. These issues currently prevent the paper from being fully accepted as a self-contained verification of its framework.

major comments (3)
  1. [§8.1 and §H] The paper's preferred explanation for the resolver's list-recall gain is that BioASQ list F1 counts any synonym in the gold reference set as a hit, making within-item synonym union precision-preserving. This is stated repeatedly (§8.1, §H) but no citation, scorer documentation, or transcript is provided. If the official scorer uses exact normalized matching or treats each emitted synonym as a distinct item, the reported +0.0249 recall gain (Table 5) would instead come from cross-item additions, which §6.4 itself flags as precision-risky and head-dependent. That would sever the link between Lemma 3's fusion-friendly taxonomy and the headline empirical win. Please verify the scorer semantics (e.g., with documentation or a minimal scoring example) or re-attribute the gain; this is load-bearing for the paper's central empirical claim.
  2. [§5.1, Table 2] The re-retrieval policy comparison selects Pipeline B as production after seeing its validation performance, and the paired-bootstrap CIs in Table 2 are then reported as evidence that Pipeline B 'significantly beats' Pipeline A. Because the policy and the significance test share the same 340 validation questions, the CIs are not selection-adjusted; the result is vulnerable to overfitting to the validation set. The paper does not report an out-of-sample confirmation of the re-retrieval policy (test 13B was not run through the retrieval pipeline; see §7.2 and §8.3). Please either add selection-aware inference, report a pre-registered or held-out policy evaluation, or explicitly temper the significance claim.
  3. [§7.3, Table 7] The headline claim of three first-place combined-exact leaderboard finishes rests entirely on self-reported preliminary leaderboard numbers. Table 7 also mixes distinct submission slots and batches, and the slot conventions changed across batches. Since no leaderboard screenshots, raw score dumps, or submission logs are provided, a reader cannot independently verify the standings. Given that this is a central competitive claim, please include a verifiable artifact (screenshot or official leaderboard record) and clarify the exact submission-to-team mapping for each cell. The current presentation is not reproducible without trusting the authors' transcription.
minor comments (6)
  1. [Abstract / Table 5] Abstract states resolver list-recall gain is +0.025; Table 5 gives +0.0249. Use one consistent decimal representation.
  2. [Table 4] The 'all' row says n=339 while §7.2 and other tables state n=340. Reconcile the missing question.
  3. [Appendices H / §8.1] Appendix H repeats the 'why list F1 benefits' and 'synonym-union free lunch' explanations almost verbatim from §8.1. Consolidate to avoid redundancy.
  4. [§6.4 / §8.2] The always-pick floor is illustrated with a GPT-5.4 judge on Task 13B, while Table 4 uses GPT-5.5 heads. Clarify the head set and temporal ordering so the diagnostic example is unambiguous.
  5. [Author affiliation line] The email/affiliation line contains garbled text ('envel⌢pe-⌢pendima.galat'); please fix the typesetting.
  6. [§8.3] The limitation about internal max-over-references ROUGE is clearly stated, but consider adding a footnote near Appendix D and Table 6 so readers do not mistake absolute ROUGE values for official BioASQ scores.

Circularity Check

1 steps flagged

Partial circularity: the resolver's list-recall win is an analytic consequence of unioning candidates, not an independent confirmation of the selection–fusion theory; the rest of the empirical work is self-contained.

specific steps
  1. self definitional [§6.4 (Synonym-union resolver), Implementation steps (i)–(v); Table 4 / §7.2]
    "For list answers we use a deterministic synonym-union aggregator rather than an LLM judge. ... (iv) Per class, take the union of all synonyms and pick a canonical rank-1 form ... the resolver’s list recall on Task 13B 2025 is 0.653 vs. the best single head’s 0.644."

    The resolver is constructed to output the union of all heads' list items. For any gold set G and answer sets A_h, recall(∪A_h) = |G ∩ ∪A_h| / |G| ≥ max_h |G ∩ A_h| / |G| = max_h recall(A_h). So 'resolver wins list recall on every head' is mathematically forced by the union construction, absent truncation by the 12-item cap. The paper itself defines list recall as fusion-friendly precisely because 'item-level union raises recall', then presents the same identity as an empirical confirmation of the framework. The empirical content is limited to the magnitude of the gain (+0.025, CI [+0.009,+0.045]) and the precision/F1 cost; the direction is analytic, not a test of Lemma 3.

full rationale

The paper's formal apparatus is mostly self-contained and not built on self-citation. Lemma 3 is an elementary pointwise bound that the authors explicitly call trivial; the re-retrieval ablation (§5/§7.1) is a controlled comparison with paired bootstrap CIs; the val-to-test-gold gap is explicitly hedged in §8.3 as consistent-with rather than clean attribution. All references are external; no author self-citation is load-bearing, and no uniqueness theorem is imported. The one genuine circularity is in presenting the synonym-union resolver's list-recall win as empirical confirmation of the selection–fusion prediction. Because the resolver is defined as the union of the candidates' items, and because list recall is classified as fusion-friendly precisely because item-level union raises recall, the inequality recall(resolver) ≥ recall(each head) is a definitional identity of the construction rather than an independent empirical discovery; the paired bootstrap CI only quantifies the magnitude, not the direction. The paper's own caveats ('We do not compute the per-question selection oracle...', 'The lemma is trivial; its consequence is not') show awareness but do not remove this definitional forcing. Separately, the claimed BioASQ scorer behavior (counting any synonym in the gold reference set as a hit, §8.1/H) is an unverified external dependency; if the scorer is stricter, the within-item synonym-union mechanism cannot produce the claimed recall lift, but this is a correctness risk, not circularity. The remaining empirical results — re-retrieval cost–quality, leaderboard standings, always-pick floor failures — are self-contained and independent of the definitional step, so the overall circularity is partial, not total. Score 5.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No code or data; thresholds and caps are fit to validation; several domain assumptions about scoring and leaderboard are load-bearing.

free parameters (4)
  • BGE gate per-type thresholds = not reported
    Calibrated on Task 12B validation; flag rates 23-25% on val and b1-b3, 40% on b4 (§4).
  • top-N caps per type = 15 factoid; 20 yes/no, list, summary
    Chosen to cap the pool after union; affects what the answering agent sees (§5).
  • resolver item cap = 12 per question
    Limits cross-item precision loss (§6.4).
  • non-inferiority margin = -0.005
    Chosen for CI verdicts in the re-retrieval ablation (§5.1).
axioms (4)
  • domain assumption BioASQ list/factoid scoring treats any normalized synonym in the gold reference set as a hit
    Central to the claim that within-item synonym union is precision-preserving (§8.1, §H).
  • domain assumption The reported leaderboard scores and gold-input pool are accurately transcribed
    All Task 14B standings are self-reported preliminary auto-metric results (§7.3).
  • domain assumption The curated 58k-document PubMed slice is a valid proxy for the BioASQ retrieval corpus
    R@200 = 99.3% is measured on this slice, not the official corpus (§3.1, Appendix A).
  • standard math Paired-bootstrap CIs with 2000 iterations are valid for the reported differences
    Used for all significance claims (§5.1).

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Cost-Pragmatic Quality Gating and Selection-Fusion Multi-Model Combiners for BioASQ Phases A+ and B." pith.science (2026). https://pith.science/paper/RNIM5U42

@misc{pith2026260713551,
  author       = {Pith},
  title        = {Pith review of: Cost-Pragmatic Quality Gating and Selection-Fusion Multi-Model Combiners for BioASQ Phases A+ and B},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNIM5U42}},
  note         = {Machine review of arXiv:2607.13551}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We describe our BioASQ Task 14B 2026 system. The work centers on two design decisions: how aggressively to re-retrieve when first-stage retrieval is weak, and how to combine multiple language-model answers. Retrieval unions two parallel pipelines - a hybrid first stage (dense BGE + BM25 + RRF, reaching R@200 = 99.3% on the BioASQ-13b historical archive) and an agent-driven pipeline that decomposes the question over PubMed, Europe PMC, and iCite - with a BGE cross-encoder quality gate flagging weakly-supported questions for selective re-retrieval. On Task 12B 2024 validation, a cost-pragmatic re-retrieval policy beats a skill-strict baseline significantly on list F1 and list precision, at 12% lower re-retrieval cost. Holding prompt and model fixed across val and test 13B (different question sets), list F1 rises by +0.132 absolute on the BioASQ-released gold-input pool, consistent with substantial retrieval-side headroom. For Phase B answering we decompose multi-model ensemble lift into a selection component bounded by the per-question oracle and a fusion component that aggregators can exceed. The decomposition predicts before any experiment that LLM-as-judge wins on selection-dominated metrics (yes/no, multi-reference ROUGE) but is structurally insufficient on the recall component of fusion-friendly metrics (factoid rank-1, list recall). On Task 13B 2025 our synonym-union resolver wins list recall on every head, while GPT-5.5 solo retains the list-F1 lead because the resolver's wider item set costs precision. On the Task 14B 2026 preliminary leaderboard our team places first on the combined-exact aggregate on three of the eight (phase x batch) leaderboards, wins four individual question-type cells, and takes #1 on Phase B b3 ideal.

Figures

Figures reproduced from arXiv: 2607.13551 by Dima Galat, Marian-Andrei Rizoiu.

Figure 1
Figure 1. Figure 1: System pipeline. Two parallel retrievers (green) feed a unioned snippet pool that the BGE quality gate scores. Flagged questions trigger selective re-retrieval whose output rejoins the union; the rest flows down through BGE rerank, the Phase B combiner (orange), and submission. The hybrid first stage and the BGE gate share the same encoder family but operate at different layers: the first stage runs over t… view at source ↗
Figure 2
Figure 2. Figure 2: Decomposition of ensemble lift over a candidate set ℋ. The aggregator Φ exceeds the strongest single head by combining candidate outputs; the residual gap to the selection oracle is the maximum lift any candidate-restricted selector could add. On fusion-friendly metrics Φ can move to the right of 𝑆(Ω* ) (positive Δfus), opening a region no candidate-restricted selector can reach. On selection-dominated met… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 5 linked inside Pith

  1. [1]

    An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition,

    G. Tsatsaronis, G. Balikas, P. Malakasiotis, I. Partalas, M. Zschunke, M. R. Alvers, D. Weissenborn, A. Krithara, S. Petridis, D. Polychronopoulos,et al., “An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition, ”BMC Bioinformatics, 16:138, 2015

    2015

  2. [2]

    BioASQ-QA: A manually curated corpus for biomedical question answering,

    A. Krithara, A. Nentidis, K. Bougiatiotis, G. Paliouras, “BioASQ-QA: A manually curated corpus for biomedical question answering, ”Scientific Data, 10:170, 2023

    2023

  3. [3]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks, ” inAdvances in Neural Information Processing Systems, 2020

    2020

  4. [4]

    Leveraging passage retrieval with generative models for open domain question answering,

    G. Izacard, E. Grave, “Leveraging passage retrieval with generative models for open domain question answering, ” inProceedings of EACL, 2021, pp. 874–880

    2021

  5. [5]

    Overview of BioASQ 2024: The Twelfth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answer- ing,

    A. Nentidis, G. Katsimpras, A. Krithara, S. Lima-López, E. Farré-Maduell, M. Krallinger, N. Loukachevitch, V. Davydova, E. Tutubalina, G. Paliouras, “Overview of BioASQ 2024: The Twelfth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answer- ing, ” inCLEF 2024 Working Notes, Grenoble, France, September 9–12, 2024, CEUR Workshop ...

    2024

  6. [6]

    Overview of BioASQ Tasks 13b and Syn- ergy13 in CLEF 2025,

    A. Nentidis, G. Katsimpras, A. Krithara, G. Paliouras, “Overview of BioASQ Tasks 13b and Syn- ergy13 in CLEF 2025, ” inCLEF 2025 Working Notes, Madrid, Spain, September 9–12, 2025, CEUR Workshop Proceedings, Vol. 4038, pp. 1–18. https://ceur-ws.org/Vol-4038/paper_1.pdf

    2025

  7. [7]

    Overview of BioASQ 2026: The fourteenth BioASQ challenge on large-scale biomedical semantic indexing and question answering,

    A. Nentidis, G. Katsimpras, A. Krithara, M. Krallinger, M. Rodríguez-Ortega, E. Rodríguez-López, N. Loukachevitch, I. Rozhkov, E. Tutubalina, D. Dimitriadis, V. Patsiou, G. Tsoumakas, G. Gian- nakoulas, A. Bekiaridou, A. Samaras, G. M. Di Nunzio, N. Ferro, S. Marchesin, M. Martinelli, G. Sil- vello, G. Paliouras, “Overview of BioASQ 2026: The fourteenth B...

    2026

  8. [8]

    Overview of BioASQ Tasks 14b and Syn- ergy14 in CLEF 2026,

    A. Nentidis, G. Katsimpras, A. Krithara, G. Paliouras, “Overview of BioASQ Tasks 14b and Syn- ergy14 in CLEF 2026, ” inCLEF 2026 Working Notes, 2026

    2026

  9. [9]

    PubMedQA: A dataset for biomedical research question answering,

    Q. Jin, B. Dhingra, Z. Liu, W. Cohen, X. Lu, “PubMedQA: A dataset for biomedical research question answering, ” inProceedings of EMNLP-IJCNLP, 2019, pp. 2567–2577

    2019

  10. [10]

    What disease does this patient have? A large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, P. Szolovits, “What disease does this patient have? A large-scale open domain question answering dataset from medical exams, ”Applied Sciences, 11(14):6421, 2021

    2021

  11. [11]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries, ” inText Summarization Branches Out, ACL Workshop, 2004, pp. 74–81

    2004

  12. [12]

    BioBERT: A pre-trained biomedical language representation model for biomedical text mining,

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang, “BioBERT: A pre-trained biomedical language representation model for biomedical text mining, ”Bioinformatics, 36(4):1234–1240, 2020

    2020

  13. [13]

    The probabilistic relevance framework: BM25 and beyond,

    S. Robertson, H. Zaragoza, “The probabilistic relevance framework: BM25 and beyond, ”Foundations and Trends in Information Retrieval, 3(4):333–389, 2009

    2009

  14. [14]

    Dense passage retrieval for open-domain question answering,

    V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, W. Yih, “Dense passage retrieval for open-domain question answering, ” inProceedings of EMNLP, 2020, pp. 6769–6781

    2020

  15. [15]

    Unsupervised dense information retrieval with contrastive learning,

    G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, E. Grave, “Unsupervised dense information retrieval with contrastive learning, ”Transactions on Machine Learning Research,

  16. [16]

    ColBERT: Efficient and effective passage search via contextualized late interaction over BERT,

    O. Khattab, M. Zaharia, “ColBERT: Efficient and effective passage search via contextualized late interaction over BERT, ” inProceedings of SIGIR, 2020, pp. 39–48

    2020

  17. [17]

    MedCPT: Contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval,

    Q. Jin, W. Kim, Q. Chen, D. C. Comeau, L. Yeganova, W. J. Wilbur, Z. Lu, “MedCPT: Contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval, ”Bioinformatics, 39(11):btad651, 2023

    2023

  18. [18]

    Document ranking with a pretrained sequence-to- sequence model,

    R. Nogueira, Z. Jiang, R. Pradeep, J. Lin, “Document ranking with a pretrained sequence-to- sequence model, ” inFindings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 708–718

    2020

  19. [19]

    BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,

    J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, Z. Liu, “BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, ” arXiv:2402.03216, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Reciprocal rank fusion outperforms Condorcet and individual rank learning methods,

    G. V. Cormack, C. L. A. Clarke, S. Buettcher, “Reciprocal rank fusion outperforms Condorcet and individual rank learning methods, ” inProceedings of SIGIR, 2009, pp. 758–759

    2009

  21. [21]

    BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models,

    N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, I. Gurevych, “BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models, ” inNeurIPS Datasets and Benchmarks Track, 2021

    2021

  22. [22]

    C-Pack: Packed resources for general Chinese embeddings,

    S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, J.-Y. Nie, “C-Pack: Packed resources for general Chinese embeddings, ” inProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24), Washington, DC, USA, July 2024. https://doi.org/10.1145/3626772.3657878

    arXiv 2024

  23. [23]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, “ReAct: Synergizing reasoning and acting in language models, ” inProceedings of ICLR, 2023

    2023

  24. [24]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, T. Scialom, “Toolformer: Language models can teach themselves to use tools, ” inAdvances in Neural Information Processing Systems, 2023

    2023

  25. [25]

    An overview of MetaMap: Historical perspective and recent advances,

    A. R. Aronson, F.-M. Lang, “An overview of MetaMap: Historical perspective and recent advances, ” Journal of the American Medical Informatics Association, 17(3):229–236, 2010

    2010

  26. [26]

    The NIH Open Citation Collection: A public access, broad coverage resource,

    B. I. Hutchins, K. L. Baker, M. T. Davis, M. A. Diwersy, E. Haque, R. M. Harriman, T. A. Hoppe, S. A. Leicht, P. Meyer, G. M. Santangelo, “The NIH Open Citation Collection: A public access, broad coverage resource, ”PLoS Biology, 17(10):e3000385, 2019

    2019

  27. [27]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray,et al., “Training language models to follow instructions with human feedback, ” inAdvances in Neural Information Processing Systems, 2022

    2022

  28. [28]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, “Attention is all you need, ” inAdvances in Neural Information Processing Systems, 2017

    2017

  29. [29]

    BERT: Pre-training of deep bidirectional transform- ers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, “BERT: Pre-training of deep bidirectional transform- ers for language understanding, ” inProceedings of NAACL-HLT, 2019, pp. 4171–4186

    2019

  30. [30]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell,et al., “Language models are few-shot learners, ” inAdvances in Neural Infor- mation Processing Systems, 2020

    2020

  31. [31]

    SciBERT: A pretrained language model for scientific text,

    I. Beltagy, K. Lo, A. Cohan, “SciBERT: A pretrained language model for scientific text, ” inProceedings of EMNLP-IJCNLP, 2019, pp. 3615–3620

    2019

  32. [32]

    Domain- specific language model pretraining for biomedical natural language processing,

    Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, H. Poon, “Domain- specific language model pretraining for biomedical natural language processing, ”ACM Transactions on Computing for Healthcare, 3(1):1–23, 2021

    2021

  33. [33]

    BioGPT: Generative pre-trained transformer for biomedical text generation and mining,

    R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, T.-Y. Liu, “BioGPT: Generative pre-trained transformer for biomedical text generation and mining, ”Briefings in Bioinformatics, 23(6):bbac409, 2022

    2022

  34. [34]

    Large language models encode clinical knowledge,

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl,et al., “Large language models encode clinical knowledge, ”Nature, 620(7972):172–180, 2023

    2023

  35. [35]

    Chain-of- thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, “Chain-of- thought prompting elicits reasoning in large language models, ” inAdvances in Neural Information Processing Systems, 2022

    2022

  36. [36]

    On combining classifiers,

    J. Kittler, M. Hatef, R. P. W. Duin, J. Matas, “On combining classifiers, ”IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(3):226–239, 1998

    1998

  37. [37]

    Self-consistency improves chain-of-thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, D. Zhou, “Self-consistency improves chain-of-thought reasoning in language models, ” inProceedings of ICLR, 2023

    2023

  38. [38]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing,et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, ” inAdvances in Neural Information Processing Systems, 2023

    2023

  39. [39]

    LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion,

    D. Jiang, X. Ren, B. Y. Lin, “LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion, ” inProceedings of ACL, 2023, pp. 14165–14178

    2023

  40. [40]

    Improving factuality and reasoning in language models through multiagent debate,

    Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, I. Mordatch, “Improving factuality and reasoning in language models through multiagent debate, ” arXiv:2305.14325, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    LM vs LM: Detecting factual errors via cross examination,

    R. Cohen, M. Hamri, M. Geva, A. Globerson, “LM vs LM: Detecting factual errors via cross examination, ” inProceedings of EMNLP, Singapore, 2023, pp. 12621–12640

    2023

  42. [42]

    Claude Opus 4.6 System Card,

    Anthropic, “Claude Opus 4.6 System Card, ” February 2026. https://www.anthropic.com/ claude-opus-4-6-system-card

    2026

  43. [43]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,

    Gemini Team, Google, “Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities, ” arXiv:2507.06261, 2025. https://arxiv. org/abs/2507.06261

    Pith/arXiv arXiv 2025

  44. [44]

    GPT-5.5 System Card,

    OpenAI, “GPT-5.5 System Card, ” April 2026. https://openai.com/index/gpt-5-5-system-card/

    2026

  45. [45]

    Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs,

    Yu. A. Malkov, D. A. Yashunin, “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs, ”IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836, 2020

    2020

  46. [46]

    Europe PMC in 2017,

    M. Levchenko, Y. Yehudi, M. Khalifa, B. Wilson, A. Hamelers, G. Saska, K. Sigourdas, S. Krasnitsky, D. Marcetić,et al., “Europe PMC in 2017, ”Nucleic Acids Research, 46(D1):D1254–D1260, 2018

    2017

  47. [47]

    Statistical significance tests for machine translation evaluation,

    P. Koehn, “Statistical significance tests for machine translation evaluation, ” inProceedings of EMNLP, 2004, pp. 388–395

    2004

  48. [48]

    Self-Refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, P. Clark, “Self-Refine: Iterative refinement with self-feedback, ” inAdvances in Neural Information Processing Systems, 2023

    2023

  49. [49]

    Chain-of-verification reduces hallucination in large language models,

    S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, J. Weston, “Chain-of-verification reduces hallucination in large language models, ” arXiv:2309.11495, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Overview of the TREC 2003 Question Answering Track,

    E. M. Voorhees, “Overview of the TREC 2003 Question Answering Track, ” inProceedings of the Twelfth Text Retrieval Conference (TREC), NIST Special Publication 500-255, 2003. Table 8 Embedding-model comparison on the BioASQ-13b retrieval slice. Higher R@10 is better. Model R@10↑Note BAAI/bge-base-en-v1.5 0.687Production choice ncbi/MedCPT (asymmetric Q+A) ...

    arXiv 2003

  51. [2022]

    https://arxiv.org/abs/2112.09118

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.