Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content

Eric de la Clergerie; Nathan Godey; Rian Touchent

REVIEW 4 major objections 5 minor 18 references

Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Paragraph-level curation lets biomedical models match baseline with one-third of the training tokens.

desk verdict A genuinely useful biomedical dataset and a careful curation study whose headline numbers are slightly over-drawn; deserves review, but needs per-class classifier validation and multiple-seed runs before the claims are secure. read the letter →

arxiv 2506.20331 v1 pith:VIDALH2Z submitted 2025-06-25 cs.CL cs.LG

Rian Touchent , Nathan Godey , Eric de la Clergerie This is my paper · ORCID

classification cs.CLcs.LG

keywords Biomed-Enrichedparagraph-levelannotationLLMdistillationcontinualpretrainingclinicalcaseextractioneducationalqualityfilteringPMCOpenAccessdatacuration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the bottleneck for biomedical language models is not the size of the training corpus but how its paragraphs are selected and mixed. It introduces Biomed-Enriched, a paragraph-level annotation layer over the PubMed Central Open Access corpus that labels every paragraph by document type, domain, and educational quality. The paper claims that filtering for educational quality, upsampling clinical-case and French content, and prefixing metadata (the BE-All recipe) reaches the same or better benchmark accuracy as the unfiltered corpus while using roughly one-third of the training tokens. It also claims that clinical upsampling alone raises MMLU Professional Medicine by about 4 points. If true, the finding matters because clinical text is normally locked behind privacy restrictions, and this pipeline turns openly licensed scientific articles into a large, targeted training signal.

What carries the argument

The load-bearing mechanism is the paragraph-level annotation layer: a large language model's scores on 400K paragraphs are distilled into a small XLM-RoBERTa-base multi-task classifier, which propagates type, domain, and educational quality labels across 133M paragraphs. These labels make it possible to extract clinical-case paragraphs (2M total, 450K high quality with commercial-use licenses), filter by educational score, and upsample specific domains. The claimed effect is that this metadata redirects continual pretraining toward content the downstream tasks actually reward.

What would settle it

Re-annotate a stratified sample of paragraphs with human reviewers, compare the distilled model's predictions on that sample, then retrain the BE variants on human-corrected labels in a small controlled run; if the benchmark gaps between BE-All and BE-Base disappear or shrink sharply, the curation effects are at least partly artifact. A cheaper test: shuffle the predicted labels within each variant and check whether the reported gains vanish.

Watch

Extended reading notes

Core claim

The central claim is that fine-grained, annotation-driven curation of biomedical text is a more efficient path to domain competence than training on the full corpus. Using a two-stage annotation process, Llama-3.1-70B labels 400K paragraphs across type, domain, and educational quality, and a distilled XLM-RoBERTa-base classifier propagates those labels to all 133M paragraphs of PMC-OA. The authors then construct variants and continual-pretrain OLMo2-7B-stage1 on each for 33.6B tokens. BE-All, which combines quality filtering, clinical and French upsampling, and metadata prefixing, reaches 61.08% average accuracy versus 60.41% for BE-Base and reaches robust performance with a third of the tokens; BE-Clinical reaches 63.97% on MMLU Professional Medicine versus 59.93% for BE-Base. The claim is that selection and mixing strategy, not raw token count, drives targeted gains.

Load-bearing premise

The distilled XLM-RoBERTa classifier's labels are accurate enough across all content slices; slice-level validation and human agreement are not reported, so if labeling errors correlate with clinical or educational content, the benchmark differences could be an artifact of classifier noise.

Editorial extensions

If this is right

Continual pretraining on a curated third of the tokens can reach the same biomedical benchmark performance as the full corpus, so compute budgets for domain adaptation can be cut by roughly two-thirds.
Openly licensed PubMed articles can serve as a large-scale substitute for privacy-restricted clinical records, because clinical-case paragraphs can be located and extracted at paragraph level.
Language-specific adaptation is achievable by upsampling annotated non-English paragraphs, as shown by BE-French's gain on FrenchMedMCQA.
Different enrichment dimensions serve different tasks: educational filtering helps knowledge-based QA, clinical upsampling helps clinical reasoning, and metadata prefixing helps structured comprehension.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If paragraph-level annotation generalizes, the same pipeline could be applied to legal documents, financial filings, or technical manuals where high-value content is buried inside heterogeneous articles.
The 2M clinical-case paragraph subset is itself a reusable resource; a natural testable extension is instruction-tuning or preference-tuning on it, not just continued pretraining.
The design leaves open whether the gains come from the filter, the upsampling, or the prefix; an ablation that isolates each component's token-share would make the causal story sharper.
The one-third-token result suggests data-efficiency metrics should be reported alongside final accuracy in domain-adaptation papers, since raw accuracy alone understates curation gains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth reading, mostly for the resource rather than the method. Two-stage LLM-to-small-model annotation is now standard practice after FineWeb-Edu and WebOrganizer, but applying it at paragraph level to the whole PMC-OA corpus, and in particular extracting 2M clinical-case paragraphs with 450K high-quality ones, is a real contribution. Clinical text of that scale and openness is genuinely scarce. The pretraining comparisons across six variants are also a sensible way to isolate the effect of each curation choice, and the authors are honest about the small-model scope and the trade-off against general biomedical knowledge.

The soft spots are real but mostly addressable. There is a single training run per variant and no standard errors, so several of the reported deltas (MedQA +1.17, MedMCQA +0.6, PubMedQA +0.6) sit inside plausible benchmark noise. The abstract's '~5%' overstates the +4.04 point ProfMed gain, which is the one result that looks robust. PubMedQA contamination from overlapping PMC abstracts is not discussed. The token-efficiency claim rests on a visual reading of Figure 2, not a number. And the dataset and code are not yet accessible, which makes the central resource hard to evaluate.

The stress-test concern about the distilled classifier is legitimate and lands. The paper reports only aggregate F1/MSE (0.805 domain, 0.854 type, 0.245 MSE) and never breaks down precision/recall on the rare clinical classes that define BE-Clinical and the clinical-case subset. Since clinical paragraphs are about 1.6% of the corpus, an 0.80 overall F1 can coexist with poor precision on that class. If the labels are wrong on clinical text, the ProfMed gain and the dataset counts are both in question. That needs a confusion matrix or per-class numbers, plus some human agreement, before the central claims are secure.

None of this is fatal to the direction. The method is sound, the experiments are controlled, and the limitations section is honest. The paper is for people who want a large open clinical-text corpus or who are thinking about fine-grained biomedical data curation; they will get value even from the current version.

I would send this to referees. It deserves a careful review, and the revisions it needs are concrete: per-class validation of the annotator, multiple seeds, explicit contamination check for PubMedQA, and release of the dataset and code.

Referee Report

4 major / 5 minor

Summary. The paper introduces Biomed-Enriched, a paragraph-level annotated biomedical dataset derived from the PMC Open Access subset. A two-stage pipeline uses Llama-3.1-70B to annotate 400K paragraphs for document type, domain, language, and educational quality, then distills these annotations into an XLM-RoBERTa-base model that labels the full ~133M-paragraph corpus. From this, the authors construct several training variants (BE-Base, BE-Educational, BE-Clinical, BE-ClinicalCase, BE-Prefix, BE-French, BE-All) and continually pretrain OLMo2-7B on each variant for 33.6B tokens with fixed hyperparameters. They report that clinical upsampling improves MMLU Professional Medicine, educational filtering improves several medical QA benchmarks, metadata prefixing helps PubMedQA, French upsampling helps FrenchMedMCQA, and BE-All reaches comparable performance with roughly one-third of the training tokens. The paper also highlights the extracted resource of 2M clinical-case paragraphs, including 450K high-quality ones from commercially licensed articles.

Significance. If the results hold, the dataset is a useful public resource for biomedical NLP, and the two-stage annotation pipeline is a scalable recipe for paragraph-level corpus enrichment. The experimental design has notable strengths: all variants are trained with identical hyperparameters, the annotation prompt is fully documented, the evaluation is limited to isolating curation effects rather than chasing SOTA, and the dataset is released openly. The central claims, however, rest on classifier validation that is not demonstrated for the rare clinical classes, on single-run training without error bars, and on a token-efficiency comparison that is not quantitatively defined. These gaps currently prevent the results from being taken as robust evidence for the stated conclusions, though they are addressable within the manuscript's scope.

major comments (4)

[Section 3.2] The distilled classifier's validation is reported only as aggregate F1 (0.805 domain, 0.854 type) and MSE (0.245 educational quality). Because BE-Clinical and BE-ClinicalCase select articles based on predicted clinical-domain and clinical-case labels, and these classes constitute only ~1.6% of the 133M paragraphs, aggregate F1 does not establish that the upsampled articles are genuinely clinical. Please provide per-class precision/recall, confusion matrices, and ideally human agreement or a slice-level evaluation on clinical paragraphs; without this, the +4.04 point ProfMed gain attributed to clinical upsampling and the reported 2M/450K clinical-case paragraph counts may reflect classifier noise correlated with article-level properties rather than clinical content.
[Section 4.1 and Table 3] Each variant is trained once with fixed hyperparameters, so no standard errors or multiple-seed results are reported. Several headline deltas (MedQA +1.17, MedMCQA +0.6, PubMedQA +0.6, ProfMed +4.04) could fall within plausible noise for a single 7B continual-pretraining run, especially with zero-shot evaluation. The abstract's '~5%' also overstates the ProfMed gain in Table 3 (63.97 vs 59.93 is +4.04 points). Please provide multiple seeds or at least a variance estimate, and rephrase the abstract to match the table.
[Section 5 and Table 3] PubMedQA is derived from PubMed abstracts, and the training corpus is the PMC Open Access subset, which contains overlapping article abstracts. The paper does not report any decontamination or overlap analysis for PubMedQA. Consequently the PubMedQA improvements (+0.6 for BE-Educational, +1.4 for BE-Prefix) could be inflated by train/test overlap. Please quantify overlap between the training paragraphs and PubMedQA instances and re-evaluate on a non-overlapping subset, or provide a reasoned argument for why contamination is not a concern.
[Section 5 and Figure 2] The claim that 'BE-All achieves target performance with approximately one-third of the training tokens required by BE-Base' is not defined quantitatively. It is unclear what 'target performance' means (e.g., BE-Base's final average, a per-task threshold, or a specific benchmark) and whether the comparison is made on the same metric. Please state the exact token count at which BE-All reaches the chosen target and provide confidence intervals or a sensitivity check, since Figure 2 shows curves without error bars.

minor comments (5)

[Figure 2] The y-axis label contains a rendering artifact ('glyph1197ormalized accuracy'); please fix the font or label generation.
[Table 3] The SOTA reference row lists 'Llama-3-8B' while Section 4.2 mentions 'Llama-3.1-8B'; please make the model naming consistent.
[Section 5] In the 'Non-English enrichment' paragraph, 'Figure 1' should be 'Figure 3' for the FrenchMedMCQA results, since Figure 1 shows the score distribution.
[Section 2] The reference to 'PMCLlama' is written inconsistently (PMCLlama, PMCLLaMA, PMC-Llama); please standardize the name and citation.
[General] Please add a dedicated Data Availability and Code Availability statement with a versioned link or commit hash; currently the repository is only mentioned in the author affiliation line of the abstract.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: dataset variants are fixed before evaluation and all headline results are measured against external benchmarks.

full rationale

I examined the paper's derivation chain: Llama-3.1-70B annotations on 400K paragraphs, distillation into XLM-RoBERTa labels over 133M paragraphs, construction of fixed dataset variants, continual pretraining, and evaluation on external benchmarks (MMLU medical subsets, MedQA, MedMCQA, PubMedQA, FrenchMedMCQA). No equation-level reduction or fitted-parameter-as-prediction occurs: the variants (BE-Base, BE-Educational, BE-Clinical, BE-ClinicalCase, BE-Prefix, BE-French, BE-All) are defined before evaluation, and the reported gains (e.g., +4.04 points on MMLU Professional Medicine for BE-Clinical, +1.17 points on MedMCQA for BE-Educational) are measured outcomes rather than quantities derived from the annotation model. The only self-referential aspect is that the same distilled classifier produces both the metadata used to construct the dataset and the reported corpus statistics (2M clinical-case paragraphs, 450K high-quality ones); this is a validity and calibration concern, not circularity, because the benchmark conclusions do not assume those counts are ground truth. External citations (FineWeb-Edu, WebOrganizer) are prior independent work, not author self-citations, and no uniqueness theorems or ansatz-smuggling citations are invoked. Potential weaknesses, such as the absence of per-class validation of the distilled classifier on rare clinical classes, are correctness risks that would not make the paper's central claims circular even if they undermined them.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims depend on the distilled annotation classifier being accurate at scale, on the choice of filtering and upsampling thresholds, and on the benchmarks being uncontaminated. No new physical or linguistic entities are introduced. The free parameters are design choices rather than fitted values, but they shape all reported results.

free parameters (5)

Clinical/French upsampling factor = 10
Chosen by hand for BE-Clinical, BE-ClinicalCase, BE-French, and BE-All; affects the magnitude of reported gains.
Educational quality threshold = >= 3
Adopted from FineWeb-Edu convention; determines which paragraphs survive BE-Educational and BE-All.
Minimum paragraph length = 64 tokens
Filtering threshold in Section 3.1, chosen by hand before annotation.
Training token budget = 33.6B
Fixed across all variants; the 'one-third tokens' claim compares curves against this budget, but no exact token count for parity is reported.
Context window = 8,192
Chosen to fit complete articles; affects how paragraph-level annotations interact with pretraining.

assumptions (4)

domain assumption Llama-3.1-70B-Instruct annotations on 400K paragraphs are accurate and unbiased ground truth for educational quality, type, and domain.
Section 3.2 uses these as training labels; no human validation or inter-annotator agreement is reported.
domain assumption The distilled XLM-RoBERTa-base classifier preserves annotation quality across 133M paragraphs.
Section 3.2; F1/MSE given but no slice-level calibration or error analysis.
domain assumption PMC Open Access Subset is a representative source of biomedical text and contains sufficient clinical case content to support pretraining.
Section 3.1; no comparison to other biomedical text sources.
domain assumption Benchmark evaluations are contamination-free even though PubMedQA is built from PubMed abstracts and the pretraining data is PMC-OA.
Section 4.2 and Section 5; the evaluation discussion does not address overlap between PMC full-text articles or abstracts and PubMedQA questions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content." pith.science (2026). https://pith.science/paper/VIDALH2Z

@misc{pith2026250620331,
  author       = {Pith},
  title        = {Pith review of: Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VIDALH2Z}},
  note         = {Machine review of arXiv:2506.20331}
}

read the original abstract

We introduce Biomed-Enriched, a biomedical text dataset constructed from PubMed via a two-stage annotation process. In the first stage, a large language model annotates 400K paragraphs from PubMed scientific articles, assigning scores for their type (review, study, clinical case, other), domain (clinical, biomedical, other), and educational quality. The educational quality score (rated 1 to 5) estimates how useful a paragraph is for college-level learning. These annotations are then used to fine-tune a small language model, which propagates the labels across the full PMC-OA corpus. The resulting metadata allows us to extract refined subsets, including 2M clinical case paragraphs with over 450K high-quality ones from articles with commercial-use licenses, and to construct several variants via quality filtering and domain upsampling. Clinical text is typically difficult to access due to privacy constraints, as hospital records cannot be publicly shared. Hence, our dataset provides an alternative large-scale, openly available collection of clinical cases from PubMed, making it a valuable resource for biomedical and clinical NLP. Preliminary continual-pretraining experiments with OLMo2 suggest these curated subsets enable targeted improvements, with clinical upsampling boosting performance by ~5% on MMLU ProfMed and educational quality filtering improving MedQA and MedMCQA by ~1%. Combinations of these techniques led to faster convergence, reaching same performance with a third of training tokens, indicating potential for more efficient and effective biomedical pretraining strategies.

Figures

Figures reproduced from arXiv: 2506.20331 by the authors.

**Figure 2.** Performance comparison across dataset variants showing training progression. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Performance on FrenchMedMCQA showing BE-French outperforming other variants, demonstrating effective languagespecific improvement. Implications for filtering. These distributions justify the score threshold (score ≥ 3) used in BE-Educational and BE-All, enabling targeted retention of higher-quality educational content while discarding noisy or low-value segments. The correlation between domain or type and educati… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 6 canonical work pages

[1]

https://www.ncbi.nlm.nih.gov/pmc/tools/textmining/

PMC open access subset. https://www.ncbi.nlm.nih.gov/pmc/tools/textmining/. Accessed 2025-03-29

work page 2025
[2]

FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain

Yanis Labrak, Adrien Bazoge, Richard Dufour, Béatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, and Mickaël Rouvier. FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 41--46, Abu Dhabi, United Arab Emi...

work page doi:10.18653/v1/2022.louhi- 2022
[3]

Meditron-70b: Scaling medical pretraining for large language models

Zeming Chen, Alejandro Hern \'a ndez-Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas K \"o pf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. Medit...

arXiv 2023
[4]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

arXiv 2018
[5]

Unsupervised cross-lingual representation learning at scale

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 8440--8451, 2020. URL https:/...

work page 2020
[6]

Measuring massive multitask language understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2021

work page 2021
[7]

B io M istral: A collection of open-source pretrained large language models for medical domains

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. B io M istral: A collection of open-source pretrained large language models for medical domains. In Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 5848--5864, Bangkok, Thailand, 2024. Association for Computational Linguisti...

doi:10.18653/v1/2024.findings-acl.348 2024
[8]

Datacomp-lm: In search of the next generation of training sets for language models

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024. URL https://arxiv.org/abs/2406.11794

arXiv 2024

Show all 18 references

[9]

Can a suit of armor conduct electricity? a new dataset for open book question answering

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 2381--2391, Brussels...

2018
[10]

Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...

2025
[11]

The fineweb datasets: Decanting the web for the finest text data at scale

Guilherme Penedo, Hynek Kydl \' c ek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024

2024 arXiv
[12]

The llama 3 herd of models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. The llama 3 herd of models. arXiv preprint arXiv:2407.2178...

2024 arXiv
[13]

Organize the web: Constructing domains enhances pre-training data curation

Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. arXiv preprint arXiv:2502.10341, 2025

2025 arXiv
[14]

PMC-LLaMA : toward building open-source language models for medicine

Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. PMC-LLaMA : toward building open-source language models for medicine. Journal of the American Medical Informatics Association, 31 0 (9): 0 1833--1843, 2024. doi:10.1093/jamia/ocae045

2024 doi
[15]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[16]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[17]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[18]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

https://www.ncbi.nlm.nih.gov/pmc/tools/textmining/

PMC open access subset. https://www.ncbi.nlm.nih.gov/pmc/tools/textmining/. Accessed 2025-03-29

work page 2025

[2] [2]

FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain

Yanis Labrak, Adrien Bazoge, Richard Dufour, Béatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, and Mickaël Rouvier. FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 41--46, Abu Dhabi, United Arab Emi...

work page doi:10.18653/v1/2022.louhi- 2022

[3] [3]

Meditron-70b: Scaling medical pretraining for large language models

Zeming Chen, Alejandro Hern \'a ndez-Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas K \"o pf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. Medit...

arXiv 2023

[4] [4]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

arXiv 2018

[5] [5]

Unsupervised cross-lingual representation learning at scale

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 8440--8451, 2020. URL https:/...

work page 2020

[6] [6]

Measuring massive multitask language understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2021

work page 2021

[7] [7]

B io M istral: A collection of open-source pretrained large language models for medical domains

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. B io M istral: A collection of open-source pretrained large language models for medical domains. In Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 5848--5864, Bangkok, Thailand, 2024. Association for Computational Linguisti...

doi:10.18653/v1/2024.findings-acl.348 2024

[8] [8]

Datacomp-lm: In search of the next generation of training sets for language models

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024. URL https://arxiv.org/abs/2406.11794

arXiv 2024