pith. sign in

arxiv: 2604.03506 · v1 · submitted 2026-04-03 · 💻 cs.AI

BioAlchemy: Distilling Biological Literature into Reasoning-Ready Reinforcement Learning Training Data

Brian Hsu , Ozan G\"okdemir , Carlo Siebenschuh , Bruce Parrello , Neil Getty , Thomas S. Brettin , Rick L. Stevens , Ian T. Foster
show 2 more authors
Nicholas Chia Arvind Ramanathan
This is my paper

Pith reviewed 2026-05-13 19:19 UTC · model grok-4.3

classification 💻 cs.AI
keywords biology reasoningreinforcement learningdataset curationscientific literatureverifiable problemstopic alignmentBioAlchemyAI training data
0
0 comments X

The pith

BioAlchemy extracts over 345,000 verifiable biology reasoning problems from research literature and uses them with reinforcement learning to lift model performance by 9.12 percent on benchmarks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current reasoning datasets contain biology questions that fail to match the topic distribution of modern biological research, which limits gains from training. The paper introduces the BioAlchemy pipeline to pull diverse, verifiable question-answer pairs directly from biology research papers. It curates BioAlchemy-345K, a dataset of more than 345,000 such problems aligned to current topic distributions. Reinforcement learning on this data produces BioAlchemist-8B, which improves 9.12 percent over its base model on biology benchmarks. The results show that targeted distillation of scientific text can strengthen domain-specific reasoning.

Core claim

BioAlchemy is a pipeline that sources a diverse set of verifiable question-and-answer pairs from a corpus of biology research text. Curating BioAlchemy-345K with over 345,000 scientific reasoning problems and aligning the dataset to the topic distribution of modern biology enables reinforcement learning that improves reasoning performance, yielding BioAlchemist-8B with a 9.12 percent gain over its base model on biology benchmarks.

What carries the argument

BioAlchemy, the pipeline that extracts verifiable question-answer pairs from biology literature and aligns them to modern topic distributions for reinforcement learning training.

If this is right

  • The extracted dataset supports measurable gains on existing biology benchmarks through reinforcement learning.
  • Topic alignment in the training data contributes to better generalization on scientific reasoning tasks.
  • The resulting BioAlchemist-8B model demonstrates stronger performance than its base reasoning model.
  • The open release of the model allows further testing and application in biology research settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same distillation approach could be applied to other scientific fields to build domain-specific reasoning datasets.
  • This method may narrow the performance gap between general reasoning models and those specialized for biology.
  • The curated problems could serve as a testbed for studying how data alignment affects real research workflows beyond benchmarks.
  • Extending the pipeline to larger or more recent literature corpora might produce additional performance lifts.

Load-bearing premise

The pipeline reliably extracts challenging and verifiable problems from biology text, and aligning the dataset to modern topic distributions produces genuine generalization rather than benchmark-specific gains.

What would settle it

Training a model on BioAlchemy-345K and then testing it on a new biology benchmark drawn from recent papers with a shifted topic distribution that shows no improvement or a drop in performance.

Figures

Figures reproduced from arXiv: 2604.03506 by Arvind Ramanathan, Brian Hsu, Bruce Parrello, Carlo Siebenschuh, Ian T. Foster, Neil Getty, Nicholas Chia, Ozan G\"okdemir, Rick L. Stevens, Thomas S. Brettin.

Figure 1
Figure 1. Figure 1: Content distribution differences between [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: BioAlchemy has more MeSH topics per biology question. Compari￾son is based on 50K subsamples. To compute Pˆ S, we first have to train a multi-label classifier to label each reasoning problem from our dataset D with its relevant set of MeSH Biology subcategory labels. To do this, we first collect an additional set of papers and MeSH topic array la￾bels via NCBI API for our training and validation sets. We t… view at source ↗
Figure 3
Figure 3. Figure 3: Prompt template for biology domain classification [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Few-shot examples for MeSH Biology classification (part 1) [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Few-shot examples for MeSH Biology classification (part 2) [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Few-shot examples for MeSH Biology classification (part 3) [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template for QA MeSH classification [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt template for content relevance evaluation [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt template for generating multiple-choice questions [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt template for evaluating multiple-choice question quality [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt template for extracting free-form QA (part 1) [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt template for extracting free-form QA (part 2) [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt template for grading free-form QA [PITH_FULL_IMAGE:figures/full_fig_p027_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompt for LAB-Bench evaluation System You are an expert scientist with graduate-level knowledge in biology, physics, and chemistry. Answer the following multiple-choice question. First state your chosen answer (the letter and the corresponding option text), then briefly explain your reasoning. User {question} A. {option_a} B. {option_b} C. {option_c} D. {option_d} Think step by step, and return your fina… view at source ↗
Figure 15
Figure 15. Figure 15: Prompt for evaluating GPQA-Diamond System You are an expert biomedical researcher. You will be given context from a research paper abstract and a yes/no/maybe question about the findings. First state your chosen answer (the letter and the corresponding option text), then briefly explain your reasoning based on the provided context. User Context: {abstract_text} Question: {question} A. yes B. no C. maybe T… view at source ↗
Figure 16
Figure 16. Figure 16: Prompt for evaluating PubMedQA 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Example reasoning question 29 [PITH_FULL_IMAGE:figures/full_fig_p029_17.png] view at source ↗
read the original abstract

Despite the large corpus of biology training text, the impact of reasoning models on biological research generally lags behind math and coding. In this work, we show that biology questions from current large-scale reasoning datasets do not align well with modern research topic distributions in biology, and that this topic imbalance may negatively affect performance. In addition, we find that methods for extracting challenging and verifiable research problems from biology research text are a critical yet underdeveloped ingredient in applying reinforcement learning for better performance on biology research tasks. We introduce BioAlchemy, a pipeline for sourcing a diverse set of verifiable question-and-answer pairs from a scientific corpus of biology research text. We curate BioAlchemy-345K, a training dataset containing over 345K scientific reasoning problems in biology. Then, we demonstrate how aligning our dataset to the topic distribution of modern scientific biology can be used with reinforcement learning to improve reasoning performance. Finally, we present BioAlchemist-8B, which improves over its base reasoning model by 9.12% on biology benchmarks. These results demonstrate the efficacy of our approach for developing stronger scientific reasoning capabilities in biology. The BioAlchemist-8B model is available at: https://huggingface.co/BioAlchemy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces the BioAlchemy pipeline to extract verifiable question-answer pairs from biology research literature, curating the BioAlchemy-345K dataset of over 345K scientific reasoning problems. It demonstrates that aligning this dataset to modern biology topic distributions, followed by reinforcement learning, yields BioAlchemist-8B, which improves 9.12% over its base reasoning model on biology benchmarks. The work addresses the misalignment of existing reasoning datasets with current research topics and the lack of methods for distilling challenging problems from text.

Significance. If the extraction reliably produces verifiable, non-leaking problems and the gains are not artifacts of topic matching or data overlap, the approach could provide a scalable route to stronger biology-specific reasoning models by leveraging the large existing literature corpus. The open release of BioAlchemist-8B and the dataset curation method are positive contributions that could be extended to other scientific domains.

major comments (3)
  1. [§3] §3 (BioAlchemy pipeline): The description of automated extraction and consistency checks for the 345K pairs provides no quantitative verification success rates, human audit results, or error analysis; without these, it is unclear whether the dataset reliably contains challenging, verifiable research problems as claimed, which is load-bearing for the RL training efficacy.
  2. [§4.2] §4.2 (Experimental setup and benchmarks): There is no explicit confirmation or ablation showing that the biology evaluation benchmarks were held out from the BioAlchemy-345K curation and alignment steps; this leaves open the possibility that reported gains partly reflect distribution overlap rather than genuine generalization.
  3. [Results] Results section (BioAlchemist-8B performance): The 9.12% improvement is stated without error bars, number of runs, baseline model details, or ablations isolating the contribution of topic alignment versus simple data volume or RL scale; this weakens the central claim that the pipeline produces superior reasoning data.
minor comments (2)
  1. [Abstract] The abstract and introduction could more clearly distinguish between the contributions of the extraction pipeline versus the topic-alignment step.
  2. [Figures/Tables] Figure captions and table legends should include more detail on how topic distributions were computed and aligned.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment point by point below. Where the manuscript was lacking in quantitative details or explicit controls, we have revised the relevant sections to incorporate the requested information and ablations, strengthening the presentation of the BioAlchemy pipeline and results.

read point-by-point responses

  1. Referee: [§3] §3 (BioAlchemy pipeline): The description of automated extraction and consistency checks for the 345K pairs provides no quantitative verification success rates, human audit results, or error analysis; without these, it is unclear whether the dataset reliably contains challenging, verifiable research problems as claimed, which is load-bearing for the RL training efficacy.

    Authors: We agree that quantitative verification metrics are necessary to substantiate the reliability of the extracted pairs. The original manuscript described the pipeline's automated consistency checks but did not report aggregate success rates or human validation. In the revised manuscript, we have added these to §3: the automated checks passed on 93% of candidate pairs, a human audit of 500 randomly sampled pairs (conducted by two biology PhDs) found 88% to be verifiable and suitably challenging with inter-annotator agreement of 0.82, and we include a categorized error analysis of the remaining cases (primarily context truncation at 7% and minor factual drift at 5%). These additions directly address the load-bearing concern for the downstream RL results. revision: yes

  2. Referee: [§4.2] §4.2 (Experimental setup and benchmarks): There is no explicit confirmation or ablation showing that the biology evaluation benchmarks were held out from the BioAlchemy-345K curation and alignment steps; this leaves open the possibility that reported gains partly reflect distribution overlap rather than genuine generalization.

    Authors: We confirm that the evaluation benchmarks were held out from both the initial curation of BioAlchemy-345K and the subsequent topic-alignment step; no benchmark questions or their source papers were included in the 345K set. We have revised §4.2 to state this explicitly. We also added a targeted ablation: we retrained using only BioAlchemy-345K samples whose topics had zero overlap with the benchmark topic distribution, and the improvement remained at 8.7%, indicating that the gains are not artifacts of distribution overlap. revision: yes

  3. Referee: [Results] Results section (BioAlchemist-8B performance): The 9.12% improvement is stated without error bars, number of runs, baseline model details, or ablations isolating the contribution of topic alignment versus simple data volume or RL scale; this weakens the central claim that the pipeline produces superior reasoning data.

    Authors: We acknowledge that the original Results section would benefit from more rigorous statistical reporting and component ablations. The revised version now reports the 9.12% average improvement with error bars of ±0.9% computed across three independent runs with different random seeds. We specify the base model as the 8B-parameter general reasoning model used in the original experiments. We further include three ablations: (i) topic-aligned vs. unaligned BioAlchemy data at fixed volume (alignment contributes an additional 4.8%), (ii) BioAlchemy-345K vs. an equal-volume sample of non-biology scientific text under identical RL (BioAlchemy yields 6.1% higher gains), and (iii) RL scale control holding data fixed. These results isolate the contribution of the pipeline. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical gains rest on external benchmarks

full rationale

The paper's central claim is an observed 9.12% improvement of BioAlchemist-8B over its base model on biology benchmarks after RL on the curated BioAlchemy-345K dataset. No equations, self-definitions, or fitted parameters are shown reducing the reported gain to the input curation steps by construction. The pipeline (text sourcing, Q&A extraction, topic alignment) is presented as a procedural contribution whose output is then evaluated externally; the benchmarks are treated as held-out test sets. No load-bearing self-citations, uniqueness theorems, or ansatz smuggling appear in the abstract or described derivation. The result is therefore self-contained against external benchmarks and receives the default non-circularity finding.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests on the unstated premise that automated extraction from papers can produce reliably verifiable and challenging reasoning problems at scale; no explicit free parameters, axioms, or invented entities are named in the abstract.

pith-pipeline@v0.9.0 · 5550 in / 1100 out tokens · 43739 ms · 2026-05-13T19:19:29.503436+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages

  1. [1]

    Winning gold at imo 2025 with a model-agnostic verification- and-refinement pipeline.arXiv preprint arXiv:2507.15855, 2025

    URLhttps://github.com/Future-House/LAB-Bench. GitHub repository. Ozan Gokdemir, Neil Getty, Robert Underwood, Sandeep Madireddy, Franck Cappello, Arvind Ramanathan, Ian T Foster, and Rick L Stevens. Automated mcqa benchmarking at scale: Evaluating reasoning traces as retrieval sources for domain adaptation of small language models. InProceedings of the SC...

    work page arXiv 2025

  2. [2]

    Botany − Study of plants (not just mentioning plants)

    work page

  3. [3]

    Cell Biology − Study of cell structure/function (not just using cells as models)

    work page

  4. [4]

    Computational Biology − Use of computational methods in biology

    work page

  5. [5]

    Cryobiology − Study of effects of low temperatures on living things

    work page

  6. [6]

    Cytology − Study of cells (microscopic anatomy)

    work page

  7. [7]

    Developmental Biology − Study of growth and development of organisms

    work page

  8. [8]

    Ecology − Study of organisms and their environment interactions

    work page

  9. [9]

    Exobiology − Study of life beyond Earth / astrobiology

    work page

  10. [10]

    Genetics − Study of genes, heredity, and genetic variation

    work page

  11. [11]

    Laboratory Animal Science − Study of lab animals for research

    work page

  12. [12]

    Marine Biology − Study of ocean organisms and ecosystems

    work page

  13. [13]

    Microbiology − Study of microorganisms (bacteria, viruses, fungi, etc.)

    work page

  14. [14]

    Molecular Biology − Molecular mechanisms (DNA/RNA/protein synthesis) − NOT general biochemistry

    work page

  15. [15]

    Natural History − Observational study of organisms in nature

    work page

  16. [16]

    Neurobiology − Study of nervous system biology

    work page

  17. [17]

    Parasitology − Study of parasites and parasitic diseases

    work page

  18. [18]

    Photobiology − Study of effects of light on living organisms

    work page

  19. [19]

    Radiobiology − Study of effects of radiation on living organisms

    work page

  20. [20]

    Sociobiology − Study of social behavior in biological terms

    work page

  21. [21]

    Synthetic Biology − Design and construction of new biological entities

    work page

  22. [22]

    Microbiology

    Zoology − Study of animals {few_shot_examples} Question: {question} Answer: {answer} Respond with ONLY a JSON array of category names that apply to this QA pair. Example response: ["Microbiology", "Genetics"] If none of the categories apply well, respond with an empty array: [] Categories: Figure 7: Prompt template for QA MeSH classification 21 Preprint. ...

    work page 2025

  23. [23]

    Begin with 1-2 sentences of contextual information that establishes the domain/topic without referencing source materials

    work page

  24. [24]

    Create a challenging question that tests deep understanding

    work page

  25. [25]

    Ensure there is EXACTLY ONE clearly correct answer

    work page

  26. [26]

    Make the other choices plausible but clearly incorrect

    work page

  27. [27]

    The question should focus on a concept or fact that is clearly stated or strongly implied in the text

    work page

  28. [28]

    Number your answer choices from 1 to {num_answers}

    work page

  29. [29]

    The correct answer must be choice number {target_correct_position}

    IMPORTANT: Place the correct answer in position {target_correct_position}. The correct answer must be choice number {target_correct_position}

    work page

  30. [30]

    DO NOT provide explanations for why each answer is correct or incorrect

    work page

  31. [31]

    CRITICAL: Both context and question must be completely self-contained. DO NOT reference any external materials including: ‘the text’, ‘the passage’, ‘the document’, ‘the paper’, ‘the study’, ‘the author states’, ‘according to the text’, ‘as mentioned’, ‘as described’, ‘Appendix’, ‘Figure’, ‘Table’, ‘Section’, ‘Chapter’, ‘above’, ‘below’, or any other refe...

    work page

  32. [32]

    The context and question should read as if testing general knowledge on the topic, not comprehension of a specific text

    work page

  33. [33]

    Answer choices should contain only direct technical information without meta-references to content or sources. Your response must follow this format precisely: CONTEXT: <1-2 sentences establishing domain/topic context> QUESTION: <the question> 1: <first answer choice> 2: <second answer choice> ... CORRECT ANSWER: {target_correct_position} Figure 9: Prompt...

    work page

  34. [34]

    Skip to ### Response Instructions and return NONE for all fields

    If the text passage DOES NOT have a score of 9, it is NOT appropriate for deriving an exam-level problem. Skip to ### Response Instructions and return NONE for all fields

    work page

  35. [35]

    CRITICAL: If you are unable to derive a question-and-answer pair that satisfies all of the steps below for ANY reason, skip to ### Response Instructions and return NONE for all fields

    work page

  36. [36]

    Begin with 1-2 sentences of concise information that establishes the domain/topic WITHOUT referencing any source materials in the text passage

    work page

  37. [37]

    It must be graduate-level or above in difficulty

    Create a challenging, concise, exam-level question that requires extensive reasoning, sequential computation, sequential logic, or pathway logic rather than simple recall. It must be graduate-level or above in difficulty

    work page

  38. [38]

    Questions with corresponding answers that are chemical compounds, protein names, quantitative values, pathways, or specific genes are appropriate

    CRITICAL: Derive a question such that the corresponding answer is a single name, entity, or value that is easily verifiable. Questions with corresponding answers that are chemical compounds, protein names, quantitative values, pathways, or specific genes are appropriate

    work page

  39. [39]

    Answers that are calculations, statements, long entities, or explanations are NOT appropriate

    CRITICAL: Corresponding answers must be a numerical value, name, or short entity STRICTLY. Answers that are calculations, statements, long entities, or explanations are NOT appropriate

    work page

  40. [40]

    Ensure the question has a single clear, precise, and unambiguous answer

    work page

  41. [41]

    Questions and their corresponding answers should be inspired from the content in the text, but CANNOT reference the text passage itself

    work page

  42. [42]

    Questions may involve mechanisms, pathways, evolutionary principles, genetic analysis, experimental design, computational biology, biological theorems, or systems-level understanding

    work page

  43. [43]

    CRITICAL: The context, question and answer must be completely self-contained. DO NOT reference any external materials including: ‘the text’, ‘the passage’, ‘the document’, ‘the paper’, ‘the study’, ‘the author states’, ‘according to the text’, ‘Appendix’, ‘Figure’, ‘Table’, ‘Section’, ‘Chapter’, ‘above’, ‘below’, or any other references to source materials

    work page

  44. [44]

    The context and question should read like an exam question for graduate-level students and above, testing general knowledge or reasoning on biology, not comprehension of a specific text

    work page

  45. [45]

    The answer must contain only direct technical information without any meta-references to content, studies, or source materials

    work page

  46. [46]

    If your question requires an answer in a specific format, specify that in the question itself at the end. ### Response Instructions: Your response must follow this format precisely: CONTEXT: <1-2 sentences establishing domain/topic context; return NONE if problem cannot be derived> QUESTION: <your exam-level question; return NONE if problem cannot be deri...

    work page