REVIEW 4 major objections 6 minor 3 references
Cancer Vaccine Adjuvant Name Recognition from Biomedical Literature using Large Language Models
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read GPT-4o identifies cancer vaccine adjuvant names in abstracts and clinical trials with 100% precision and F1 scores of 77–82% when given few-shot examples and context.
desk verdict A competent but modest LLM NER benchmark whose headline 100% precision is largely an artifact of manual adjudication; the curated datasets and error analysis are the real value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a structured zero-shot and few-shot prompting pipeline over an instruction-tuned transformer. Each prompt defines an adjuvant for the model, forbids generic mentions, demands exact wording, caps the number of outputs, and requests a tab-separated table ending with a 'Done' marker. The two experimental levers are the number of demonstrations, from zero to four, and optional context fields: 'Substances' for PubMed abstracts and 'Interventions' for clinical trials. Outputs are cleaned, deduplicated, matched case-insensitively against a curated dictionary, and every mismatch is adjudicated by two domain experts with a third expert breaking ties. The context fields do the main work, because including them raises recall more than adding examples alone.
What would settle it
Have an independent expert panel, blind to the authors' adjudication and to the gold-standard labels, classify all 928 mismatched LLM outputs from the VAC and AdjuvareDB experiments. If the panel confirms substantially fewer of those outputs as genuine adjuvants than the authors' manual validation did, the 100% precision and the reported F1 scores would not survive recomputation; a second check is whether rerunning the best GPT-4o prompts on a fresh set of cancer-vaccine abstracts yields a similar recall, since the current numbers come from a single curated corpus.
Extended reading notes
Core claim
The central claim is that instruction-following LLMs, especially GPT-4o, can perform adjuvant-name recognition in cancer vaccine literature and clinical trial records with near-perfect precision and meaningfully high recall, and that recall improves when prompts include contextual lists such as substances for abstracts and interventions for trials. This is established by prompting GPT-4o and Llama-3.2-3B with zero-to-four examples, checking outputs first against a curated dictionary, and then having domain experts manually judge the mismatches. Under manual validation, GPT-4o achieves an F1 of 77.32% on VAC abstracts with substances and four shots, and 81.67% on AdjuvareDB clinical trials with interventions and three shots, with 100% precision in both best settings. The paper further claims that some LLM-identified adjuvants are valid but absent from the gold standard, so strict comparison to curated labels may understate true recall.
Load-bearing premise
The results depend on expert manual review treating LLM outputs that are absent from the gold standard as true positives; if that judgment is subjective or inconsistent, the reported precision and F1 are higher than an objective reference would give.
Editorial extensions
If this is right
- Including contextual data in the prompt is a cheap way to improve recall: for GPT-4o, it moved the best VAC F1 from 75.73% to 77.32% and the AdjuvareDB F1 from 77.98% to 81.67%.
- Because precision is essentially 100%, the models' errors are misses rather than false alarms, so a screening pipeline that trusts GPT-4o outputs can direct human effort at unretrieved names instead of filtering out wrong ones.
- LLMs can surface valid adjuvants that are absent from curated databases, making the extraction method useful as a database-curation aid rather than only as a lookup against a fixed standard.
- The same prompt framework may transfer to other vaccine-component or clinical-trial entity extraction tasks without fine-tuning, as the paper states as future work.
- The best number of examples was not always the maximum: three shots beat four for GPT-4o on AdjuvareDB, so prompt design and context matter more than adding demonstrations.
Reading between the lines
- If the manual-validation reclassification is accepted, the natural next step is to build an expanded public benchmark that merges gold-standard labels with adjudicated LLM outputs; without such a benchmark, future systems cannot be directly compared against these F1 numbers.
- The gap between GPT-4o's near-perfect precision and roughly 63–69% recall suggests the limiting factor is coverage of rare surface forms, so injecting ontology synonyms or spelling normalization into the prompt or postprocessor may close the gap faster than more few-shot examples.
- The paper's note that the 1B Llama model struggled to follow the output-format instructions implies that obeying the TSV format, not entity knowledge, may be the binding constraint for smaller open models.
- A testable extension would randomize which context field is included and measure recall on a dataset whose gold-standard labels are known to be complete, isolating the effect of intervention lists from the effect of the model's prior knowledge.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript studies whether off-the-shelf large language models can recognize cancer vaccine adjuvant names from two datasets: 290 PubMed abstracts annotated with the Vaccine Adjuvant Compendium (VAC) and 97 clinical trial records from AdjuvareDB. The authors test GPT-4o and Llama-3.2-3B in zero-shot and few-shot settings, with and without additional contextual fields (substances for abstracts, interventions for trials). They report automated exact-match evaluation followed by manual expert validation of mismatches. The central claim is that GPT-4o attains 100% Precision and F1 scores of 77.32% on VAC abstracts and 81.67% on AdjuvareDB trials after manual validation, outperforming Llama-3.2-3B. The paper also notes that some LLM outputs are valid adjuvants absent from the gold standard.
Significance. If the quantitative results are reliable, the paper would provide a useful demonstration that prompting a general-purpose LLM can partially automate a narrow biomedical curation task, with human-in-the-loop validation compensating for incomplete dictionaries. The release of source code is a practical strength. However, the evaluation protocol as described does not currently support the headline numbers: the manual-validation procedure is underspecified and can convert any mismatch into a true positive, the metric definitions in Eqs. (1)-(3) are non-standard, and no statistical tests are reported despite the Table 4 caption promising them. The claim of LLM superiority over baselines is therefore not yet established.
major comments (4)
- [Section 4.6, Section 5] The manual-validation protocol is not sufficiently specified to reproduce the reported Precision and F1 values. The manuscript states that 928 mismatches were reviewed by at least two experts, with a third adjudicating 144 disagreements, but provides no validator instructions beyond the prompt definition, no per-case decision records, and no inter-annotator agreement metric such as Cohen's kappa. Because every non-exact output is eligible for re-labeling as a true positive, the resulting Precision is bounded below by the adjudication rule rather than by model behavior, and the manual F1 values in Tables 3 and 4 are not reproducible from the information given. The additional filter that only mismatches appearing in at least two runs were reviewed is mentioned in Section 5 but not defined in the Methods; please describe this filter precisely, report validator agreement, and provide a sensitivity analysis that treats manual re-labeling as a separate annotation step rather than as ground truth.
- [Section 5, paragraph on 'valid but unlisted' adjuvants] The treatment of gold-standard-negative outputs is internally ambiguous. The text says that valid adjuvants not in the gold standard were 'considered incorrect' in the current evaluation, yet the manual-validation step appears to re-label such outputs as correct. If manually validated outputs outside the gold standard are counted as true positives, then the recall denominator and precision numerator in Eqs. (1)-(2) are no longer computed against the same reference, and the manual F1 values are not comparable to the automated ones. If they are excluded, then the manual validation cannot be the source of the reported recall gains. Please clarify exactly how true-positive, false-positive, and false-negative counts were computed after manual review, and provide a breakdown of how many gold-negative outputs were re-labeled as true positives per model and setting.
- [Section 4.6, Eqs. (1)-(3)] The metric definitions in Eqs. (1)-(3) are non-standard and insufficiently precise. Precision is written as (True Positive - Nonspecific Output)/True Positive and Recall as True Positive/(Total Identification + Missed Instances), but 'Nonspecific Output', 'Total Identification', and 'Missed Instances' are never defined, and these expressions do not correspond to the conventional TP/(TP+FP) and TP/(TP+FN). Without a formal mapping to the confusion matrix, the reported percentages cannot be independently verified or compared with standard information-extraction benchmarks. Please replace these equations with standard definitions, or explicitly define every variable in them and justify the alternative formulation.
- [Table 4 caption, Section 5] The caption of Table 4 states 'Includes statistical significance of observed improvements,' but no statistical test, confidence interval, or p-value appears anywhere in the manuscript. All comparisons between GPT-4o and Llama are point estimates, and the best-performing shot count is selected post hoc across zero to four shots in Tables 3 and 4. With up to ten comparisons per dataset, the reported maxima are optimistically biased. Please report paired significance tests (e.g., McNemar or bootstrap), confidence intervals for F1 and Recall, and, where the shot count is selected on the test set, an appropriate correction for multiple comparisons.
minor comments (6)
- [Section 5, abstract] The abstract claims GPT-4o attains 100% Precision 'across all situations', but Table 3 reports values of 99.80%, 99.86%, and the two-shot without-substances row is explicitly excepted in the text; please reconcile this wording.
- [Section 1, references] The in-text attributions for Ferber et al. and Hou et al. appear to be swapped: the text describes Ferber as generating physician letters with QLoRA, but the reference entry is about end-to-end clinical trial matching, while the text describes Hou as matching patients to clinical trials, but the reference entry is about physician letter generation.
- [Table 2] In Table 2, the GPT-4o clinical-trial example uses a 'PMID' column and the Llama abstract example uses an 'NCT' column; the column headers appear to be swapped between the two panels.
- [Figures 5 and 6] Figures 5 and 6 have captions that are identical to the corresponding table captions and do not describe the plotted quantities; please make the figure captions self-contained.
- [Throughout] There are numerous typos and stylistic inconsistencies, including 'notabl e' in Section 1, 'LlaMA', 'GPT' vs 'GPT-4', and 'Llama-3.2-3 B' in the abstract; a careful proofreading pass is needed.
- [Section 4.6] The phrase 'curated dictionary of predefined mappings' should be made concrete: the dictionary is never listed, and it is unclear whether it contains exact gold-standard names, aliases, or both. A supplementary file with the dictionary would improve reproducibility.
Circularity Check
No circularity: empirical benchmark with fixed external gold standards and non-load-bearing self-citations.
full rationale
This paper reports an empirical evaluation of LLM outputs against two external gold-standard resources, AdjuvareDB and the Vaccine Adjuvant Compendium. There is no fitted parameter that is later renamed as a prediction, and no uniqueness theorem or ansatz is imported from prior work to force the central result. The only potentially self-referential element is manual validation of automated mismatches in Section 4.6. However, the paper explicitly states that valid adjuvants not present in the gold standard were still counted as incorrect: 'In the current evaluation settings, such valid adjuvants were considered incorrect simply because they were not part of the gold standard reference.' This means the manual review did not renegotiate the reference standard, so the reported precision and F1 scores are not true by construction. The self-citations that exist, such as VaxLLM and Vaxjo, are background descriptions or related work and do not carry the argument. The main quantitative claim therefore has independent empirical content, and any concern about manual validation is an evaluation-reproducibility issue, not circularity.
Assumptions & free parameters
free parameters (5)
- temperature =
0.0001
- max token limit =
100
- few-shot example count =
0 to 4, best selected
- manual validation agreement rule =
two validators plus third tie-breaker
- mismatch review filter =
appears in at least two runs
assumptions (4)
- domain assumption Adjuvant names appear explicitly in text and can be recognized from surface mentions without external knowledge.
- domain assumption The AdjuvareDB and VAC gold standard annotations are accurate references for scoring.
- domain assumption Expert manual judgment is a valid ground truth for reclassifying LLM outputs not in the gold standard.
- domain assumption The two small datasets represent the broader cancer vaccine adjuvant literature.
Cite this review
Pith. "Pith review of Cancer Vaccine Adjuvant Name Recognition from Biomedical Literature using Large Language Models." pith.science (2026). https://pith.science/paper/YRJ6MJPQ
@misc{pith2026250209659,
author = {Pith},
title = {Pith review of: Cancer Vaccine Adjuvant Name Recognition from Biomedical Literature using Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/YRJ6MJPQ}},
note = {Machine review of arXiv:2502.09659}
}
read the original abstract
Motivation: An adjuvant is a chemical incorporated into vaccines that enhances their efficacy by improving the immune response. Identifying adjuvant names from cancer vaccine studies is essential for furthering research and enhancing immunotherapies. However, the manual curation from the constantly expanding biomedical literature poses significant challenges. This study explores the automated recognition of vaccine adjuvant names using Large Language Models (LLMs), specifically Generative Pretrained Transformers (GPT) and Large Language Model Meta AI (Llama). Methods: We utilized two datasets: 97 clinical trial records from AdjuvareDB and 290 abstracts annotated with the Vaccine Adjuvant Compendium (VAC). GPT-4o and Llama 3.2 were employed in zero-shot and few-shot learning paradigms with up to four examples per prompt. Prompts explicitly targeted adjuvant names, testing the impact of contextual information such as substances or interventions. Outputs underwent automated and manual validation for accuracy and consistency. Results: GPT-4o attained 100% Precision across all situations while exhibiting notable improve in Recall and F1-scores, particularly with incorporating interventions. On the VAC dataset, GPT-4o achieved a maximum F1-score of 77.32% with interventions, surpassing Llama-3.2-3B by approximately 2%. On the AdjuvareDB dataset, GPT-4o reached an F1-score of 81.67% for three-shot prompting with interventions, surpassing Llama-3.2-3 B's maximum F1-score of 65.62%. Conclusion: Our findings demonstrate that LLMs excel at identifying adjuvant names, including rare variations of naming representation. This study emphasizes the capability of LLMs to enhance cancer vaccine development by efficiently extracting insights. Future work aims to broaden the framework to encompass various biomedical literature and enhance model generalizability across various vaccines and adjuvants.
Figures
Reference graph
Works this paper leans on
-
[2]
Exploring Large Language Models for Specialist-level Oncology Care
Personalized cancer vaccine design using AI - powered technologies. Frontiers in Immunology, 15, 1357217. LI, X., ZHENG, Y., HU, J., ZHENG, J., WANG, Z. & HE, Y. 2024. VaxLLM: Leveraging Fine -tuned Large Language Model for automated annotation of Brucella Vaccines. bioRxiv, 2024.11. 25.625209. PALEPU, A., DHILLON, V., NIRAVATH, P., WENG, W. -H., PRASAD, ...
work page Pith review arXiv 2024
-
[2021]
Vaccine development throughout history. Cureus, 13. SAYERS, S., ULYSSE, G. & HE, Y. 2012. Vaxjo: A Web -Based Vaccine Adjuvant Database and Its Application for Analysis of Vaccine Adjuvants and Their Uses. SUSHIL, M., KENNEDY, V. E., MANDAIR, D., MIAO, B. Y., ZACK, T. & BUTTE, A. J. 2024. CORAL: expert -curated oncology reports to advance language model i...
arXiv 2012
-
[2024]
End-To-End Clinical Trial Matching with Large Language Models
End-to-end clinical trial matching with large language models. arXiv preprint arXiv:2407.13463. HOU, Y., BERT, C., GOMAA, A., LAHMER, G., HOEFLER, D., WEISSMANN, T., VOIGT, R., SCHUBERT, P., SCHMITTER, C. & DEPARDON, A. 2024. Fine -Tuning a Local LLaMA-3 Large Language Model for Automated Privacy-Preserving Physician Letter Generation in Radiation Oncolog...
work page Pith review arXiv 2024
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.