{"id":"81a246c3-4e46-46c1-95b8-824baa3d0f78","arxiv_id":"2502.09659","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GPT-4o outperforms Llama 3.2 at extracting cancer vaccine adjuvant names from clinical trial records and abstracts, reaching F1 scores of 81.67% and 77.32% on two curated datasets.","lead":"This paper tests whether GPT-4o and Llama 3.2 can automatically pull vaccine adjuvant names from cancer vaccine clinical trial records and PubMed abstracts. If reliable, this automation could speed up curation of adjuvant databases and support cancer vaccine research.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manual validation is the linchpin: the 100% Precision and manual F1 scores rest on expert adjudication of every mismatch; without a pre-registered protocol or agreement metric, those numbers are not reproducible.","rationale":"The reader's weakest assumption identifies manual validation as the source of potential inflation, and that is exactly where the argument is least secure. The paper's quantitative claims—especially the 100% Precision—cannot be separated from the subjective process used to adjudicate every non-exact match. Without decision rules, validator agreement statistics, or released adjudication data, the reported numbers are not independently verifiable. I considered whether the absence of fine-tuned NER baselines is more load-bearing, but the paper's central claim is about LLM capability, not relative superiority over all existing methods; the baseline gap weakens the practical conclusion but does not directly invalidate the reported metrics. The manual-validation issue, by contrast, determines whether the metrics themselves mean what they appear to mean. Since the reader already imposed a CONDITIONAL verdict, my read does not move the verdict; it reinforces the condition.","tokens_in":12048,"tokens_out":4181,"duration_ms":35072,"concrete_test":"Ask the authors to release the full adjudication data for all 928 mismatch cases (source snippet, candidate output, each validator's label, and the third-validator decision) along with the exact decision rules used. Independently re-adjudicate a random 100-case subset with two annotators blinded to the original labels, using only the paper's stated definition of adjuvant and a pre-specified alias/normalization list; compute Cohen's kappa and recompute F1 both exact-match-only and with the re-adjudicated labels. If kappa < 0.8, or if the re-adjudicated F1 for GPT-4o differs by more than about 2 points, the manual-validation-based precision and F1 claims are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result—GPT-4o reaching 100% Precision and F1 ~77–82%—depends entirely on Section 4.6's manual validation. Automated validation is only exact-match against a dictionary; every non-matching output is then reviewed by experts, and mismatches that experts deem correct are counted as true positives. The paper reports 928 such cases and 144 validator disagreements, but no decision criteria, no validator instructions beyond the prompt's definition of 'adjuvant', no per-case labels, and no inter-annotator agreement (e.g., Cohen's kappa) are provided. Because the manual step can re-label any LLM output as correct, Precision cannot fall: a candidate is wrong only if two validators say so, and a third can break ties. This makes the headline 100% Precision partly a property of the adjudication procedure, not of the model. The same subjectivity inflates the manual F1 relative to the automated F1. The paper also notes valid-but-unlisted adjuvants were 'considered incorrect' in the current evaluation, but it is unclear whether such cases were later counted as true positives; this ambiguity changes the reported numbers. A fixed alias/normalization table, pre-registered rules, or independent blind re-annotation is needed before the quantitative claims can be treated as load-bearing evidence for LLM superiority.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies whether off-the-shelf large language models can recognize cancer vaccine adjuvant names from two datasets: 290 PubMed abstracts annotated with the Vaccine Adjuvant Compendium (VAC) and 97 clinical trial records from AdjuvareDB. The authors test GPT-4o and Llama-3.2-3B in zero-shot and few-shot settings, with and without additional contextual fields (substances for abstracts, interventions for trials). They report automated exact-match evaluation followed by manual expert validation of mismatches. The central claim is that GPT-4o attains 100% Precision and F1 scores of 77.32% on VAC abstracts and 81.67% on AdjuvareDB trials after manual validation, outperforming Llama-3.2-3B. The paper also notes that some LLM outputs are valid adjuvants absent from the gold standard.","tokens_in":12276,"tokens_out":4607,"duration_ms":42592,"significance":"If the quantitative results are reliable, the paper would provide a useful demonstration that prompting a general-purpose LLM can partially automate a narrow biomedical curation task, with human-in-the-loop validation compensating for incomplete dictionaries. The release of source code is a practical strength. However, the evaluation protocol as described does not currently support the headline numbers: the manual-validation procedure is underspecified and can convert any mismatch into a true positive, the metric definitions in Eqs. (1)-(3) are non-standard, and no statistical tests are reported despite the Table 4 caption promising them. The claim of LLM superiority over baselines is therefore not yet established.","major_comments":[{"comment":"The manual-validation protocol is not sufficiently specified to reproduce the reported Precision and F1 values. The manuscript states that 928 mismatches were reviewed by at least two experts, with a third adjudicating 144 disagreements, but provides no validator instructions beyond the prompt definition, no per-case decision records, and no inter-annotator agreement metric such as Cohen's kappa. Because every non-exact output is eligible for re-labeling as a true positive, the resulting Precision is bounded below by the adjudication rule rather than by model behavior, and the manual F1 values in Tables 3 and 4 are not reproducible from the information given. The additional filter that only mismatches appearing in at least two runs were reviewed is mentioned in Section 5 but not defined in the Methods; please describe this filter precisely, report validator agreement, and provide a sensitivity analysis that treats manual re-labeling as a separate annotation step rather than as ground truth.","section":"Section 4.6, Section 5"},{"comment":"The treatment of gold-standard-negative outputs is internally ambiguous. The text says that valid adjuvants not in the gold standard were 'considered incorrect' in the current evaluation, yet the manual-validation step appears to re-label such outputs as correct. If manually validated outputs outside the gold standard are counted as true positives, then the recall denominator and precision numerator in Eqs. (1)-(2) are no longer computed against the same reference, and the manual F1 values are not comparable to the automated ones. If they are excluded, then the manual validation cannot be the source of the reported recall gains. Please clarify exactly how true-positive, false-positive, and false-negative counts were computed after manual review, and provide a breakdown of how many gold-negative outputs were re-labeled as true positives per model and setting.","section":"Section 5, paragraph on 'valid but unlisted' adjuvants"},{"comment":"The metric definitions in Eqs. (1)-(3) are non-standard and insufficiently precise. Precision is written as (True Positive - Nonspecific Output)/True Positive and Recall as True Positive/(Total Identification + Missed Instances), but 'Nonspecific Output', 'Total Identification', and 'Missed Instances' are never defined, and these expressions do not correspond to the conventional TP/(TP+FP) and TP/(TP+FN). Without a formal mapping to the confusion matrix, the reported percentages cannot be independently verified or compared with standard information-extraction benchmarks. Please replace these equations with standard definitions, or explicitly define every variable in them and justify the alternative formulation.","section":"Section 4.6, Eqs. (1)-(3)"},{"comment":"The caption of Table 4 states 'Includes statistical significance of observed improvements,' but no statistical test, confidence interval, or p-value appears anywhere in the manuscript. All comparisons between GPT-4o and Llama are point estimates, and the best-performing shot count is selected post hoc across zero to four shots in Tables 3 and 4. With up to ten comparisons per dataset, the reported maxima are optimistically biased. Please report paired significance tests (e.g., McNemar or bootstrap), confidence intervals for F1 and Recall, and, where the shot count is selected on the test set, an appropriate correction for multiple comparisons.","section":"Table 4 caption, Section 5"}],"minor_comments":[{"comment":"The abstract claims GPT-4o attains 100% Precision 'across all situations', but Table 3 reports values of 99.80%, 99.86%, and the two-shot without-substances row is explicitly excepted in the text; please reconcile this wording.","section":"Section 5, abstract"},{"comment":"The in-text attributions for Ferber et al. and Hou et al. appear to be swapped: the text describes Ferber as generating physician letters with QLoRA, but the reference entry is about end-to-end clinical trial matching, while the text describes Hou as matching patients to clinical trials, but the reference entry is about physician letter generation.","section":"Section 1, references"},{"comment":"In Table 2, the GPT-4o clinical-trial example uses a 'PMID' column and the Llama abstract example uses an 'NCT' column; the column headers appear to be swapped between the two panels.","section":"Table 2"},{"comment":"Figures 5 and 6 have captions that are identical to the corresponding table captions and do not describe the plotted quantities; please make the figure captions self-contained.","section":"Figures 5 and 6"},{"comment":"There are numerous typos and stylistic inconsistencies, including 'notabl e' in Section 1, 'LlaMA', 'GPT' vs 'GPT-4', and 'Llama-3.2-3 B' in the abstract; a careful proofreading pass is needed.","section":"Throughout"},{"comment":"The phrase 'curated dictionary of predefined mappings' should be made concrete: the dictionary is never listed, and it is unclear whether it contains exact gold-standard names, aliases, or both. A supplementary file with the dictionary would improve reproducibility.","section":"Section 4.6"}],"recommendation":"major_revision","confidential_remarks":"The central idea is reasonable and the code availability is a plus, but the evaluation protocol is the crux. The manual-validation step can re-classify any mismatch as a true positive, and the text is ambiguous about how gold-negative outputs are counted; this issue alone prevents the quantitative claims from being treated as evidence. The citation swap for Ferber/Hou should also be corrected. The paper is potentially salvageable with a substantially expanded validation section, but the current version does not meet the standard for a archival benchmark paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, modest benchmark of GPT-4o and Llama 3.2 for cancer vaccine adjuvant name extraction. The headline 100% precision is not meaningful as stated because it is produced largely by the manual adjudication procedure. The real contributions are the two curated datasets, the finding that contextual fields help recall, and the observation that LLMs surface valid adjuvants missing from the gold standard.\n\nWhat is new: the authors apply zero/few-shot prompting to two existing databases (AdjuvareDB clinical trials and VAC abstracts) for a narrow NER task. VaxLLM already did fine-tuned vaccine-component annotation on Brucella vaccines; this paper shifts to cancer vaccine adjuvants and to prompting rather than fine-tuning. The manual review step also provides a genuine signal about database incompleteness: the fact that LLM outputs include correct adjuvants absent from the reference is useful for curators.\n\nWhat the paper does well: the prompts are given in full, the code is on GitHub, the two-step automated-plus-manual validation is described clearly, and the comparison of GPT-4o versus Llama is plausible and likely reproducible. The discussion of formatting failures (e.g., Llama 1B) is honest.\n\nSoft spots, in order of severity:\n\nThe manual validation is the load-bearing wall. Precision cannot drop materially because every non-matching output is either accepted by expert judgment or rejected only if two validators agree. There is no decision criteria, no inter-annotator agreement, no per-case labels. The 928 mismatches and 144 disagreements are mentioned but not itemized. The paper even admits that valid-but-unlisted adjuvants were 'considered incorrect' in the current evaluation, which muddies what the numbers mean.\n\nTable 4's caption promises statistical significance, but no test is reported.\n\nBest few-shot counts are chosen post hoc across 0–4 shots, so the headline F1 values are selected maxima rather than a pre-registered comparison.\n\nNo fine-tuned NER baseline (e.g., BioBERT or VaxLLM-style fine-tuned Llama) is compared. The claim that off-the-shelf LLMs 'excel' is only against another off-the-shelf LLM, not against the standard supervised approach.\n\nNone of these flaws kill the paper's main use case: a first-pass automation tool for a niche curation problem. If the authors release the manual review protocol and adjudicated labels, add a fine-tuned baseline, and report the actual significance test, the numbers would be worth taking seriously. As it stands, treat the quantitative claims as suggestive rather than load-bearing.\n\nI would send this to peer review with the expectation of heavy revision, mostly to pin down the evaluation protocol. For a reading group, it is a good case study in how evaluation design can inflate or obscure model quality.","headline":"A competent but modest LLM NER benchmark whose headline 100% precision is largely an artifact of manual adjudication; the curated datasets and error analysis are the real value.","tokens_in":12871,"tokens_out":2675,"would_cite":false,"duration_ms":44410,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4o identifies cancer vaccine adjuvant names in abstracts and clinical trials with 100% precision and F1 scores of 77–82% when given few-shot examples and context.","keywords":["adjuvant name recognition","cancer vaccines","large language models","GPT-4o","Llama 3.2","named entity recognition","biomedical text mining","few-shot prompting"],"falsifier":"Have an independent expert panel, blind to the authors' adjudication and to the gold-standard labels, classify all 928 mismatched LLM outputs from the VAC and AdjuvareDB experiments. If the panel confirms substantially fewer of those outputs as genuine adjuvants than the authors' manual validation did, the 100% precision and the reported F1 scores would not survive recomputation; a second check is whether rerunning the best GPT-4o prompts on a fresh set of cancer-vaccine abstracts yields a similar recall, since the current numbers come from a single curated corpus.","tokens_in":11831,"feed_emoji":"💉","tokens_out":6345,"duration_ms":50349,"temperature":0.7,"pith_summary":"This paper tests whether off-the-shelf large language models can pull cancer vaccine adjuvant names out of PubMed abstracts and clinical trial records. On Vaccine Adjuvant Compendium (VAC) abstracts, GPT-4o reached 100% precision and a manual-validation F1 of 77.32% when the prompt included the article's listed substances. On 97 AdjuvareDB clinical trials, GPT-4o reached an F1 of 81.67% with three-shot prompting plus interventions, beating Llama-3.2-3B by about 16 points. The authors also found that the models routinely output valid adjuvant names that are missing from the gold standard, suggesting curated databases are incomplete and LLMs can help fill those gaps. If these results hold, prompt-only LLM extraction is a workable partial substitute for manual curation of adjuvant names.","feed_headline":"GPT-4o finds vaccine adjuvants in text with 100% precision","feed_subtitle":"Adding substances and trial interventions to prompts lifts F1 to 77–82%, a step toward automated literature curation.","key_machinery":"The mechanism is a structured zero-shot and few-shot prompting pipeline over an instruction-tuned transformer. Each prompt defines an adjuvant for the model, forbids generic mentions, demands exact wording, caps the number of outputs, and requests a tab-separated table ending with a 'Done' marker. The two experimental levers are the number of demonstrations, from zero to four, and optional context fields: 'Substances' for PubMed abstracts and 'Interventions' for clinical trials. Outputs are cleaned, deduplicated, matched case-insensitively against a curated dictionary, and every mismatch is adjudicated by two domain experts with a third expert breaking ties. The context fields do the main work, because including them raises recall more than adding examples alone.","core_discovery":"The central claim is that instruction-following LLMs, especially GPT-4o, can perform adjuvant-name recognition in cancer vaccine literature and clinical trial records with near-perfect precision and meaningfully high recall, and that recall improves when prompts include contextual lists such as substances for abstracts and interventions for trials. This is established by prompting GPT-4o and Llama-3.2-3B with zero-to-four examples, checking outputs first against a curated dictionary, and then having domain experts manually judge the mismatches. Under manual validation, GPT-4o achieves an F1 of 77.32% on VAC abstracts with substances and four shots, and 81.67% on AdjuvareDB clinical trials with interventions and three shots, with 100% precision in both best settings. The paper further claims that some LLM-identified adjuvants are valid but absent from the gold standard, so strict comparison to curated labels may understate true recall.","pith_inferences":["If the manual-validation reclassification is accepted, the natural next step is to build an expanded public benchmark that merges gold-standard labels with adjudicated LLM outputs; without such a benchmark, future systems cannot be directly compared against these F1 numbers.","The gap between GPT-4o's near-perfect precision and roughly 63–69% recall suggests the limiting factor is coverage of rare surface forms, so injecting ontology synonyms or spelling normalization into the prompt or postprocessor may close the gap faster than more few-shot examples.","The paper's note that the 1B Llama model struggled to follow the output-format instructions implies that obeying the TSV format, not entity knowledge, may be the binding constraint for smaller open models.","A testable extension would randomize which context field is included and measure recall on a dataset whose gold-standard labels are known to be complete, isolating the effect of intervention lists from the effect of the model's prior knowledge."],"forward_implications":["Including contextual data in the prompt is a cheap way to improve recall: for GPT-4o, it moved the best VAC F1 from 75.73% to 77.32% and the AdjuvareDB F1 from 77.98% to 81.67%.","Because precision is essentially 100%, the models' errors are misses rather than false alarms, so a screening pipeline that trusts GPT-4o outputs can direct human effort at unretrieved names instead of filtering out wrong ones.","LLMs can surface valid adjuvants that are absent from curated databases, making the extraction method useful as a database-curation aid rather than only as a lookup against a fixed standard.","The same prompt framework may transfer to other vaccine-component or clinical-trial entity extraction tasks without fine-tuning, as the paper states as future work.","The best number of examples was not always the maximum: three shots beat four for GPT-4o on AdjuvareDB, so prompt design and context matter more than adding demonstrations."],"supporting_citations":[{"why":"Supplies the GPT-4o system card for the proprietary model used in the central experiments.","marker":"Hurst et al., 2024"},{"why":"Provides the AdjuvareDB clinical-trial dataset and its gold-standard annotations evaluated in the study.","marker":"Ren et al., 2024"},{"why":"Introduces the Llama model family used as the open-source comparison system.","marker":"Touvron et al., 2023"},{"why":"VaxLLM is the prior fine-tuned LLM approach for vaccine-component annotation that this zero-shot and few-shot study builds on and contrasts with.","marker":"Li et al., 2024"},{"why":"Demonstrates zero-shot and few-shot LLM learning in personalized oncology, motivating the prompting strategy used here.","marker":"Benary et al., 2023"},{"why":"Prior evaluation of GPT and BERT models for biomedical relation extraction, supporting the choice of GPT for biomedical text mining.","marker":"Rehana et al., 2024b"}],"fun_headline_variants":["GPT-4o nails cancer vaccine adjuvants with 100% precision","Adding context lifts GPT-4o's adjuvant F1 to 82%","LLMs spot rare adjuvant names in cancer vaccine literature","GPT-4o outperforms Llama on adjuvant extraction with context","Automated curation: GPT-4o reads cancer trials for adjuvants"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on expert manual review treating LLM outputs that are absent from the gold standard as true positives; if that judgment is subjective or inconsistent, the reported precision and F1 are higher than an objective reference would give.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o nails cancer vaccine adjuvants with 100% precision","Adding context lifts GPT-4o's adjuvant F1 to 82%","LLMs spot rare adjuvant names in cancer vaccine literature","GPT-4o outperforms Llama on adjuvant extraction with context","Automated curation: GPT-4o reads cancer trials for adjuvants"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2835,"prompt_tokens":1084,"completion_tokens":1751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":1657}},"tokens_in":700,"tokens_out":1751,"duration_ms":11908,"temperature":1.0,"reasoning_tokens":1657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T10:14:03.081540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent expert panel, blind to the authors' adjudication and to the gold-standard labels, classify all 928 mismatched LLM outputs from the VAC and AdjuvareDB experiments. If the panel confirms substantially fewer of those outputs as genuine adjuvants than the authors' manual validation did, the 100% precision and the reported F1 scores would not survive recomputation; a second check is whether rerunning the best GPT-4o prompts on a fresh set of cancer-vaccine abstracts yields a similar recall, since the current numbers come from a single curated corpus.","supporting_citations":[],"review_version":1}