{"id":"00ce4d6a-b1c2-4723-b7de-66b0cf426fcb","arxiv_id":"2502.00694","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning the MAMMAL protein language model predicts influenza HA binding and hemagglutination inhibition from antibody and antigen sequences, with high accuracy on seen antibodies but only moderate accuracy on novel ones.","lead":"The authors fine-tune the MAMMAL biomedical language model to predict whether antibodies bind to and block influenza A hemagglutinin (HA) proteins, using only antibody and HA sequences. The model reaches AUROC above 0.9 for known antibody/HA combinations but drops to 0.63-0.73 when predicting entirely new antibodies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HA-exclusive AUROC=0.9 may reflect per-antibody breadth memorization rather than HA-specific prediction; an antibody-only ablation is needed.","rationale":"The central claim is that fine-tuning MAMMAL on a few thousand assay pairs yields a sequence-only model that predicts activity for existing antibodies against unseen HAs (AUROC 0.9) and gives moderate signal for novel antibodies. What must be true for this claim is that the model uses both antibody and HA sequences to make interaction-specific predictions. In the HA-exclusive split, however, test antibodies appear in training; the model can solve the task by memorizing each antibody's overall positivity. The paper's own mAb-exclusive splits demonstrate the fragility: once antibodies are held out, AUROC drops to 0.73 and to 0.63–0.66 under cluster-exclusive splits. The absence of an antibody-only baseline means the 0.9 HA-exclusive number cannot be attributed to HA generalization. This is a load-bearing gap because the abstract and Discussion explicitly emphasize utility for strain surveillance and antibody evaluation against emerging variants, which is exactly the HA-exclusive scenario. The reader's weakest assumption was threshold-derived binary labels; that is a valid concern about label meaning, but even with perfect continuous labels the shortcut would persist. I therefore partially agree with the reader: the missing-baseline issue is closely related to their call for stronger baselines, but the specific mechanism I identify, antibody-identity memorization in the HA-exclusive split, is more decisive and testable. The verdict stays CONDITIONAL; the paper should add the ablation or baseline before the unseen-HA claim is accepted.","tokens_in":15113,"tokens_out":4454,"duration_ms":47271,"concrete_test":"Under the exact HA-exclusive 5-fold protocol, retrain the MAMMAL-finetuned model with every HA sequence replaced by a single fixed placeholder (or the HA portion of the input removed), leaving antibody sequences unchanged. If the AUROC remains near 0.9 or drops by less than 0.05, the reported HA-exclusive performance is explained by antibody-identity/breadth memorization, not by HA-specific sequence modeling. A complementary check is to score test pairs by each antibody's training-set positive rate alone; if that baseline AUROC is at least 0.85 under HA-exclusive splits, the same conclusion follows.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the HA-exclusive split (Methods: Data splits), each test pair contains an antibody whose sequence and, for other HAs, assay labels appeared in training. The model input includes the full antibody HC/LC sequence. The model can therefore learn an antibody-specific base rate (broad vs narrow reactivity) from the antibody's many training pairs (mean 28±17 assays per mAb) and rank test pairs by that base rate, essentially ignoring the HA sequence. The observed pattern is consistent with this shortcut: AUROC is 0.90 when antibodies are seen (HA-exclusive) but falls to 0.73 when antibodies are unseen (mAb-exclusive) and to 0.63–0.66 when antibody clusters are held out (Table 4). The paper's only control is random weight initialization; it does not compare against an antibody-only model that receives no HA information. Consequently, the headline claim of 0.9 AUROC for 'unseen HAs' is not yet evidence of HA-specific generalization; it may simply reflect memorized antibody identity and breadth. The threshold-label issue noted by the reader is secondary: even with perfect labels, this shortcut would inflate the HA-exclusive result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper fine-tunes the MAMMAL biomedical language model on antibody–hemagglutinin (HA) sequence pairs to predict binary binding (ELISA) and hemagglutination inhibition (HAI) activity, using a dataset of 188 mAbs and 79 HAs assembled largely from the authors' prior studies. The evaluation applies 5-fold cross-validation under four split regimes (lenient, HA-exclusive, mAb-exclusive, and mAb-cluster-exclusive) and reports AUROC and AUPRC, comparing MAMMAL-initialized models against randomly initialized models. The headline results are AUROC ≥ 0.91 for lenient splits, 0.90 for HA-exclusive splits, 0.73 for mAb-exclusive splits, and 0.63–0.66 for the mAb-cluster-exclusive split, with parallel trends for binding and HAI tasks. The paper also reports a supplementary analysis of antibody breadth prediction.","tokens_in":15318,"tokens_out":3515,"duration_ms":37667,"significance":"If the reported performance is taken at face value, the approach could be practically useful for prioritizing antibody candidates against known and emerging HA sequences, and the study is one of a small number to evaluate sequence-only antibody–antigen activity prediction under different generalization scenarios. The evaluation design is a real strength: four explicitly defined split types, per-fold label-balance maintenance, both AUROC and AUPRC, and a random-initialization control that supports attributing some gains to pretrained MAMMAL weights. The breadth-prediction analysis is a useful additional check. However, the central claim of high performance on unseen HAs (HA-exclusive split) is not yet established, because the split design allows the model to exploit antibody identity and per-antibody breadth without genuinely using HA sequence information; the paper lacks the antibody-only ablation needed to test this. The hand-chosen assay thresholds also affect every reported number. With additional controls and benchmarks, the manuscript could support its conclusions.","major_comments":[{"comment":"The HA-exclusive split keeps all pairs with the same HA sequence in the same fold, but the same antibodies appear in both training and test folds across many other HA partners (mean 28±17 assays per mAb). The model sees the full antibody HC/LC sequences at test time and can therefore rank test pairs almost entirely by an antibody-specific base rate of broad versus narrow reactivity, without reading the HA sequence. The observed progression—AUROC 0.90 for HA-exclusive, 0.73 for mAb-exclusive, and 0.63–0.66 for mAb-cluster-exclusive—is exactly the pattern an antibody-only shortcut would produce. The random-initialization control does not rule this out, because the randomly initialized model also receives antibody identity. I request an ablation that removes or masks the HA input (e.g., an antibody-only model) or, at minimum, a simple per-antibody training-positive-rate baseline, to quantify how much of the HA-exclusive AUROC is attributable to HA-specific generalization. Without this, the abstract's claim that the model achieves 'an AUROC of 0.9 for unseen HAs' is not supported as a statement about HA-specific prediction.","section":"Data splits"},{"comment":"All binary labels are derived from hand-chosen assay thresholds: AUC-ELISA > 1 for binding and minimum inhibitory concentration < 10 μg/mL for HAI. The paper does not report the distribution of raw assay values near these cutoffs, nor any sensitivity analysis with alternative thresholds. If assay noise or different source-paper protocols systematically straddle the thresholds, the reported AUROC values measure threshold crossing rather than a stable biological activity. Please add a sensitivity analysis (e.g., AUC-ELISA > 2 and HAI < 1 μg/mL) and report the number of pairs within, say, ±1 AUC unit or ±1 dilution of the cutoff. This is load-bearing because every headline number in the abstract and Table 4 is computed against these particular cutoffs.","section":"Data description"},{"comment":"The paper provides no comparison against non-LLM baselines or previously published antibody–HA predictors, such as sequence-similarity k-nearest-neighbors, logistic regression on antibody or HA embeddings, or the existing HAI machine-learning models cited in Related Work. Without an external reference, the practical significance of the AUROC differences between MAMMAL fine-tuning and random initialization is unclear, and the added value of the pretrained model over simpler sequence features is not demonstrated. Please add at least one lightweight sequence-based baseline (e.g., k-NN on antibody CDR similarity, or a linear classifier on OAS/UniProt embeddings) across the same four splits.","section":"Results"}],"minor_comments":[{"comment":"The text contains a broken cross-reference: 'As shown in Error! Reference source not found.3' should refer to Figure 3.","section":"Results"},{"comment":"In the breadth-prediction paragraph, the sentence 'Table 4 presents antibody statistics for each assay type' appears to refer to Table 3, which is the correct table for antibody breadth statistics; Table 4 reports model evaluation.","section":"Results"},{"comment":"Methods, model training: 'ensuring input sequences are not truncates' should read 'not truncated'.","section":"Methods"},{"comment":"The abstract contains a typo: 'eYicient' should be 'efficient'.","section":"Abstract"},{"comment":"The Figure 1 caption mentions a spider diagram for performance comparison, but no spider diagram appears in the main text or appendix; either add it or remove the mention.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The dataset is a convenience sample drawn largely from the authors' own prior studies; the Discussion acknowledges this, but the absence of any independent external validation set is a limitation that editors should weigh when considering the paper's generalizability claims. The HA-exclusive split result is the most likely to be over-interpreted, and the requested antibody-only ablation is essential before publication. The paper would also benefit from a benchmark against at least one published antibody–HA prediction method, although the absence of a benchmark alone would not be disqualifying."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid application paper, not a breakthrough. The four-split evaluation is genuinely careful, and the random-init control is a nice touch. The numbers in the abstract match the tables. But the headline HA-exclusive result is undercut by a known shortcut, and the paper doesn't run the one control that would settle it.\n\nWhat's new: a new antibody-HA binding/HAI dataset (mostly from the authors' prior work), the MAMMAL fine-tune applied to it, and an honest breakdown across lenient/HA-exclusive/mAb-exclusive/cluster splits. The breadth-of-protection analysis is a useful addition. The citation pattern looks fine; MAMMAL and related antibody language models are cited.\n\nSoft spots, in order of seriousness:\n\n1. The 0.90 AUROC on the HA-exclusive split does not mean what the prose says. In that split, each test pair contains an antibody whose sequence and labels for other HAs are in training. The model inputs the full antibody sequence, and with 28±17 training pairs per antibody it can learn a per-antibody base rate (broad vs narrow) and rank test pairs by that rate, largely ignoring the HA. The fact that lenient and HA-exclusive are nearly identical (0.92/0.90) while mAb-exclusive drops to 0.73 is exactly the signature of this shortcut. The right control is an 'antibody-only' model with the HA sequence removed or shuffled; the paper has only the random-init control, which doesn't address this. This is a load-bearing gap for the main claim.\n\n2. The binary labels rest on hand-picked thresholds (ELISA AUC>1, HAI<10 µg/mL). That's not fatal, but it's another reason the AUROC numbers shouldn't be read as biological ground truth.\n\n3. No strong baselines. The random-init model is a weak comparison; a k-mer or sequence-identity baseline would help calibrate whether MAMMAL adds anything beyond antibody identity.\n\n4. The dataset is not released. Fine for now, but it blocks independent check.\n\nMy read: the stress-test concern is real and should be addressed before the paper is accepted. The mAb-exclusive and cluster results (0.73, 0.63–0.66) are probably closer to the true generalization ability and are still moderately useful. This is a legitimate contribution to antibody informatics, worth peer review, but the authors should be pushed to add the ablation and calibrate claims.","headline":"Competent MAMMAL-for-HA application with careful split design, but the 0.9 AUROC for 'unseen HAs' likely leans on memorized antibody breadth; needs an antibody-only ablation.","tokens_in":15894,"tokens_out":3016,"would_cite":false,"duration_ms":30406,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A sequence-only language model fine-tuned on around 5,000 assay pairs predicts antibody binding and hemagglutination inhibition against influenza A hemagglutinin, reaching AUROC ≥ 0.91 for known antibodies and 0.90 for unseen HAs.","keywords":["antibody activity prediction","influenza A hemagglutinin","MAMMAL","language model fine-tuning","sequence-based prediction","hemagglutination inhibition","ELISA binding","data splits"],"falsifier":"Re-run the same fine-tuning procedure on an independently generated antibody–HA dataset with harmonized assay protocols, and check whether AUROC on an antibody-exclusive split stays near 0.73; alternatively, retrain after shifting the positive cutoffs (for example AUC-ELISA > 5 or HAI MIC < 1 µg/mL) and see whether the ranking of model variants reverses.","tokens_in":14902,"feed_emoji":"💉","tokens_out":6834,"duration_ms":64317,"temperature":0.7,"pith_summary":"The paper claims that a language model fine-tuned on a few thousand antibody–hemagglutinin assay pairs can predict, from amino acid sequence alone, whether an antibody binds influenza A HA and whether it blocks receptor binding. With roughly 5,000 labeled pairs across 188 antibodies and 79 HAs, the fine-tuned model reaches AUROC ≥ 0.91 for held-out pairs of known antibodies, 0.90 for known antibodies tested against unseen HA sequences, and 0.73 for entirely new antibodies, falling to 0.63–0.66 when the new antibody is dissimilar to anything in the training set. If this holds, antibody candidates could be prioritized in silico before wet-lab testing, and existing antibody panels could be screened against emerging viral variants. The paper also reports that breadth-of-protection scores derived from the model correlate with measured breadth, especially for H3 hemagglutination inhibition.","feed_headline":"Sequence-only AI predicts flu antibody activity at 0.9 AUROC","feed_subtitle":"Known antibodies hit 0.9 even on unseen flu strains; new ones drop to 0.73, and 0.66 for dissimilar candidates.","key_machinery":"The load-bearing object is MAMMAL, a transformer-based biomedical language model pretrained on protein, antibody, and interaction data, paired with its AbAg Bind input format: the antibody heavy-chain and light-chain variable-region sequences and the HA antigen sequence are fed as a single prompt, and the model is fine-tuned with a classification head to output a binding or HAI activity score. The evaluation machinery is the four-way split design: lenient random pairs, HA-exclusive folds, antibody-exclusive folds, and antibody-cluster-exclusive folds built by clustering at 50% sequence identity. That split design is what lets the authors separate 'can the model fill in missing measurements' from 'can it extrapolate to new molecules.'","core_discovery":"The central claim is that a sequence-only classifier, built by fine-tuning the pretrained biomedical language model MAMMAL on antibody heavy- and light-chain variable regions paired with full HA sequences, accurately separates active from inactive antibody–HA pairs in the assay-defined sense. The authors report that this transfer-learning approach outperforms random weight initialization across nearly every metric comparison, with the largest gaps for binding activity (AUROC 0.92 vs 0.62 in the lenient split) and for HAI. They interpret the performance gradient across splits—lenient, HA-exclusive, mAb-exclusive, mAb-cluster-exclusive—as evidence that the model generalizes to new viral strains but only moderately to new antibodies. They further use aggregated validation scores to rank antibodies by breadth of protection, with Pearson correlations up to 0.49.","pith_inferences":["The same recipe is likely portable to other viral antigens with small assay panels, such as RSV or SARS-CoV-2; the binding constraint will be label diversity, not model capacity.","The monotone drop from lenient to mAb-cluster-exclusive suggests a quantifiable scaling law: expected AUROC for novel antibodies should track the number and diversity of antibody clusters in the training set, so adding one representative antibody per cluster could be tested as a deliberate data-acquisition strategy.","Because 21% of the antigens here are computationally optimized COBRA designs, the 0.90 AUROC on unseen HAs should be revalidated against genuinely drifted field strains before relying on it for variant surveillance.","A practical screening pipeline could combine this classifier with a diversity filter: candidates within existing clusters can be ranked confidently, while cluster-novel candidates need wet-lab confirmation, mirroring the 0.73 versus 0.63–0.66 gap."],"forward_implications":["For panels of already characterized antibodies, the model can impute untested mAb–HA pairs with AUROC ≥ 0.91, reducing the number of wet-lab assays needed to complete strain-by-antibody matrices.","Screening existing antibody panels against a newly emerged HA sequence is predicted to work at AUROC ≈ 0.90, giving strain-surveillance efforts a fast sequence-only triage step.","For de novo antibody candidates, the model's AUROC of 0.73 (0.63–0.66 for dissimilar antibodies) means it should be used as a coarse filter, not a final arbiter, in antibody discovery.","Aggregated model scores rank antibodies by measured breadth of protection with Pearson correlations up to 0.49, so the same model can help select broadly protective leads, not just single-pair positives.","Fine-tuning MAMMAL beats random initialization on nearly all metric comparisons, and the gains are largest on the easier splits, confirming that pretrained protein-language representations carry useful signal even with roughly 5,000 labeled pairs."],"supporting_citations":[{"why":"Defines the MAMMAL framework, the pretrained weights, and the AbAg Bind prompt syntax used for all fine-tuning experiments.","marker":"[50]"},{"why":"Supplies the 188 human and mouse anti-HA monoclonal antibodies and the assay data used to build the binding and HAI datasets.","marker":"[7–18,65–67]"},{"why":"Provides the clustering method used to construct the mAb-cluster-exclusive split at 50% sequence identity.","marker":"[81]"},{"why":"Supplies the precision-recall analysis that justifies AUPRC as the more informative metric for the imbalanced HAI dataset.","marker":"[86]"}],"fun_headline_variants":["Sequence-only AI predicts flu antibody activity, AUROC 0.9","MAMMAL AI predicts flu antibody function from sequence alone","AI predicts flu antibody activity at 0.9 AUROC, drops to 0.73 for novel","Sequence-based model forecasts flu antibody activity with 0.9 AUROC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire accuracy story rests on the binary labels: binding is called positive when AUC-ELISA exceeds 1, and HAI is called positive when the minimum inhibitory concentration is below 10 µg/mL, and those hand-chosen cutoffs come mostly from the authors' own prior assays.","fun_headline_variants_meta":{"raw":{"variants":["Sequence-only AI predicts flu antibody activity, AUROC 0.9","MAMMAL AI predicts flu antibody function from sequence alone","AI predicts flu antibody activity at 0.9 AUROC, drops to 0.73 for novel","Sequence-based model forecasts flu antibody activity with 0.9 AUROC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00072,"raw_usage":{"total_tokens":3270,"prompt_tokens":1019,"completion_tokens":2251,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":2167}},"tokens_in":635,"tokens_out":2251,"duration_ms":17527,"temperature":1.0,"reasoning_tokens":2167,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:02:01.015554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same fine-tuning procedure on an independently generated antibody–HA dataset with harmonized assay protocols, and check whether AUROC on an antibody-exclusive split stays near 0.73; alternatively, retrain after shifting the positive cutoffs (for example AUC-ELISA > 5 or HAI MIC < 1 µg/mL) and see whether the ranking of model variants reverses.","supporting_citations":[],"review_version":1}