{"id":"90710c74-be9e-4648-8826-ddd4aded0c22","arxiv_id":"2505.02022","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"NbBench provides the first unified nanobody benchmark and shows that antibody language models lead on binding tasks while affinity and stability regressions remain hard for all tested models.","lead":"NbBench is a new benchmark for nanobody language models, spanning eight tasks from binding prediction to stability estimation. It evaluates eleven frozen protein and antibody language models and finds that no single model wins across tasks, while regression tasks stay difficult.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Split-similarity and pretraining-overlap reporting are internally inconsistent, so the Table 3 model rankings may be leakage-driven until rechecked.","rationale":"The paper is a genuinely useful benchmark artifact: it assembles eight nanobody tasks, provides public code and data, evaluates eleven frozen models with three seeds, and reports full metric tables. Those are real contributions, and the central claim that NbBench is the first comprehensive nanobody benchmark is plausible and not undermined by any internal inconsistency I found in the task inventory itself. However, the empirical conclusions about which model families excel on which tasks depend entirely on the split fairness. The manuscript contains an internal contradiction: Sec. 3.4 mandates 70% MMseqs2 clustering, while Appendix A.2 reports average train-test similarities of 73-84%, and Sec. 3.1 states a 75% average similarity target. If the split truly prevents cross-split pairs above 70%, the reported averages are hard to explain; if the averages are correct, the clustering guarantee is looser than stated. The reviewer identified the same zone of concern, and I agree with the conditional verdict. The affinity task description also conflicts (Sec. 3.3 defines regression via Eq. 1, while Appendix A.2 mentions binary classification with score thresholds), which is a secondary reproducibility defect worth fixing, but the leakage question is more load-bearing because it threatens the validity of the model rankings themselves. The proposed test is feasible without retraining any backbone, since all models are frozen; it only requires comparing benchmark test sequences against pretraining corpora and recomputing the classifier metrics on the filtered subset. If the rankings survive that test, CONDITIONAL is appropriately resolved; if not, the central empirical claims would need substantial revision.","tokens_in":22898,"tokens_out":5254,"duration_ms":59593,"concrete_test":"Recompute Table 3 on a leakage-controlled subset: for each of the 12 tasks and each baseline model, run MMseqs2 easy-search at 70% sequence identity between every test sequence and every sequence in that model's pretraining corpus (INDI, VHHCorpus-2M, OAS, UniRef50/100, etc.), and between test and training splits; then retain only test examples with no 70%-identity hit in the model's pretraining set or in the task training split, and re-rank models. If NanoBERT's VRCls/CDRInf gains and VHHBERT's SARS-2 result shrink toward the other baselines, the current rankings are leakage-driven. As a minimal accompanying check, reconcile Appendix A.2's reported 73-84% train-test similarity with the Sec. 3.4 claim of a 70% clustering threshold by reporting actual average and maximum cross-split identities for every task.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing assumption for the central comparison claim is that the clustering split (Sec. 3.4) actually prevents train/test leakage. That assumption is internally contradicted. Sec. 3.4 says MMseqs2 at 70% identity assigns whole clusters to splits and minimizes data leakage, yet Appendix A.2 reports average train-test similarities of 73-84% for all tasks, and Sec. 3.1 states VRCls 'ensure[s] the average sequence similarity between training and test sets remains at 75%'. If the highest cross-split pairwise identity were below 70%, the average could not exceed 70% by that much; the numbers imply either the clustering threshold was not applied as stated or the similarity metric was computed differently than described. Either way, the 'low redundancy' guarantee is not established. This matters because at least two baselines have pretraining corpora that overlap the benchmark sources: NanoBERT was pretrained on INDI and is evaluated on INDI-derived VRCls/CDRInf tasks, and VHHBERT was pretrained on VHHCorpus-2M and is evaluated on the AVIDa-SARS-CoV-2 dataset from the same paper. Since all models are frozen, a model that already encoded test-distribution sequences during pretraining can produce better linear-probe results without better transfer. The headline empirical conclusions, including 'antibody LMs excel on antigen-related tasks' and 'regression remains challenging', are rankings across these models; if leakage inflates any Table 3 entries, those conclusions are unreliable. The paper does not disclose overlap between each model's pretraining corpus and each benchmark test set, so the bias is unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"NbBench is introduced as the first comprehensive benchmark suite for nanobody representation learning. The paper curates nine datasets into eight tasks spanning structure annotation (VRCls, CDRInf), binding prediction (SARS-CoV-2 and hIL6 antigen binding, paratope prediction), and biophysical properties (thermostability, polyreactivity, nanobody type, VHH affinity). Eleven pretrained protein, antibody, and nanobody language models are evaluated in a frozen setting with a fixed classifier head, using three random seeds. The main empirical findings are that antibody LMs dominate antigen-related tasks, regression tasks such as thermostability and affinity remain difficult for all models, and no single model wins across all tasks. The authors release code and data publicly.","tokens_in":23233,"tokens_out":4479,"duration_ms":45120,"significance":"If the benchmark construction and the comparison protocol are sound, NbBench would be a useful community resource: it standardizes task definitions, splits, and metrics for a domain that currently lacks a unified evaluation, and it provides a reproducible head-to-head comparison of eleven publicly available models. The paper also ships public code and datasets, which is a concrete strength. The findings about task difficulty and the absence of a universal winner are plausible and practically relevant. However, as detailed below, the validity of the comparison is currently undermined by an internal inconsistency in the redundancy control for the splits, by a contradictory description of the affinity task, and by unaddressed pretraining-overlap risks. These issues are load-bearing for the headline empirical claims and need to be fixed before the results can be trusted.","major_comments":[{"comment":"The stated redundancy control is internally inconsistent. Section 3.4 says MMseqs2 clustering at a 70% sequence identity threshold assigns whole clusters to splits, which would imply that no cross-split sequence pair has identity above 70%. Yet Appendix A.2 reports that all tasks have average train-test similarities between 73% and 84%, and Section 3.1 says VRCls 'ensure[s] the average sequence similarity between training and test sets remains at 75%.' If the average pairwise identity exceeds 70%, then the 70% clustering threshold cannot have been enforced as described; conversely, if the clustering was enforced, the reported 73-84% figures must use a different definition of 'similarity' than the one implied by the clustering threshold. The paper must state exactly how the similarity was computed and report the actual distribution of cross-split pairwise identities. Because every ranking in Table 3 depends on the low-redundancy guarantee, this inconsistency needs to be resolved before the model comparison can be considered reliable.","section":"Section 3.4, Appendix A.2, Section 3.1"},{"comment":"The affinity task definition is contradictory. Section 3.3 and Table 1 describe VHH Affinity Prediction as a regression task with the continuous score defined in Eq. (1), and Table 9 reports Spearman correlation, R2, RMSE, and MAE for Affinity-seq and Affinity-score. However, Appendix A.2 states: 'For affinity prediction, only sequences with binding scores ≥ 5 or ≤ 1 were retained for binary classification.' This describes a binary classification setup on a filtered subset, not the regression setup of Section 3.3. If the regression experiments were trained on the full score range, the appendix description is wrong; if a binary-filtered subset was used, then the regression metrics in Table 9 are not interpretable as stated, and the conclusion that 'regression remains hard' for affinity is not supported by the described data. Please clarify the exact filtering, the response variable used, and how Affinity-seq differs from Affinity-score.","section":"Section 3.3, Table 1, Appendix A.2, Table 9"},{"comment":"The fairness of the model comparison is not established because some baselines appear to have been pretrained on the same source databases used to construct the benchmark tasks. Specifically, NanoBERT was pretrained on INDI (Table 2) and is evaluated on VRCls/CDRInf tasks built from INDI, while VHHBERT was pretrained on VHHCorpus-2M and is evaluated on the AVIDa-SARS-CoV-2 dataset introduced in the same paper as VHHBERT. Since all models are frozen, a model that encountered test-distribution sequences during pretraining can produce better linear-probe results without actually displaying better transfer. The paper should quantify the sequence-level overlap between each task's test split and each model's pretraining corpus (for example, by an MMseqs2 search), or otherwise demonstrate that the rankings in Table 3 are not driven by pretraining contamination. Without this, the headline conclusion that 'antibody LMs excel on antigen-related tasks' is at risk.","section":"Section 4, Table 2, Table 3"}],"minor_comments":[{"comment":"The text says the models were evaluated on 'the twelve tasks in NbBench,' but Table 1 lists eight tasks (with two binding datasets and two regression variants each). Please align the task count with the actual benchmark structure.","section":"Section 5.2"},{"comment":"The text contains unresolved references: 'Table ?? provides a comprehensive overview' and 'Table 9, which has been reused for consistency.' Please update these to the correct table numbers and ensure that each table is cited where it is discussed.","section":"Appendix A.5"},{"comment":"The suffixes '-seq', '-tm', and '-score' are introduced as indicating sequence-clustering splits versus stratified splits, but the description is ambiguous for Affinity-seq and Affinity-score. Please state explicitly which split type applies to each of the four regression-variant tasks.","section":"Section 3.4, Table 1"},{"comment":"The definition of BLOSUM62 Recovery says 'scores are normalized if needed for comparability,' but the reported values in Table 7 are not accompanied by a statement of which normalization was applied. Please specify the normalization or say that raw BLOSUM62 scores are reported.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark artifact is potentially valuable, and the public release of data and code is a strength. However, the split-similarity inconsistency is severe enough that any conclusion drawn from Table 3 is currently in doubt until the authors re-run or re-report the similarity controls. The pretraining-overlap concern is also real, although it may be partially mitigated by the fact that NanoBERT and VHHBERT do not dominate the leaderboard. I believe the issues are fixable within the scope of a revision, so major revision is appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading, and worth sending to review, but the empirical claims in Table 3 need to be treated as provisional. NbBench is the first unified nanobody benchmark I know of: eight tasks, nine datasets, eleven frozen models, standardized splits, and public code and data. That alone is a real service to the subfield, comparable to what TAPE and PEER did for proteins. The related-work survey is solid, and the task coverage—structure annotation, binding, developability—is well chosen. Credit where earned.\n\nThe soft spots are all in the reporting and the leakage controls. First, the affinity task is defined as regression in Section 3.3 and Table 1, but Appendix A.2 says sequences with binding scores ≥5 or ≤1 were retained for binary classification. That is a plain internal contradiction. Second, and more load-bearing, the split-similarity numbers do not cohere. Section 3.4 says MMseqs2 at 70% identity assigns whole clusters to splits, yet Appendix A.2 reports average train-test similarity of 73–84% across tasks, and Section 3.1 says VRCls keeps the average at 75%. If the highest cross-split pairwise identity were under 70%, the average could not exceed 70%. Either the threshold was not actually applied that way, or the similarity metric is being computed differently than the text implies. The paper needs to say which. Third, the pretraining/test overlap is not disclosed. NanoBERT was pretrained on INDI and is evaluated on INDI-derived tasks; VHHBERT was pretrained on VHHCorpus-2M and evaluated on the AVIDa-SARS-CoV-2 dataset from the same paper. With frozen backbones, a model that already encoded the test distribution can do better at linear-probe time without better transfer. This makes the headline conclusions—antibody LMs excel on antigen tasks, regression remains hard—subject to an unquantified bias.\n\nNone of this sinks the benchmark artifact. The resource is still useful even if some rankings shift after re-evaluation. But the internal inconsistency in the similarity numbers and the affinity definition are exactly the kind of thing a referee should catch before publication, so the paper needs a careful revision, not a desk rejection.\n\nThe paper deserves a serious referee. I would send it out with a request for a leakage audit and clarification of the split protocol. If you work on antibody or nanobody language models, cite it for the resource; just don't quote the current Table 3 as gospel.","headline":"A genuinely useful first nanobody benchmark whose model rankings are provisional until the authors fix an internal split-similarity contradiction, an affinity-task definition mix-up, and undisclosed pretraining overlap.","tokens_in":23754,"tokens_out":1988,"would_cite":true,"duration_ms":22975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NbBench aims to be the first standardized nanobody benchmark, and its frozen evaluation shows no single language model dominates across tasks.","keywords":["NbBench","nanobodies","language models","benchmark","binding prediction","thermostability","developability","representation learning"],"falsifier":"Compare each model's pretraining corpus against the test sequences of each NbBench task and remove any test sequence whose closest pretraining neighbor is above 70% identity, then rerun the eleven-model evaluation; if the reported task rankings change, the leakage controls are carrying the conclusion.","tokens_in":22718,"feed_emoji":"🧬","tokens_out":7648,"duration_ms":71247,"temperature":0.7,"pith_summary":"The paper introduces NbBench, a benchmark suite that aims to be the first standardized way to evaluate language models on nanobody tasks. It assembles eight tasks from nine datasets spanning structure annotation, binding prediction, and developability, and runs eleven pretrained protein, antibody, and nanobody language models through a frozen evaluation protocol. The central finding is that no single model wins across all tasks: antibody-specific models lead on antigen-binding and paratope predictions, structure-annotation tasks are effectively saturated, and regression tasks such as thermostability and affinity remain difficult for every model. The authors argue that NbBench matters because it replaces inconsistent single-task evaluations with a reproducible common protocol for nanobody modeling.","feed_headline":"First nanobody benchmark finds no single model winner","feed_subtitle":"Eight tasks, eleven frozen models: structure is solved, binding is led by antibody LMs, regressions remain hard.","key_machinery":"The central object is NbBench itself: eight tasks across nine curated datasets, with clustering-based splits built by MMseqs2 at 70% sequence identity so that similar sequences stay in the same split. Evaluation is frozen: pretrained model weights stay fixed and only a task-specific MLP head is trained, which isolates the quality of the representations. For nanobody-antigen tasks, antigen embeddings are always produced by the same ESM-2 (650M) model and concatenated with the nanobody embedding, so differences between models reflect how well each nanobody encoder represents the variable domain. This shared pipeline is what lets the paper attribute observed performance gaps to the pretraining domain rather than to task formatting.","core_discovery":"NbBench claims to be the first comprehensive benchmark for nanobody modeling, and its evaluation of eleven frozen models maps out where current representations succeed and fail. Variable-region classification and nanobody-type classification are essentially solved at >98% accuracy. Antibody language models trained on large immunoglobulin corpora transfer best to antigen-related tasks, with AntiBERTa2-CSSP reaching 0.92 AUROC on hIL6 binding and 0.94 AUROC on paratope prediction. Regression on thermostability is modest at best (Spearman $\\rho \\approx 0.59$) and affinity prediction stays below 0.20 Spearman. The paper concludes that no universal model exists for nanobody tasks, and that future progress will require domain-matched pretraining combined with structure-aware or parameter-efficient fine-tuning.","pith_inferences":["Beyond the paper: the reported rankings depend on the leakage controls; if a model was pretrained on the same source database as its test set, its apparent edge could shrink once near-duplicate test sequences are removed.","Beyond the paper: fixing the antigen encoder to ESM-2 (650M) means binding-task results confound nanobody representations with that single antigen representation; ablating or varying the antigen encoder would show how much of the ranking is really about the nanobody model.","Beyond the paper: the paper's own limitation note implies that adding structure-aware tasks as nanobody structure databases grow is a natural next step; a plausible testable expectation is that paratope and binding rankings will shift once structural tokens are included."],"forward_implications":["Any new nanobody language model can be compared against eleven baselines on identical splits and metrics, so a reported improvement is attributable to the model rather than the evaluation protocol.","Structure-annotation tasks (VRCls and NbType) are saturated, so they should be treated as completed sanity checks rather than discriminative benchmarks.","For antigen-binding and paratope tasks, antibody-specific models such as AntiBERTa2 provide the strongest starting point among the tested frozen backbones.","Thermostability and affinity regressions are not solved by frozen sequence embeddings; the paper points to structure-aware or parameter-efficient fine-tuning as the next direction.","Because no model wins across all tasks, model selection for real nanobody engineering should be task-dependent."],"supporting_citations":[{"why":"Provides the nanobody sequences and labels for VRCls and CDRInf, and is also the pretraining corpus of NanoBERT, making split control here decisive.","marker":"[14]"},{"why":"Supplies the SARS-CoV-2 binding dataset and the VHHCorpus-2M pretraining data for VHHBERT.","marker":"[15]"},{"why":"Supplies the large-scale VHH-antigen pairs for the hIL6 binding task.","marker":"[48]"},{"why":"Provides the MMseqs2 70%-identity clustering algorithm used for train/validation/test splits and similarity checks.","marker":"[57]"},{"why":"Provides the general protein backbones and the fixed antigen encoder used for nanobody-antigen tasks.","marker":"[16]"},{"why":"Provides AntiBERTa2 and AntiBERTa2-CSSP, the top-scoring models on several antigen-related tasks.","marker":"[24]"},{"why":"Provides the deep-sequencing biopanning data used to construct the affinity regression task.","marker":"[56]"},{"why":"Provides nanobody melting-temperature measurements used in the thermostability regression task.","marker":"[49]"},{"why":"Provides camelid and shark nanobody sequences used for the NbType classification task.","marker":"[54]"}],"fun_headline_variants":["First nanobody benchmark finds no universal model","Nanobody LM benchmark: 11 models, 8 tasks, no winner","Antibody LMs lead nanobody binding, regression still hard","NbBench: structure solved, binding led by antibody LMs","No solo champ in nanobody modeling across 8 tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the clustering-based splits keep test nanobodies genuinely unseen, so that no model has already memorized its test examples during pretraining.","fun_headline_variants_meta":{"raw":{"variants":["First nanobody benchmark finds no universal model","Nanobody LM benchmark: 11 models, 8 tasks, no winner","Antibody LMs lead nanobody binding, regression still hard","NbBench: structure solved, binding led by antibody LMs","No solo champ in nanobody modeling across 8 tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1443,"prompt_tokens":904,"completion_tokens":539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":454}},"tokens_in":520,"tokens_out":539,"duration_ms":5151,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:03:31.663564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare each model's pretraining corpus against the test sequences of each NbBench task and remove any test sequence whose closest pretraining neighbor is above 70% identity, then rerun the eleven-model evaluation; if the reported task rankings change, the leakage controls are carrying the conclusion.","supporting_citations":[{"cited_title":"Indi—integrated nanobody database for immunoinformatics","cited_arxiv_id":null,"evidence_quote":"Provides the nanobody sequences and labels for VRCls and CDRInf, and is also the pretraining corpus of NanoBERT, making split control here decisive."},{"cited_title":"A SARS-CoV-2 interaction dataset and VHH sequence corpus for antibody language models","cited_arxiv_id":null,"evidence_quote":"Supplies the SARS-CoV-2 binding dataset and the VHHCorpus-2M pretraining data for VHHBERT."},{"cited_title":"Wei, Zelda Mariet, Poomarin Phloyphisut, Hidetoshi Shimokawa, Joseph R","cited_arxiv_id":null,"evidence_quote":"Supplies the large-scale VHH-antigen pairs for the hIL6 binding task."},{"cited_title":"Enhancing antibody language models with structural information","cited_arxiv_id":null,"evidence_quote":"Provides AntiBERTa2 and AntiBERTa2-CSSP, the top-scoring models on several antigen-related tasks."},{"cited_title":"Extensive antibody search with whole spectrum black-box optimization","cited_arxiv_id":null,"evidence_quote":"Provides the deep-sequencing biopanning data used to construct the affinity regression task."},{"cited_title":"Nbthermo: a new thermostability database for nanobodies","cited_arxiv_id":null,"evidence_quote":"Provides nanobody melting-temperature measurements used in the thermostability regression task."},{"cited_title":"Plabdab-nano: a database of camelid and shark nanobodies from patents and literature","cited_arxiv_id":null,"evidence_quote":"Provides camelid and shark nanobody sequences used for the NbType classification task."}],"review_version":1}