{"id":"e06b5e61-acc1-4977-9af7-c64330748b51","arxiv_id":"2505.11812","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VenusX is a new multi-task benchmark for fine-grained protein function prediction, testing models on residue-level, fragment-level, and pairwise similarity tasks across six annotation types.","lead":"The authors built VenusX, a large benchmark with over 878,000 protein samples annotated at the residue, fragment, and domain level for functional roles like active sites, binding sites, motifs, domains, and epitopes. It lets researchers test whether AI protein models understand function locally rather than just recognizing whole-protein patterns.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Residue-level tasks implicitly treat all unannotated residues as negatives (Sec. 3.1); if annotation coverage is incomplete, AUPR scores may measure annotation recovery rather than functional understanding, undercutting the benchmark's biological grounding.","rationale":"We agree with the reader that the negative-label assumption in Section 3.1 is the load-bearing threat to the central claim. If unannotated residues are not confidently non-functional, the benchmark measures recovery of known annotations rather than functional understanding, and the phrase 'biologically grounded' is overstated. This concern is broader than the reader's phrasing suggests because it affects every residue-level dataset, including the large Dom target, and it can distort cross-family gaps if annotation coverage differs by family. A secondary observation, which we did not use as the primary attack, is that Section 3.4 states clustering is applied before splitting for both strategies, yet Table 9's cross-family protein counts sum to the full dataset size (e.g., Act 7,701+880+1,086 = 9,667) while mix-family counts are far smaller (3,662), implying cross-family splits received no redundancy reduction. That inconsistency should be clarified but does not change our verdict. The benchmark remains a large and reusable resource, and the label-noise issue can be addressed by a validation study and explicit caveats, so we keep the reader's CONDITIONAL verdict rather than escalating to rejection.","tokens_in":24285,"tokens_out":15244,"duration_ms":156930,"concrete_test":"Sample 500 proteins per residue-level target (Act, BindI, BindB, Evo, Dom, Epi) and independently re-annotate VenusX-negative residues using complementary high-precision resources: Catalytic Site Atlas for active sites, PDBe/PDBsum ligand-contact residues for binding sites, and IEDB or a curated paratope/epitope set for epitopes. If more than 5% of sampled VenusX negatives are positive in the external resource, the negative-label assumption is violated materially. As a second check, drop annotation-dense proteins (top quartile by positive fraction) and re-run the Table 4 baseline ranking; if the ordering of ESM2-T33 and ANKH-BASE changes on a target such as Act or BindI, the headline conclusions are not robust to this label noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines residue-level binary classification, and the construction implicitly treats every unannotated residue as a negative. This negative-label assumption is not validated anywhere in the paper, yet it underpins all seven residue-level targets. For InterPro, a residue outside a signature match can still be catalytic or binding; for BioLiP, only residues contacting a co-crystallized ligand are positive, so alternative or condition-dependent binding sites are labeled negative; for SAbDab, epitope membership is set by a 10 Å Cα cutoff, so residues beyond it are negatives even if they participate in a larger conformational epitope. The resulting noise is not uniform: proteins with more structural and functional characterization have denser positive annotations, so a model can achieve high AUPR by predicting positivity wherever annotation density is high, without learning biology. Tables 4, 14, and 15 may therefore reflect recovery of known annotations, and the cross-family gaps may be inflated by annotation-coverage differences across families. Appendix A's generic limitation that the data 'may still contain biases' does not identify this structural assumption. Since residue-level tasks constitute the majority of the benchmark and all OOD residue results depend on them, the central claim that VenusX is 'biologically grounded' for fine-grained functional understanding is not yet fully supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"VenusX is a benchmark for fine-grained protein function understanding at residue, fragment, and protein-pair levels. It curates 7 residue-level binary classification tasks from InterPro, BioLiP, and SAbDab, 5 fragment-level multi-class classification tasks based on InterPro families, and 5 pairwise similarity scoring tasks, together forming 56 datasets with mixed-family and cross-family splits at three sequence identity thresholds. The authors evaluate a broad set of pretrained protein language models, sequence-structure hybrid models, structure-only networks, and alignment-based tools, reporting AUPR, accuracy/Macro-F1, and AUC. The central claims are that VenusX is the first large-scale and biologically grounded fine-grained functional benchmark and that strong performance on global protein-level tasks does not necessarily translate to fine-grained functional understanding.","tokens_in":24535,"tokens_out":7552,"duration_ms":79946,"significance":"If the task-construction concerns are resolved, VenusX would be a valuable resource for evaluating representation models on fine-grained functional annotations. The paper has clear strengths: the curation pipeline is described in detail, split statistics are reported transparently, the training setup is reproducible, and the code and data are publicly released. The baseline coverage is broad for residue-level and fragment-level classification, and the use of AUPR for imbalanced residue tasks is appropriate. The main limitations are that the residue-level labels implicitly treat all unannotated residues as negatives, and the pairwise 'functional similarity' task is defined by InterPro family membership rather than direct functional equivalence. These issues affect the interpretation of the majority of the reported results and the strength of the 'biologically grounded' claim.","major_comments":[{"comment":"Residue-level binary classification implicitly labels every residue not present in the curated annotation as negative. For InterPro, only residues matching annotated signatures are positive; for BioLiP, only residues contacting a co-crystallized ligand are positive; for SAbDab, only residues within the 10 Å Cα cutoff are positive. Because annotation coverage is incomplete and uneven across protein families and sequence regions, many true functional residues are treated as negatives, so AUPR can measure recovery of known annotations rather than biological function. The generic bias statement in Appendix A does not identify this structural assumption, which underpins all seven residue-level tasks and the corresponding cross-family conclusions. Please state this assumption explicitly and add robustness analyses, such as evaluating on high-confidence negatives derived from multiple databases or reporting results stratified by annotation density.","section":"Section 3.1, Tables 4 and 14-16"},{"comment":"The pairwise functional similarity task defines ground-truth positive pairs as proteins or fragments belonging to the same InterPro family. InterPro family membership is substantially determined by sequence and structural homology, so this task largely evaluates homology detection rather than direct functional equivalence. The near-perfect AUC values achieved by Foldseek and TM-align are therefore expected under this definition and do not by themselves demonstrate functional understanding. The paper's claim of 'function-based pairing' is stronger than the operational definition supports. Please either reframe this task as family-membership similarity or validate functional relatedness with an independent functional annotation such as shared Gene Ontology terms, and report the agreement between the two.","section":"Section 3.3, Table 6"},{"comment":"The benchmark description advertises comprehensive evaluation at three sequence identity thresholds and with both mixed-family and cross-family splits for residue-level and fragment-level tasks. The reported baseline experiments, however, cover only a subset: Table 4 reports MF50/MP50/Cross for five InterPro residue targets and MP70/MP90 only for BindP and Epi, while Table 5 and Appendix Table 17 report fragment-level results only for the MF50 split, with no fragment-level cross-family results appearing anywhere. The empirical support for the claims of out-of-distribution and multi-threshold evaluation should be completed, or the claims should be narrowed to match the reported experiments.","section":"Section 3.4, Tables 4-5 and 17"}],"minor_comments":[{"comment":"The heading contains a typo: 'Calssification' should be 'Classification'.","section":"Section 4.3 heading"},{"comment":"The heading 'Disscusion and Conclusion' should be 'Discussion and Conclusion'.","section":"Section 6 heading"},{"comment":"Table 2 reports 13,459 classes for the Dom fragment task, while Table 8 reports 12,529 fragment-level and 12,580 protein-level InterPro types for Dom. Please reconcile the discrepancy or explain the filtering difference.","section":"Tables 2 and 8"},{"comment":"The text states that the Dom cross-family AUPR drops by less than 10%, but the best MP50 value is about 0.665 and the best Cross value is about 0.564, which is roughly a 15% relative drop; please clarify whether the comparison is absolute or relative.","section":"Section 4.3"},{"comment":"The tables report precision, recall, and F1 for the positive class without stating the decision threshold used to convert model outputs to binary predictions; please specify the thresholding procedure.","section":"Table 14 and Appendix D"},{"comment":"The cross-family columns in Table 9 are labeled 'Family' and 'Protein', but the caption does not explain that the former counts InterPro families and the latter counts proteins; please make this explicit.","section":"Appendix B.3, Table 9"}],"recommendation":"major_revision","confidential_remarks":"This is a benchmark paper with a substantial resource contribution, but the 'biologically grounded' claim is currently stronger than the task definitions support. The negative-label assumption and the homology-based definition of pairwise functional similarity are fixable through additional analyses and reframing, so I do not recommend rejection. I would also ask the authors to verify the 'first large-scale' novelty claim against existing residue-level resources for binding site, active site, and epitope prediction in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing about VenusX. First, it is a genuinely useful, well-built benchmark resource: it turns existing residue-level annotations from InterPro, BioLiP, and SAbDab into a multi-task evaluation suite with careful splits and broad baseline coverage. Second, its central residue-level tasks rest on an unstated assumption—that every unannotated residue is non-functional—which the paper never validates and which materially shapes the reported numbers.\n\nWhat is actually new is the assembled package: 56 datasets spanning residue, fragment, and pairwise similarity tasks, with mix-family and cross-family splits at three sequence-identity thresholds. The curation is thorough: MMseqs2 clustering, redundancy removal, structural alignment, and a wide set of baselines including sequence LMs, structure-sequence hybrids, and alignment tools. Code and data are public, and the appendix is unusually complete. That is real value for the field.\n\nThe soft spots are real but concentrated. The 'unannotated = negative' assumption applies to all seven residue-level tasks. InterPro signatures are incomplete; BioLiP only sees residues contacting a co-crystallized ligand; SAbDab's 10 A Cα cutoff is generous but still a cutoff. If annotation coverage is correlated with how well a protein is studied, then AUPR can reward predicting annotation density rather than biological function, and the cross-family gaps may be inflated by coverage differences across families. The paper's limitations section is generic and does not flag this. That needs an explicit discussion and ideally a robustness check—for example, reporting results restricted to proteins with dense annotations, or at least a caveat on interpreting absolute AUPR.\n\nSecond, the pairwise similarity tasks define functional similarity as shared InterPro family. That makes the task substantially a remote-homology detection problem, which is fine, but it means strong Foldseek performance is expected and the benchmark's 'biologically grounded' framing should acknowledge the homology confound. Third, classification results come from a single seed, so we cannot judge whether a few AUPR points of difference between models is signal or noise. That is minor compared with the negative-label issue.\n\nNone of this undermines the resource. The central claim—that VenusX enables fine-grained functional evaluation—holds up, with the caveat that residue-level numbers should be read as 'recovery of known annotations' rather than direct measurements of all functional residues.\n\nI would send this to a serious referee, and I would expect a conditional acceptance after revision. The paper is for anyone building or evaluating protein representation models, and it will probably be a standard citation in that area.","headline":"A well-engineered fine-grained protein function benchmark whose residue-level numbers hinge on an unvalidated negative-label assumption; deserves peer review and a conditional revision.","tokens_in":25055,"tokens_out":2967,"would_cite":true,"duration_ms":30135,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VenusX, the first large-scale benchmark for fine-grained protein function, shows that strong whole-protein performance does not transfer to residue- and fragment-level functional tasks.","keywords":["protein function prediction","benchmark","residue-level annotation","fragment classification","pairwise similarity scoring","protein language models","structure-aware models","out-of-distribution generalization"],"falsifier":"Take a random sample of residues VenusX labels negative, check them against experimental mutagenesis or manually curated catalytic-site records, and see whether a nontrivial share are actually functional; if they are, the residue-level tasks reward conservative majority-class predictors rather than genuine functional-site recognition.","tokens_in":24099,"feed_emoji":"🧬","tokens_out":7237,"duration_ms":68999,"temperature":0.7,"pith_summary":"VenusX is proposed as the first large-scale benchmark for fine-grained protein functional understanding, spanning residue, fragment, and domain levels with 56 datasets and over 878,000 curated samples. The benchmark's design lets evaluators ask not just whether a model assigns the right global function label, but whether it can locate catalytic sites, binding sites, conserved positions, motifs, domains, and epitopes within a protein. Its baseline results support a central practical message: strong performance on conventional whole-protein tasks does not automatically transfer to fine-grained functional tasks, with cross-family residue prediction of active and binding sites dropping sharply. The authors argue this points to a need for models that capture precise local biological signals rather than global or family-level shortcuts.","feed_headline":"Benchmark: protein models miss fine-grained function","feed_subtitle":"VenusX's 56 datasets test residues, fragments, and domains; cross-family active-site scores collapse","key_machinery":"The central object is the benchmark's label construction: curated residue-level annotations from InterPro (active sites, binding sites, conserved sites, motifs, domains), BioLiP (ligand-binding residues from experimental complexes), and SAbDab (epitope residues from antibody-antigen complexes) are turned into binary residue labels, fragment-level family labels, and pair-level same-family labels. The machinery that makes these labels usable is the sequence-identity clustering with MMseqs2 at 50%, 70%, and 90% thresholds, the separate mixed-family and cross-family split protocols that control whether training and test proteins can share families, and the negative-label convention that unannotated residues count as non-functional. Pairwise scoring is driven by the subsampling of 10,000 positive and 10,000 negative pairs per seed, keeping evaluation tractable despite combinatorial pair counts in the hundreds of billions.","core_discovery":"The paper establishes VenusX as a biologically grounded evaluation platform built from residue-level annotations in InterPro, BioLiP, and SAbDab, processed into three task categories: residue-level binary classification of functional positions, fragment-level multi-class classification into InterPro families, and pairwise functional-similarity scoring without supervision. Across these categories the authors define 56 datasets, with mixed-family and cross-family splits at 50%, 70%, and 90% sequence identity and a pair-subsampling protocol that draws 10,000 positive and 10,000 negative pairs per seed. The documented findings are that protein language models reach high AUPR on in-distribution residue tasks, that sequence-structure hybrids such as SaProt and PROTSSN win most fragment-level and cross-family residue comparisons, that structure aligners such as Foldseek remain the strongest pairwise similarity baseline, and that epitope prediction stays uniformly weak. The paper's interpretive claim is that these patterns show current models often lean on global or distributional cues and do not yet reliably encode localized functional information.","pith_inferences":["Editorial inference: the cross-family AUPR collapse on active and binding sites could serve as a quantitative shortcut-dependence diagnostic, since a model that relies on family-level patterns will drop sharply when test families are unseen.","Editorial inference: if the negative-label assumption is incomplete, residue-level rankings may favor models that hedge toward the majority class; masking proteins with sparse annotations would test this.","Editorial inference: adding a temporal split—train on InterPro families released before a cutoff, test on families released after—would turn VenusX into a continual-learning benchmark for fine-grained knowledge accumulation.","Editorial inference: pair AUC over 10,000 positive and negative pairs mainly measures coarse family separation; stratifying negatives by family distance would probe whether models capture graded functional similarity."],"forward_implications":["Model rankings on fine-grained function will differ from global benchmarks: sequence-structure hybrids like SaProt-650M and PROTSSN lead fragment-level classification and several cross-family residue tasks.","Cross-family residue prediction of catalytic and binding sites is far from solved; on Act and BindI the best AUPR drops by roughly 70–80% compared with in-distribution settings.","Structure-based aligners remain the reference for pairwise functional similarity: Foldseek achieves near-perfect AUC on some tasks while BLAST trails by more than 40 points, so structure-aware representations matter for detecting functional relatives.","Evaluation of protein models should therefore include fine-grained, family-disjoint tasks; global accuracy alone is insufficient evidence of biological understanding.","Epitope residue prediction is a consistently weak spot, with no tested model exceeding 0.30 AUPR, marking a concrete gap for antibody and antigen modeling."],"supporting_citations":[{"why":"InterPro supplies residue-level active-site, binding-site, motif, and domain annotations that form the core of VenusX.","marker":"[24]"},{"why":"BioLiP provides experimentally derived ligand-binding site residues labeled by a van der Waals distance criterion.","marker":"[25]"},{"why":"SAbDab offers antibody-antigen complex structures from which epitope residues are derived by a 10 Å Cα distance threshold.","marker":"[26]"},{"why":"MMseqs2 performs the 50/70/90% sequence identity clustering that defines the benchmark's redundancy-reduced splits.","marker":"[44]"},{"why":"Foldseek serves as the structure-based alignment baseline for pairwise similarity scoring.","marker":"[40]"},{"why":"ESM2 provides the protein language model baselines evaluated across all three task categories.","marker":"[31]"},{"why":"SaProt provides the sequence-structure hybrid baseline that leads several fragment-level and cross-family residue tasks.","marker":"[33]"},{"why":"AlphaFold protein structure database supplies the predicted structures used to require structural coverage for curated entries.","marker":"[9]"},{"why":"UniProt supplies canonical protein sequences paired with the InterPro annotations.","marker":"[10]"}],"fun_headline_variants":["Fine-grained protein function: models stumble on active sites","VenusX: 56 datasets expose protein model blind spots","Protein AI: good at whole, weak at parts","Protein function: residue-level test stumps models","Cross-family active sites: protein models collapse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The residue-level tasks assume every unannotated residue is non-functional, so if the source databases miss real functional residues, those residues are scored as negative and every residue-level metric is affected.","fun_headline_variants_meta":{"raw":{"variants":["Fine-grained protein function: models stumble on active sites","VenusX: 56 datasets expose protein model blind spots","Protein AI: good at whole, weak at parts","Protein function: residue-level test stumps models","Cross-family active sites: protein models collapse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000766,"raw_usage":{"total_tokens":3429,"prompt_tokens":1012,"completion_tokens":2417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2342}},"tokens_in":628,"tokens_out":2417,"duration_ms":17789,"temperature":1.0,"reasoning_tokens":2342,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:46:49.795688+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of residues VenusX labels negative, check them against experimental mutagenesis or manually curated catalytic-site records, and see whether a nontrivial share are actually functional; if they are, the residue-level tasks reward conservative majority-class predictors rather than genuine functional-site recognition.","supporting_citations":[{"cited_title":"InterPro in 2022.Nucleic Acids Research, 51(D1):D418–D427, 2023","cited_arxiv_id":null,"evidence_quote":"InterPro supplies residue-level active-site, binding-site, motif, and domain annotations that form the core of VenusX."},{"cited_title":"BioLiP: a semi-manually curated database for biologically relevant ligand–protein interactions.Nucleic Acids Research, 41(D1):D1096– D1103, 2012","cited_arxiv_id":null,"evidence_quote":"BioLiP provides experimentally derived ligand-binding site residues labeled by a van der Waals distance criterion."},{"cited_title":"SAbDab: the structural antibody database.Nucleic Acids Research, 42(D1):D1140–D1146, 2014","cited_arxiv_id":null,"evidence_quote":"SAbDab offers antibody-antigen complex structures from which epitope residues are derived by a 10 Å Cα distance threshold."},{"cited_title":"Fast and accurate protein structure search with Foldseek.Nature Biotechnology, 42(2):243–246, 2024","cited_arxiv_id":null,"evidence_quote":"Foldseek serves as the structure-based alignment baseline for pairwise similarity scoring."},{"cited_title":"SaProt: protein language modeling with structure-aware vocabulary","cited_arxiv_id":null,"evidence_quote":"SaProt provides the sequence-structure hybrid baseline that leads several fragment-level and cross-family residue tasks."},{"cited_title":"Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models.Nucleic Acids Research, 50(D1):D439–D444, 2022","cited_arxiv_id":null,"evidence_quote":"AlphaFold protein structure database supplies the predicted structures used to require structural coverage for curated entries."},{"cited_title":"UniProt: the universal protein knowledgebase in 2025.Nucleic Acids Research, 53(D1):D609–D617, 2025","cited_arxiv_id":null,"evidence_quote":"UniProt supplies canonical protein sequences paired with the InterPro annotations."}],"review_version":1}