{"id":"1978d67b-c922-4dc2-bb30-4bceb0085e67","arxiv_id":"2611.16361","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A 300M-score PDB-wide mutation atlas combined with a disagreement-aware preference distillation method reaches 0.498 macro-averaged Spearman on ProteinGym without experimental labels.","lead":"This paper builds a protein-wide atlas of more than 300 million predicted single-mutation effects, combining energy, language-model, and inverse-folding scores across 86,479 protein chains. It then trains a model without experimental labels that treats disagreements between these sources as information, reporting the top zero-shot scores among the methods it compared on ProteinGym.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified","rationale":"The reader identified three concerns: (1) no error bars or significance tests, (2) λ selected on the evaluation set, (3) ambiguous ESM-IF-only ablation. These are valid points about presentation and rigor, but they do not constitute a load-bearing attack on the central claim. The paper's method is internally consistent, the ablation ladder is monotonic (hard 0.468 → soft 0.475 → disagreement-aware 0.498), and the λ sweep is disclosed in Table 2. The ESM-IF-only ablation is clearly labeled as a dataset ablation (Table 1: 'ESM-IF aug only'), not a method ablation; the contribution of FoldX is therefore isolated, albeit not perfectly. The performance differences are modest but the trend is clear. The paper's own limitations are stated. The central claim is that the method achieves the best overall performance among evaluated zero-shot baselines and naive fusion strategies; this is supported by the reported numbers. The lack of error bars is a weakness in presentation but not a logical flaw. The reader's verdict of CONDITIONAL is appropriate; no adjustment is needed. I partially agree with the reader because the concerns are real but not fatal.","tokens_in":16342,"tokens_out":627,"duration_ms":6662,"concrete_test":"Independently recompute the macro-averaged Spearman for the disagreement-aware (λ=0.2) variant on the 201 ProteinGym substitution datasets using the released code and check whether the overall value reproduces within ±0.005 of 0.498. If it does, the central claim holds; if it shifts by more than 0.01, the claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that disagreement-aware preference distillation from FoldX, ESM2, and ESM-IF achieves the best overall zero-shot performance on ProteinGym—is supported by a coherent method, a clear ablation ladder, and consistent results. The reader's weakest_assumption (lack of error bars, λ selection on the evaluation set, ambiguous ESM-IF-only ablation) are acknowledged limitations but do not invalidate the claim. The paper itself states that λ=0.2 is the best in a sweep (Table 2) and that the ESM-IF-only ablation is a dataset ablation, not a method ablation; these are disclosed. The performance differences (0.498 vs 0.487 for VespaG, vs 0.480 for ESM-IF-only) are modest but the trend across ablations is monotonic and the method is clearly described. No internal inconsistency or hidden assumption undermines the central argument. The paper's own limitations (b24-b26, b144, b154) are stated and not double-counted. The claim is conditional on the released artifacts verifying the numbers, but the argument as presented is sound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper constructs a large-scale PDB-wide mutation dataset with signals from FoldX, ESM2, and ESM-IF, analyzes cross-source disagreement, and proposes a disagreement-aware preference distillation method that achieves the best overall zero-shot performance on ProteinGym among compared methods.","tokens_in":16579,"tokens_out":1574,"duration_ms":10813,"significance":"If the results hold, the work provides a valuable resource (the PDB-wide mutation atlas) and a principled framework for integrating heterogeneous computational mutation signals without experimental labels. The analysis of cross-source disagreement is informative, and the method's performance is competitive. The release of dataset and code would support reproducibility.","major_comments":[{"comment":"The headline claim (best overall zero-shot performance) rests on a single macro-averaged Spearman value (0.498) with no error bars or significance tests. Given that the differences from VespaG (0.487) and the ESM-IF-only ablation (0.480) are modest, it is unclear whether the improvement is statistically meaningful. The authors should report confidence intervals or perform a statistical test (e.g., paired bootstrap across datasets) to support the claim that the disagreement-aware method is significantly better than the next best baseline.","section":"Table 1, §6.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The regularization weight λ is selected by sweeping over values (0, 0.01, 0.1, 0.2, 0.5, 1.0) and reporting the best overall Spearman on ProteinGym. This constitutes tuning on the evaluation set, which risks overfitting to the benchmark. The authors should clarify whether a held-out validation set was used for λ selection, or if the reported results are the best among a sweep on the test set. If the latter, the central claim is weakened and should be caveated accordingly.","section":"Table 2, §A","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The student model is initialized from ESM2-650M and fine-tuned on the PDB-wide mutation preferences. Since ESM2-650M is also one of the reference sources (ESM2), the student has direct access to ESM2's preferences through its initialization. This raises a potential circularity: the student may simply retain or refine ESM2's own predictions, and the improvement from disagreement-aware training could partly reflect better utilization of the ESM2 prior rather than genuine multi-source integration. The authors should discuss this and ideally include an ablation where the student is initialized from a different backbone (e.g., a randomly initialized model or a different PLM) to disentangle the effect of initialization from the distillation objective.","section":"§5.1, Eq. (2)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The ESM-IF-only ablation (ρ=0.480) is described as a dataset ablation that excludes FoldX supervision. However, the student is still initialized from ESM2-650M, which already provides ESM2-like signals. The comparison between the full method (0.498) and ESM-IF-only (0.480) is meant to show the benefit of adding FoldX, but the ESM-IF-only ablation still uses ESM2 initialization and ESM-IF as a reference source. It is not a clean ablation of FoldX contribution because the student already has ESM2 knowledge. The authors should clarify what exactly is ablated: is FoldX removed from the reference sources, or is the entire FoldX dataset removed? The text says 'ESM-IF aug only' which suggests only ESM-IF augmentation is used, but the student still has ESM2 initialization. This needs clarification.","section":"§6.2, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract states 'best overall performance among the evaluated zero-shot baselines and naive multi-source fusion strategies on ProteinGym.' The phrase 'naive multi-source fusion strategies' is vague; consider specifying what these strategies are (e.g., averaging, voting) for clarity.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The conversion of FoldX energy differences to logits via ℓ = -(E(i,a) - E(i,a_WT)) is a linear transformation. It might be helpful to note that this preserves ranking but changes scale, which is consistent with the subsequent use of pairwise preferences.","section":"§3.2, Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The normalization constant τ_k is defined as the average absolute pairwise margin over training pairs. It would be useful to specify whether this is computed once over the entire training set or per mini-batch, as this affects reproducibility.","section":"§5.2, Eq. (5)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The agreement weight A_{i,a,b} is defined as |K^+_{i,a,b}| / |K|. Since |K|=3, this takes values in {0, 1/3, 2/3, 1}. This is a coarse discretization; the authors might discuss whether finer-grained agreement measures (e.g., based on margin magnitudes) could be beneficial.","section":"§5.3, Eq. (9)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The evaluation filters out ProteinGym datasets whose corresponding protein sequences contain fewer than 1000 amino acids. The rationale for this filter should be stated (e.g., computational constraints, relevance to the training data).","section":"§6.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The row for 'FoldX (reference)' reports 0.424 overall. However, FoldX is also used as a reference source in the proposed method. It would be informative to include a baseline where FoldX scores are used directly (without any student) to contextualize the improvement from distillation.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The right panel of Fig. 5 shows per-position Spearman correlation distributions. The text states that ESM2 and ESM-IF have higher ranking similarity, while FoldX has lower correlation with both. It would be helpful to report the median or mean correlation values for each pair to quantify this.","section":"§4.3, Fig. 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The entropy–disagreement analysis is interesting but the figure is small and the axes are not clearly labeled. Consider enlarging and adding axis labels for readability.","section":"Appendix B.6, Fig. 11","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper would benefit from a discussion of limitations, particularly regarding the reliance on FoldX for the physics-based signal (FoldX has known limitations in capturing certain mutation effects) and the restriction to single-site substitutions.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper builds a PDB-wide atlas of ~300M single-site mutation preference scores from FoldX, ESM2, and ESM-IF, aligned per residue. That resource alone is valuable—two orders of magnitude wider than the largest experimental DMS collections. The descriptive analysis of cross-source disagreement (Section 4) is thorough and genuinely informative: they show that FoldX gives a flatter, less conservative landscape, and that disagreement persists even at low-entropy positions, which supports their claim that the sources encode different inductive biases rather than just noise. The disagreement-aware distillation objective (Section 5) is well-motivated and the ablation ladder (hard 0.468 → soft 0.475 → disagreement-aware 0.498) is internally consistent and directionally clear. The λ=0 row exactly reproduces the soft-consensus row, as Eq. 14 predicts, which is a nice sanity check.\n\nThe soft spots are real but not fatal. First, the headline claim—best overall zero-shot performance on ProteinGym—rests on a single macro-averaged Spearman over 201 datasets with no error bars or significance tests. The gap to the next best method (VespaG, 0.487 vs 0.498) is modest, and without uncertainty estimates it is impossible to know whether the difference is meaningful. Second, λ=0.2 is selected by sweeping the same metric on the same benchmark (Table 2) with no stated validation procedure. The authors do not hide this—they report the sweep—but it means the reported 0.498 is an optimistically biased estimate. A proper validation split or a held-out tuning set would have addressed this. Third, the ESM-IF-only ablation is described as a dataset ablation, not a method ablation, so the contribution of FoldX is not cleanly isolated. The paper acknowledges this implicitly but the reader has to work to see it.\n\nThese are standard weaknesses for a resource+method paper at a top venue. They do not undermine the atlas or the method design; they just mean the performance claim should be read as \"comparable to state-of-the-art zero-shot methods\" rather than \"clearly best.\" The paper is for researchers working on mutation effect prediction, protein engineering, or multi-source distillation. It deserves a serious referee—the resource alone justifies it. I would recommend acceptance with a request to add error bars and clarify the λ selection procedure.","headline":"A solid resource paper with a real method contribution, but the headline performance claim is weaker than advertised because of missing error bars and a validation procedure that effectively tunes λ on the test set.","tokens_in":17219,"tokens_out":595,"duration_ms":6471},"pith_extraction":null,"created_at":"2026-11-13T06:26:10.756015+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}