{"id":"d4dc38dd-f04c-48ad-b2d3-3e98015423b4","arxiv_id":"2411.17795","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CrossDesign aligns pretrained protein language models with structure encoders to improve enzyme sequence design and zero-shot mutation fitness prediction.","lead":"This paper introduces CrossDesign, a protein design model that transfers knowledge from pretrained protein language models to structure-based sequence generation, targeting enzyme design where data is scarce. It reports gains over prior inverse folding models on CATH, enzyme datasets, and ProteinGym fitness benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot fitness claim depends on Eq. 8, a non-standard cross-entropy against the wildtype sequence that is never validated against the model's own mutant likelihood or a simple baseline; without such a control, the reported ProteinGym rank does not demonstrate fitness prediction.","rationale":"The reader's weakest assumption correctly identifies Eq. 8 as the load-bearing unvalidated component. My read confirms this: the model is trained with autoregressive objectives, yet the fitness scoring applies NAR decoding without additional training, and the score is defined as cross-entropy with the wildtype sequence rather than the probability of the mutant under the wildtype backbone. This is not a standard fitness proxy, and no ablation against the standard likelihood or a trivial baseline is provided. The paper's own limitations admit the lack of wet-lab validation, but the more immediate gap is the absence of computational validation of the scoring rule. The CATH inverse-folding results are more conventional and less affected by this issue, but the central claim of excelling in fitness prediction rests directly on this flawed scoring rule. The proposed concrete test would settle whether Eq. 8 is meaningful: if the standard likelihood or a BLOSUM baseline performs comparably, the fitness prediction claim is not unique to the model and does not support the task-adaptive generalization narrative.","tokens_in":9283,"tokens_out":9177,"duration_ms":81158,"concrete_test":"On ProteinGym, recompute the fitness score using the standard inverse-folding likelihood p(Y_mut | X_wild) = sum_i log softmax(logits_i)[Y_mut_i] for the same CrossDesign NAR model, and also compute a trivial baseline such as the BLOSUM62 substitution score between Y_mut and Y_wild. Compare the average matching rank (and Spearman rho) of these two scores with Eq. 8. If Eq. 8 does not outperform the trivial baseline or the model's own mutant likelihood, the claimed fitness prediction advantage is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of task-adaptive generalization includes mutation-effect prediction. In Section III.C.3, the score is Effectmut = CE(Prob_mut, Y_wild), where Prob_mut = p(logits | Decoder(Y_mut); IPS(X_wild)). This is the cross-entropy between the model's output distribution (conditioned on the mutant sequence and wildtype backbone) and the wildtype sequence itself. It is not the model's likelihood of the mutant sequence, which is the standard inverse-folding fitness proxy. The paper provides no theoretical or empirical justification that this 'reconstruction of wildtype from mutant' error correlates with fitness across all 217 ProteinGym assays. No per-assay Spearman correlations are reported, nor is there a comparison against standard alternatives such as ESM-IF's mutant log-likelihood or a simple substitution-score baseline. The paper's own limitation statement admits the lack of wet-lab validation, but it also omits computational validation of the scoring rule. Therefore, the matching rank of 0.445 may reflect sequence similarity or model artifacts rather than a genuine fitness signal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CrossDesign, an encoder-decoder protein inverse-folding framework that couples a structure-to-sequence stream with a frozen pretrained protein language model (PPLM) stream through cross-modal alignment, cross-layer consistency, and a shared decoder. The model supports autoregressive decoding for conditioning design and non-autoregressive decoding for mutation-effect scoring. The authors evaluate CrossDesign on CATH, Ts50/Ts500, enzyme-specific datasets (EnzPetDB and EnzFoldDB), and ProteinGym, reporting state-of-the-art perplexity and amino-acid recovery on CATH, improved out-of-domain recovery on Ts50/Ts500, AAR scores around 60% on enzyme folds, and a best matching rank of 0.445 on ProteinGym zero-shot fitness prediction. The paper also contributes curated enzyme structure-sequence benchmarks for PET hydrolases and fold-aware enzyme design.","tokens_in":9575,"tokens_out":4442,"duration_ms":41732,"significance":"If the claims hold, the paper is a useful contribution to low-resource enzyme design. The architectural idea of transferring sequence-level knowledge from a PPLM into a structure-conditioned inverse-folding model is coherent, and the evaluation strategy is laudably external: the headline CATH, Ts50/Ts500, enzyme, and ProteinGym results are not obtained by fitting the model to the test labels. The protein-design evaluations use standard benchmarks and the zero-shot fitness score is not trained on ProteinGym labels, which reduces circularity concerns. The enzyme benchmark collection is also a practical asset. However, the paper's central fitness-prediction claim rests on a non-standard scoring rule that is not validated against any baseline, the main quantitative claims lack error bars and statistical tests, and the absence of training details, code, and data prevents independent verification. These gaps are fixable within a revision, but they currently leave the strength of the claims overstated.","major_comments":[{"comment":"The zero-shot fitness score is not the model's likelihood of the mutant sequence, which is the standard inverse-folding proxy, but the cross-entropy between the model's mutant-conditioned output distribution and the wildtype sequence: Effectmut = CE(Prob_mut, Y_wild). The paper provides no theoretical justification, no per-assay Spearman correlations, and no comparison against standard alternatives such as ESM-IF's mutant log-likelihood or a simple substitution-score baseline. Without such controls, the reported ProteinGym rank of 0.445 does not demonstrate that the score captures fitness rather than sequence similarity or model artifacts. This is load-bearing because mutation-effect prediction is a stated headline contribution and the abstract claims the model 'excels in fitness prediction.' The limitation section, admitted lack of wet-lab validation, also omits this missing computational validation of the scoring rule itself.","section":"§III.C.3, Eq. (8)"},{"comment":"The paper reports 'best matching rank (0.445)' on ProteinGym, but never defines what 'matching rank' means, nor how it relates to the Spearman ρ values cited for VESPA (0.437) and ESM-IF (0.422). ProteinGym's standard evaluation metric is per-assay Spearman's rank correlation, averaged or median-aggregated; if 'matching rank' is a different aggregation, the comparison to baseline ρ values is not meaningful. Additionally, the introduction says ProteinGym spans 'over 178 standardized deep mutational scanning assays,' while Section III.C.3 says 'all 217 ProteinGym proteins'; the discrepancy needs to be resolved, as it changes the scope of the claim.","section":"§III.C.3 and abstract"},{"comment":"The headline CATH and Ts50/Ts500 results are reported as single numbers with no error bars, no repeated-seed variance, and no statistical significance tests, so the statement that CrossDesign outperforms ProteinMPNN, PiFold, and ESM-IF 'by a significant margin' is unsupported. This is especially important because the margins on some Ts500 recovery values are small (60.90 vs. 60.42 for PiFold). The paper also says 'no canonical training and validation sets exist for Ts50 or Ts500,' but does not specify which checkpoint or training protocol was used to produce the Ts50/Ts500 rows, which is needed to interpret those generalization claims.","section":"§III.B, Table I"},{"comment":"Reproducibility is compromised by omitted training details that are load-bearing for the method's identity. The loss weights for LexpCE, LCLC, LInterMA, and LasyAE are not given; the specific PPLM is not named; and architecture hyperparameters such as hidden dimension d, number of TGT layers, decoder layers, and number of attention heads are absent. Without these, a reader cannot tell whether the reported gains come from the cross-modal alignment, the exponential CE, the auxiliary reconstruction loss, or simply from scaling, and the claimed contribution cannot be reimplemented. The authors should provide the full training configuration, code, or at least a detailed hyperparameter table.","section":"§II, Eq. (2)–Eq. (5)"}],"minor_comments":[{"comment":"The summation in Eq. (2) starts at b=0, but batch indices conventionally run from 1 to B; please correct the range or explain the indexing.","section":"§II.3, Eq. (2)"},{"comment":"The caption contains stray text '输出aar_enzyme.pdf', which appears to be an artifact; it should be removed.","section":"Figure 2 caption"},{"comment":"The acronym AAR is used in tables and figures but is never defined; please spell out 'amino-acid recovery' at first use.","section":"§III.C, Figure 2"},{"comment":"EnzFoldDB is described as having 691 enzymes with 'each fold has an equal number of enzymes,' but 691 is not divisible by 10; the fold sizes in Figure 2 are 69 or 70, so the text should say 'approximately equal.'","section":"§III.A, Enzyme datasets"},{"comment":"The sentence 'encompasses a broad collection of over 178 standardized deep mutational scanning assays, spanning millions of mutated sequences' is inconsistent with the later '217 ProteinGym proteins'; please use a single consistent count and cite the exact ProteinGym version.","section":"§III.A, ProteinGym"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CrossDesign is a competent inverse folding model with genuinely good results on CATH and the enzyme benchmarks, but the zero-shot fitness result on ProteinGym is built on a scoring rule that the paper never validates. I'd send it to review, but the fitness section needs to be redone before publication.\n\nWhat's new and worth taking seriously: the architecture is a clean combination of a structure encoder (tGVP + TGT) with a PPLM auxiliary stream that is only used at training time, so inference is cheap. The CATH numbers are strong: perplexity 3.67, recovery 57.11%, better than ProteinMPNN and ESM-IF, and the Ts50 recovery above 60% is a nice first. The enzyme benchmarks (EnzPetDB, EnzFoldDB) are a useful contribution, and the AR/NAR task-adaptive design is a sensible way to handle both generation and mutation scoring. The authors are honest in the limitations section about the lack of wet-lab validation and small enzyme datasets.\n\nThe main soft spot is the ProteinGym claim. The scoring rule in Eq. 8 is cross-entropy of the model's output distribution (conditioned on the mutant sequence and wildtype backbone) against the wildtype sequence. That is not the mutant likelihood, and the paper gives no justification or comparison showing this proxy tracks fitness. The stress-test note is right: without a control against the model's own mutant log-likelihood or a simple substitution baseline, and without per-assay Spearman correlations, the reported matching rank of 0.445 could just reflect sequence similarity. This is a load-bearing problem for the 'zero-shot fitness prediction' part of the abstract, not a minor issue.\n\nOther concerns: no error bars or statistical tests anywhere, so the margin over VESPA (0.445 vs 0.437) and ESM-IF (0.422) might not be significant. Training details are thin—loss weights, temperature, PPLM variant, hyperparameters are missing. No code or data release, which makes it hard to reproduce the enzyme benchmarks. The novelty is incremental given the authors' own MMDesign and MetaEnzyme papers, though the enzyme-specific evaluation is a real extension.\n\nBottom line: this is a solid engineering paper that deserves a serious referee, but the fitness claim needs to be validated or dropped. If the scoring rule is fixed and the code released, I'd cite it; as is, I'd not rely on the ProteinGym numbers.","headline":"Solid inverse folding paper with strong enzyme benchmarks, but the ProteinGym zero-shot fitness claim rests on an unvalidated scoring rule.","tokens_in":10070,"tokens_out":3222,"would_cite":false,"duration_ms":28523,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By aligning protein structures with pretrained protein language model representations, CrossDesign transfers sequence knowledge into structure-based design, outperforming larger baselines on enzyme design and zero-shot mutation effect…","keywords":["inverse folding","protein design","enzyme design","protein language models","cross-modal alignment","zero-shot mutation effect prediction","low-resource learning","transfer learning"],"falsifier":"Compute the Spearman correlation between the NAR cross-entropy score and experimentally measured fitness for each of the 217 ProteinGym assays; if the average rank of CrossDesign falls below the ESM-IF baseline, or if a sizable fraction of assays show near-zero or negative correlation, the zero-shot fitness claim is refuted. Alternatively, test the scoring rule against a single designed enzyme mutant in the lab and compare predicted and measured activity.","tokens_in":9120,"feed_emoji":"🧬","tokens_out":8105,"duration_ms":62369,"temperature":0.7,"pith_summary":"The paper sets out to show that the data bottleneck in computational protein design can be broken by borrowing the knowledge already captured in pretrained protein language models. It presents CrossDesign, which aligns a structure-to-sequence model with a frozen language model during training, then discards the language model at inference. On the CATH benchmark the model reports a perplexity of 3.67 (a lower-is-better measure of sequence uncertainty) and 57.11% amino-acid recovery (the fraction of native residues reproduced), above ProteinMPNN, PiFold, and ESM-IF. On out-of-domain enzyme datasets its recovery reaches about 60%, and on ProteinGym it reports the best average matching rank among compared inverse folding and language models. The payoff, if the method is right, is a practical route to enzyme design when structure-sequence pairs are scarce.","feed_headline":"Protein language prior lifts low-resource enzyme design","feed_subtitle":"A cross-modal alignment lets small models beat ProteinMPNN and ESM-IF on enzymes.","key_machinery":"The load-bearing component is the asymmetric auto-encoder formed by the frozen pretrained protein language model (PPLM), the linear alignment unit (LAU), and the shared decoder in the auxiliary stream. During training the LAU maps PPLM output into the same dimension as the temporal graph transformer's alignment features, and the inter-modality alignment loss (InterMA) applies knowledge distillation between those two representations at temperature τ=8; a second distillation, cross-layer consistency (CLC), does the same between early geometric features and late contextual features in the main stream. This forces the structure encoder to internalize sequence-context knowledge without needing the PPLM at inference. The main stream's transformation-enhanced GVP (tGVP) makes the representation invariant to rigid-body transformations, and the shared triangular-mask decoder enables both autoregressive sampling for de novo sequence design and non-autoregressive parallel scoring for mutation effects.","core_discovery":"The central claim is that pretrained protein language models contain implicit structural and functional knowledge that can be transferred to an inverse folding model through an auxiliary supervised stream, rather than by fine-tuning the language model itself. CrossDesign contains a main structure-to-sequence stream built from transformation-enhanced geometric vector perceptrons, a temporal graph transformer, and a shared autoregressive decoder, plus an auxiliary stream that feeds the native sequence into a frozen PPLM and maps its representation back into the same feature space. Two knowledge-distillation losses—cross-layer consistency within the structure stream and inter-modality alignment between streams—push the structural features to mimic the language model's contextual features. The model is trained end-to-end with an exponential cross-entropy loss, and at inference only the structure stream runs. With non-autoregressive decoding, the same model scores mutant sequences against the wildtype structure, which the paper treats as a zero-shot fitness predictor.","pith_inferences":["If the cross-entropy scoring rule is taken at face value, the method should transfer to other property prediction tasks, such as antibody affinity or thermostability, where per-position sequence likelihood under a structure may correlate with function; this is not tested in the paper.","The alignment strategy suggests a general recipe: any frozen sequence-only model could be used to regularize a structure-based predictor in low-data regimes, not just protein language models for inverse folding.","A straightforward stress test would be to train CrossDesign on progressively smaller fractions of CATH to see whether the PPLM alignment preserves its advantage as data shrinks; the paper does not report such a curve.","The fitness predictor is uncalibrated; adding per-assay calibration or an ensemble over multiple PPLMs could raise or clarify the rank, but the paper does not explore it."],"forward_implications":["CrossDesign reports the best perplexity and recovery among the compared models on CATH, Ts50, Ts500, and the enzyme benchmarks, including the first reported recovery above 60% on Ts50.","The framework produces consistent out-of-domain enzyme design results, with fold-averaged recovery around 60% and up to about 64% on PET hydrolases.","Zero-shot mutation effect scoring with non-autoregressive decoding achieves an average matching rank of 0.445 on ProteinGym, ahead of compared language models and inverse folding baselines.","Training with the PPLM stream and distilling into the structure stream gives low-resource structure models an advantage over models trained on much larger structure-sequence corpora.","The shared decoder with AR and NAR modes lets one trained model serve both conditional design and mutation effect prediction without task-specific retraining."],"supporting_citations":[{"why":"Supplies the ProteinMPNN baseline that CrossDesign must beat on CATH and enzyme benchmarks.","marker":"[1]"},{"why":"Supplies the PiFold baseline, the strongest small-data inverse folding competitor in the comparison.","marker":"[3]"},{"why":"Supplies the ESM-IF baseline and the large CATH+AlphaFoldDB training regime that CrossDesign outperforms with fewer data.","marker":"[5]"},{"why":"Defines the CATH benchmark, its Short and Single-chain subsets, and the StrucTransformer baseline.","marker":"[10]"},{"why":"Provides the GVP geometric module and GVP-GNN baselines that tGVP builds on and compares against.","marker":"[11]"},{"why":"Pretrained protein language model whose sequence representations the auxiliary stream distills into the structure encoder.","marker":"[15]"},{"why":"Establishes language-model zero-shot mutation effect prediction, the approach CrossDesign's NAR scoring is contrasted with.","marker":"[17]"},{"why":"Provides ProteinGym, the benchmark and baseline scoring protocol for zero-shot fitness prediction.","marker":"[18]"}],"fun_headline_variants":["Protein language priors boost enzyme design with limited data","Distilling protein language knowledge into structure models for enzymes","Auxiliary stream transfers PPLM knowledge to enzyme inverse folding","CrossDesign uses frozen PPLM for low-resource enzyme design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The zero-shot fitness claim rests on treating the cross-entropy between the model's mutant-sequence probabilities under the wildtype structure and the wildtype sequence as a universal fitness proxy, with no per-assay calibration or wet-lab validation.","fun_headline_variants_meta":{"raw":{"variants":["Protein language priors boost enzyme design with limited data","Distilling protein language knowledge into structure models for enzymes","Auxiliary stream transfers PPLM knowledge to enzyme inverse folding","CrossDesign uses frozen PPLM for low-resource enzyme design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1290,"prompt_tokens":880,"completion_tokens":410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":341}},"tokens_in":496,"tokens_out":410,"duration_ms":4079,"temperature":1.0,"reasoning_tokens":341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:53:05.682933+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the Spearman correlation between the NAR cross-entropy score and experimentally measured fitness for each of the 217 ProteinGym assays; if the average rank of CrossDesign falls below the ESM-IF baseline, or if a sizable fraction of assays show near-zero or negative correlation, the zero-shot fitness claim is refuted. Alternatively, test the scoring rule against a single designed enzyme mutant in the lab and compare predicted and measured activity.","supporting_citations":[{"cited_title":"Learning inverse folding from millions of predicted structures","cited_arxiv_id":null,"evidence_quote":"Supplies the ESM-IF baseline and the large CATH+AlphaFoldDB training regime that CrossDesign outperforms with fewer data."},{"cited_title":"Generative models for graph-based protein design","cited_arxiv_id":null,"evidence_quote":"Defines the CATH benchmark, its Short and Single-chain subsets, and the StrucTransformer baseline."},{"cited_title":"Language models enable zero-shot prediction of the effects of mutations on protein function","cited_arxiv_id":null,"evidence_quote":"Establishes language-model zero-shot mutation effect prediction, the approach CrossDesign's NAR scoring is contrasted with."},{"cited_title":"Proteingym: Large-scale benchmarks for protein design and fitness prediction","cited_arxiv_id":null,"evidence_quote":"Provides ProteinGym, the benchmark and baseline scoring protocol for zero-shot fitness prediction."}],"review_version":1}