{"id":"021edfe5-78d8-410c-a317-a4e7cc0529e8","arxiv_id":"2505.12511","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DS-ProGen combines backbone and surface encoders with a pretrained autoregressive decoder and reports 61.47% recovery on PRIDE, though baseline fairness and subset selection weaken the SOTA claim.","lead":"DS-ProGen is a protein design model that reads both the backbone geometry and the molecular surface of a target structure and autoregressively generates an amino acid sequence. It reports a 61.47% recovery rate on PRIDE, but the state-of-the-art claim is weakened by missing baselines and a filtered evaluation subset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 61.47% SOTA claim depends on an unfair comparison: ESM-IF is not fine-tuned, iProFold is omitted from Table 1 despite appearing in the appendix, and the surface model's test subset is undefined. A controlled rerun is needed before the claim stands.","rationale":"Read in good faith, the architecture is coherent and the ablation shows both encoders contribute. But the empirical SOTA claim cannot be evaluated from the current text. The reader's weakest assumption identifies the same point: the comparison is not apples-to-apples. I agree. The concrete check is straightforward and should settle it. If the rerun survives, the paper is a solid conditional accept; if not, the headline claim should be withdrawn. Secondary inconsistencies: Table 4 says pretraining on 4M pairs while the text says 40M, and the surface-pooling description in Section 3.2 is underspecified; these do not change the main concern.","tokens_in":13561,"tokens_out":5924,"duration_ms":61002,"concrete_test":"Reproduce Table 1 with official PRIDE code and iProFold weights under a single protocol: fine-tune ESM-IF on the same 32,389 CATH training proteins, use the paper's decoding (temperature 0.1, top-k 10), evaluate on the full 504-protein test set, and report iProFold's recovery rate on that same set. Also report DS-ProGen's recovery separately for surface-valid and surface-invalid test proteins. If fine-tuned ESM-IF or iProFold reaches/exceeds 61.47%, or if the full-model number was computed only on the surface-valid subset, the SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that 61.47% recovery on PRIDE is state-of-the-art and demonstrates that fusing backbone and surface encoders helps. This claim requires the comparison to be equal on the same test set, with the same training data and decoding. Section 4.3 violates that requirement: ProteinMPNN and PiFold are retrained on the PRIDE training set, while ESM-IF is evaluated directly from released pretrained weights without fine-tuning. DS-ProGen, by contrast, is fine-tuned on the PRIDE training set and initialized from ProGen2-small plus ESM-IF. Fine-tuning ESM-IF on the same 32,389 proteins could plausibly close a large part of the 61.47 vs 51.85 gap; the paper gives no control experiment. Appendix Fig. 7 lists iProFold-B and iProFold in the RMSD comparison, but Table 1 omits their recovery rates without explanation. iProFold is a published PRIDE baseline; its absence makes the 'current state-of-the-art' label unverifiable. Section 4.1 also states that the surface model is 'trained only on samples with available surface information' and the surface encoder discards proteins with atom types outside C,N,O,S,Se,H. The paper never reports how many of the 504 test proteins satisfy this condition or whether Table 1 is computed on the full test set or the surface-valid subset. If the reported number is only on the easier subset, it is not comparable to baselines evaluated on the full test set. The Limitations section's admission of limited compute and unaccounted AlphaFold errors is secondary; the comparison protocol is the load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DS-ProGen, an autoregressive protein design model that combines a GVP-based backbone encoder (initialized from ESM-IF) with a surface point-cloud encoder and a ProGen2-small-initialized decoder. The central empirical claim is a 61.47% amino-acid recovery rate on the PRIDE benchmark, presented as state-of-the-art and as evidence that fusing backbone geometry with molecular surface features improves inverse protein folding. The paper also reports AlphaFold3-based structural fidelity metrics and qualitative ligand/ion design case studies.","tokens_in":13922,"tokens_out":5539,"duration_ms":52002,"significance":"If the empirical claims held under a fully controlled comparison, this would be a useful contribution: it would demonstrate that surface descriptors and backbone geometry can be fused in a language-model inverse-folding framework, with improved sequence recovery and plausible structural predictions. The architecture description is fairly detailed, and the data-leakage analysis in Appendix A.3 is a positive feature. However, the current evaluation does not support the state-of-the-art claim because the baseline comparison is not apples-to-apples: ESM-IF is not fine-tuned while other baselines are retrained, a published PRIDE baseline (iProFold) is absent from the main recovery table, and the test-set definition for the surface model is ambiguous.","major_comments":[{"comment":"The comparison against ESM-IF is not controlled. ProteinMPNN and PiFold are retrained on the PRIDE training set, while ESM-IF is evaluated from released pretrained weights without fine-tuning; DS-ProGen, in contrast, is initialized from ProGen2-small and ESM-IF weights and fine-tuned on the PRIDE training set. Because ESM-IF's backbone encoder and language-model prior are directly used as the starting point, the 9.62-point gap (51.85 vs. 61.47) cannot be attributed to the dual-structure architecture unless a fine-tuned ESM-IF baseline is provided under identical training data, initialization, and decoding. Please add this control.","section":"§4.3, Table 1"},{"comment":"The test-set definition for the surface model is ambiguous. Section 4.1 states that the surface encoder is trained only on samples with available surface information, and Section 3.2 states that proteins containing atom types outside C, N, O, S, Se, and H are discarded. The paper never reports how many of the 504 PRIDE test proteins have valid surface data, nor whether Table 1 is computed on the full test set or on a surface-valid subset, nor what happens at inference when surface data are unavailable. If the reported 61.47% is on an easier subset, it is not comparable with baselines evaluated on the full test set. Please report the exact test subset and re-evaluate all baselines on that subset.","section":"§4.1, Table 1"},{"comment":"The main recovery table omits iProFold and iProFold-B even though both appear in the RMSD comparison of Appendix Figure 7. iProFold is a published PRIDE baseline, so the abstract and conclusion's 'current state-of-the-art' label cannot be verified without its recovery rate in Table 1 (or a documented reason for exclusion). Please add these numbers or explicitly state why they are not comparable.","section":"Appendix Fig. 7, Table 1"},{"comment":"There is an inconsistency in the pretraining-dataset size: Section 4.1 says the large pretraining set contains 40 million structure-sequence pairs and the surface pretraining set contains 80,000, while Table 4's caption says 'Pretraining on 4M structure-sequence pairs.' Since the ablation claims a roughly 19% gain from pretraining, the correct dataset sizes must be stated and the caption reconciled.","section":"§4.1, Table 4"}],"minor_comments":[{"comment":"The decoding protocol (temperature, top-k) is specified for DS-ProGen but not for the baselines; please report whether the same sampling settings were used for all models.","section":"§4.3"},{"comment":"The text notes that DS-ProGen has the lowest RMSD but a lower mean TM-Score than PiFold and ESM-IF; the discussion is appropriately cautious, but the comparison would be strengthened by reporting confidence intervals or per-sample paired statistics.","section":"Table 2"},{"comment":"The ligand/ion interaction evaluation is qualitative; a quantitative metric (e.g., binding-pocket RMSD or interaction fingerprint similarity) would be needed to support the claim of 'functional retention.'","section":"Figures 4, 10–12"},{"comment":"The label 'DS-ProGen-B' is used in the figure while the main text uses 'backbone-only'; please define this abbreviation.","section":"Appendix Fig. 9"},{"comment":"The manuscript does not state whether code and trained models will be released, which is important for reproducibility of the reported architecture and benchmarks.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the uncontrolled baseline comparison. If the authors cannot supply a fine-tuned ESM-IF baseline and recovery numbers for iProFold, the state-of-the-art claim should be removed from the abstract and conclusion. The ambiguity about the surface-valid test subset is equally important, since it directly affects whether the reported 61.47% is comparable to any baseline number."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a real architectural step for inverse protein folding, but the headline SOTA number is not trustworthy as reported. The authors fuse a GVP backbone encoder with a surface point-cloud encoder, feed both into an autoregressive transformer decoder initialized from ProGen2-small. That combination is new and the idea is sensible. The paper does a lot of things right: the data-leakage analysis between train and test is unusually thorough, and the ablation shows both encoders matter (removing the surface encoder drops recovery from 61.47 to 50.71; removing backbone drops it to 31.06).\n\nThe soft spot is the comparison protocol. In Table 1, ProteinMPNN and PiFold are retrained on PRIDE, but ESM-IF is used as-is from pretrained weights. DS-ProGen is fine-tuned on PRIDE and also inherits ESM-IF's backbone encoder and ProGen2's decoder. That is not an equal test. The gap over ESM-IF (61.47 vs 51.85) could shrink a lot if ESM-IF were fine-tuned on the same 32k training proteins. The paper needs that control experiment before claiming SOTA. Also, iProFold appears in the appendix RMSD figure but its recovery rate is missing from Table 1, which makes the \"state of the art\" claim hard to verify.\n\nSecond issue: the surface encoder is trained only on proteins with valid surface data and limited atom types (C,N,O,S,Se,H). For the 504-protein test set, the paper never states how many pass that filter, nor whether Table 1 is on the full set or the filtered subset. If it's the latter, the comparison is on an easier subset and the number is not comparable to the baselines. That's a fixable but essential omission.\n\nThird, the functional validation (ligand/ion binding) is mostly qualitative visualization with ChimeraX. There are no quantitative binding or function metrics. The limitations section honestly admits compute limits and AlphaFold noise, but it doesn't acknowledge the baseline fairness issue, which is the load-bearing problem.\n\nWould I send it to review? Yes. It's a plausible new architecture, and the claim, if verified with a controlled rerun and proper baseline list, would be worth publishing. But as submitted, the central empirical claim is overreaching. The right referee would ask for: fine-tuned ESM-IF, iProFold recovery numbers, and an exact description of the test subset used for the surface model. I'd also want code release. It deserves a serious referee, but the verdict should hinge on those experiments.\n\nFor your purposes: this is a good paper to bring up in a reading group as a case study in benchmark hygiene, but I wouldn't cite the 61.47% number today.","headline":"A genuinely new dual-structure architecture for inverse protein folding, but the 61.47% SOTA claim is not supported by the experiments as written due to unfair baselines and an undefined test subset.","tokens_in":14494,"tokens_out":3463,"would_cite":true,"duration_ms":33367,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-structure language model for inverse protein folding recovers 61.47% of native amino acids by conditioning on both backbone geometry and molecular surface features.","keywords":["inverse protein folding","protein design","dual-structure encoding","backbone geometry","molecular surface features","autoregressive language model","sequence recovery","PRIDE benchmark"],"falsifier":"Re-run the benchmark with every baseline fine-tuned on the same PRIDE training data, decoded with the same temperature 0.1 and top-k 10, and include the missing published baseline; if a backbone-only baseline trained on the same surface-filtered subset reaches 61.47%, or the margin over the next-best method shrinks to noise, the dual-structure fusion is not the source of the reported gain.","tokens_in":13349,"feed_emoji":"🧬","tokens_out":10052,"duration_ms":94331,"temperature":0.7,"pith_summary":"The paper claims that inverse protein folding — choosing an amino acid sequence that folds into a given three-dimensional structure — is better solved when the structure is read twice: once through backbone coordinates and once through the molecular surface. To test this, it builds DS-ProGen, an autoregressive language model that sums a backbone-geometry embedding and a surface-chemistry embedding into a single structural condition and feeds that condition to every next-amino-acid prediction. On the PRIDE benchmark the model recovers 61.47% of native residues, and its ablations show both channels matter: dropping the surface encoder lowers recovery to 50.71%, while dropping the backbone encoder lowers it to 31.06%. The paper concludes that surface features and backbone features are complementary constraints, not redundant views, for sequence design.","feed_headline":"Protein design hits 61.47% by fusing fold and surface","feed_subtitle":"Combining backbone geometry with surface chemistry pushes native-sequence recovery past every single-view model on PRIDE.","key_machinery":"The central object is the dual-structure fusion embedding $R = B + S$, a simple sum of two structural embeddings that becomes the conditioning context for every amino-acid prediction. $B$ comes from a backbone encoder: N, C, and Cα coordinates, dihedral angles, pairwise distances, and local orientation vectors updated by four geometric-vector-perceptron layers and a transformer, producing one vector per residue. $S$ comes from a surface encoder: a point cloud of the solvent-accessible surface built with a smooth distance function and van der Waals radii, each point carrying atom-type one-hots, multiscale mean and Gaussian curvatures, and the chemical identity and distance of its 16 nearest atoms, patched, message-passed, and pooled to one vector per residue. The mechanism does its work through the additive fusion: the decoder never sees the modalities separately, so every next-token choice must satisfy global topology and local surface chemistry simultaneously.","core_discovery":"On the paper's own terms, the discovery is that fusing backbone and surface information into one structural prior improves inverse-folding accuracy beyond either modality alone. The model extracts a backbone embedding from N, C, and Cα coordinates using rotation-equivariant geometric vector perceptrons and a transformer, extracts a surface embedding from surface point clouds with atom types, multiscale curvatures, and local chemical neighborhoods, then forms the structural condition $R = B + S$ and prepends it to the sequence tokens of an autoregressive decoder. Trained on the PRIDE benchmark with pretraining on large structure-sequence collections, this dual-conditioned decoder reaches 61.47% average recovery and 63.50% on short chains, while generating sequences whose predicted structures stay close to the target fold. The paper presents this as evidence that internal fold and external chemistry jointly constrain the native sequence, and that a language-model decoder can exploit both.","pith_inferences":["A decisive test not in the paper: fine-tune a strong backbone-only model on exactly the same surface-filtered training subset and decode with the same temperature and top-k; if its recovery approaches 61.47%, the gain currently attributed to fusion is really due to training-subset differences.","The same dual-stream conditioning should transfer to neighboring tasks the paper only gestures at, such as RNA inverse folding or ligand-pocket redesign, where surface chemistry dominates the design constraints; a successful transfer would support the general mechanism rather than the specific benchmark.","Because the surface encoder is restricted to proteins containing only C, N, O, S, Se, and H atoms, the reported gain may partly reflect selection of chemically simple targets; extending the atom vocabulary to metals and modified residues would test whether the fusion idea survives on harder proteomes."],"forward_implications":["If the 61.47% figure holds in a fair comparison, inverse folding no longer requires choosing between backbone-only and surface-only designs; the two channels are complementary and can be combined additively.","The surface channel's largest contribution appears on short chains (63.50% versus 43.75% for the best single-modality baseline), suggesting surface chemistry is most informative where the backbone graph is too sparse for geometry-only encoders.","The ablations imply the two encoders carry different information: removing surface features costs roughly 11 percentage points of recovery, while removing backbone features costs roughly 30, so global fold is the primary constraint and surface chemistry is a local refinement.","The generated sequences are about 38% different from native sequences yet fold back to the target with low RMSD, indicating the model has learned a structure-to-sequence mapping rather than memorized training sequences."],"supporting_citations":[{"why":"Serves as a backbone-only inverse-folding baseline that the model must beat and defines the graph-based approach the paper contrasts.","marker":"Dauparas et al., 2022"},{"why":"Supplies the pretrained backbone encoder weights the model is initialized with and the main inverse-folding baseline for comparison.","marker":"Hsu et al., 2022"},{"why":"Supplies the pretrained autoregressive protein language model that initializes the sequence decoder.","marker":"Nijkamp et al., 2023"},{"why":"Provides the geometric vector perceptron formulation used in the backbone encoder.","marker":"Jing et al., 2020"},{"why":"Provides the transformer encoder and decoder blocks that carry both structural encoders and the sequence decoder.","marker":"Vaswani et al., 2017"},{"why":"Provides the PRIDE benchmark's CATH training split and CAMEO test set used for fine-tuning and evaluation.","marker":"chq1155, 2024"},{"why":"Supplies the surface point-cloud representation and the 16-nearest-atom neighborhood convention used by the surface encoder.","marker":"Yuan et al., 2023"},{"why":"Supplies the structure-prediction model used to fold generated sequences for TM-score and RMSD evaluation.","marker":"Abramson et al., 2024"}],"fun_headline_variants":["Dual-structure deep model recovers 61.47% of native sequences","Backbone and surface synergy sets protein recovery record","DS-ProGen: fusing fold and surface for SOTA protein design","Two structural cues beat one in inverse protein folding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the headline 61.47% recovery rate comes from a fair, apples-to-apples comparison: the baselines are evaluated under the same fine-tuning, decoding, and test-set conditions as DS-ProGen, even though the surface model was trained on a chemically filtered subset and at least one published PRIDE baseline is missing from the main table.","fun_headline_variants_meta":{"raw":{"variants":["Dual-structure deep model recovers 61.47% of native sequences","Backbone and surface synergy sets protein recovery record","DS-ProGen: fusing fold and surface for SOTA protein design","Two structural cues beat one in inverse protein folding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1909,"prompt_tokens":930,"completion_tokens":979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":908}},"tokens_in":546,"tokens_out":979,"duration_ms":9568,"temperature":1.0,"reasoning_tokens":908,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:32:02.857564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark with every baseline fine-tuned on the same PRIDE training data, decoded with the same temperature 0.1 and top-k 10, and include the missing published baseline; if a backbone-only baseline trained on the same surface-filtered subset reaches 61.47%, or the margin over the next-best method shrinks to noise, the dual-structure fusion is not the source of the reported gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained backbone encoder weights the model is initialized with and the main inverse-folding baseline for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the surface point-cloud representation and the 16-nearest-atom neighborhood convention used by the surface encoder."}],"review_version":1}