{"id":"e1c33977-6fc4-486c-a2f2-6fa371dfeb19","arxiv_id":"2507.20243","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Protein-SE(3) is a unified benchmark that retrains six SE(3) protein backbone generative models on the same data and evaluates them with identical metrics for designability, diversity, and novelty.","lead":"The paper introduces Protein-SE(3), a benchmark that re-trains six SE(3)-based protein structure generative models under a shared PyTorch Lightning framework with matched data and metrics. The goal is to enable fair, apples-to-apples comparison of DDPM, score-matching, and flow-matching designs, and to test a high-level mathematical abstraction for prototyping these models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fair-comparison claim is undermined because RfDiffusion is evaluated with an official checkpoint trained outside the benchmark's dataset, while retrained methods' hyperparameters are undisclosed.","rationale":"The paper's core contribution is a fair, unified benchmark in which models are retrained on the same dataset and evaluated with the same metrics. That claim requires every compared method to be trained under comparable conditions on the same data. Section 5 explicitly concedes that RfDiffusion could not be retrained and is evaluated with its official checkpoint, so the 'same training dataset' condition fails for the method that performs best in Table 1. The label 'supplementary reference' is not reflected in the main comparison table, and Section 7.1's observations about which families perform better include RfDiffusion in the ranking. Separately, hyperparameters for the retrained methods are not disclosed, so the mutual fairness among Genie1, Genie2, FrameDiff, FoldFlow, and FrameFlow is not independently auditable. The mathematical abstraction in Section 3 is suggestive but is not load-bearing for the benchmark's validity, so I do not base the objection on it. Because the issue is identifiable and correctable by disclosure or by removing RfDiffusion from the head-to-head comparison, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":15821,"tokens_out":4531,"duration_ms":57854,"concrete_test":"Recompute Table 1 rankings and the Section 7.1 qualitative conclusions after removing the RfDiffusion row or explicitly flagging it as non-comparable, using only the five retrained methods. If the relative ranking and qualitative conclusions are unchanged, the fairness issue is confined to RfDiffusion; if they change, the headline comparison is confounded by the official-checkpoint inclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that all integrated methods are fairly investigated with the same training dataset and evaluation metrics is contradicted in Section 5: RfDiffusion's training code is unavailable, so its official checkpoint is used as a 'supplementary reference.' Yet Table 1 lists RfDiffusion as a full row, and Section 7.1 draws comparative conclusions using rankings that include it. Because the official checkpoint was trained on RfDiffusion's original data and configuration rather than the benchmark's Section 4 dataset, the 'same training dataset' condition is violated for the method that dominates the quality columns. Additionally, the paper does not disclose learning rates, noise schedules, optimizer settings, or training budgets for the five retrained methods, and Table 2 shows training epochs ranging from 100 to 800. The fairness of the mutual comparison among retrained methods is therefore an unverified assumption rather than a demonstrated property of the benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Protein-SE(3), a benchmark for SE(3)-based generative protein backbone models. It integrates six published methods (Genie1, Genie2, FrameDiff, RfDiffusion, FoldFlow variants, FrameFlow) into a PyTorch Lightning framework, retrains most of them on a common PDB-derived dataset, and evaluates them on unconditional scaffolding, motif scaffolding, secondary-structure distribution, and efficiency. The paper also provides a mathematical abstraction of the underlying DDPM, score-matching, and flow-matching formulations, together with small-scale MLP experiments on R3 and SO(3) alignment. The central claims are that the framework enables fair cross-method comparison under the same training data and evaluation metrics, and that the mathematical toolkit supports rapid prototyping and provides insight into model performance.","tokens_in":16008,"tokens_out":4568,"duration_ms":52396,"significance":"If the fairness claim can be substantiated, this benchmark would be a useful community resource: it standardizes dataset construction, training, and evaluation for a fast-moving area, and its release of code and data is a genuine service. The inclusion of multiple metric families (designability, diversity, novelty, secondary structure, efficiency) is a strength, as is the attempt to abstract the mathematical cores of the three generative paradigms. The toy R3/SO(3) experiments are a useful didactic device, though their predictive value for full protein models is not established. However, the paper's headline claim that all methods are fairly investigated with the same training dataset is currently undermined by the treatment of RfDiffusion and by the lack of disclosed training hyperparameters, so the benchmark's central contribution is not yet demonstrated.","major_comments":[{"comment":"The paper's central fairness claim ('All integrated methods are fairly investigated with the same training dataset and evaluation metrics') is violated for RfDiffusion. Section 5 states that RfDiffusion's training code is unavailable and that its official checkpoint is reported only as a 'supplementary reference,' yet Table 1 lists RfDiffusion as a full row in the main comparison, and Section 7.1 draws comparative conclusions that include it (for example, the claim that flow-matching methods demonstrate relatively better quality, while Table 1 shows RfDiffusion with the highest scTM and lowest scRMSD at every length). Because the official checkpoint was trained outside the benchmark's Section 4 dataset and configuration, including it in the same ranking conflates algorithmic merit with training-set and training-configuration differences. The authors should either remove RfDiffusion from the main comparative tables and rankings, clearly separate it as reference-only in all analyses, or retrain it under the unified framework and report those training details.","section":"Section 5, Table 1, Section 7.1"},{"comment":"The 'unified training framework' is not shown to provide aligned training setups across the retrained methods. Table 2 reports training epochs ranging from 100 (Genie1, Genie2, FoldFlow) to 150 (FrameDiff) to 800 (FrameFlow), and the paper does not disclose learning rates, noise schedules, optimizer settings, batch sizes, or training budgets for any retrained model. Without these details, observed differences among the retrained methods are confounded by training configuration, so the claim of fair comparison among them remains an assumption. The authors should disclose all hyperparameters and schedules, or provide a sensitivity analysis showing that the reported rankings are robust to reasonable training-configuration variations.","section":"Sections 7.1 and 7.4, Table 2"},{"comment":"The qualitative summary in Section 7.1 is contradicted by the paper's own table. The text states that 'in terms of the Quality metric (scTM and scRMSD), flow-matching based methods (FrameFlow and Foldflow) demonstrates relatively better performance,' but Table 1 shows RfDiffusion outperforming all flow-matching methods on both scTM and scRMSD at every length (e.g., at length 100, RfDiffusion scTM 0.97 ± 0.01 versus FrameFlow 0.90 ± 0.10; at length 500, RfDiffusion scTM 0.90 ± 0.11 versus FrameFlow 0.56 ± 0.20). If the authors intend to exclude RfDiffusion from this qualitative claim because it was not retrained, that exclusion must be explicit, and the table should be annotated or restructured so that its content matches the narrative.","section":"Section 7.1, Table 1"},{"comment":"The paper uses the synthetic MLP experiments to conclude that 'the curves of Flow Matching on R3 and SO(3) alignment exhibit better convergence, indicating the superior design performance evaluated in Section 7.' This inference is not justified as stated. The toy experiments model simple MLPs on synthetic distributions, not the equivariant IPA-style architectures and full protein datasets used in Section 7, and no experiment in the paper establishes that convergence in the toy setting transfers to protein backbone quality. Moreover, the benchmark results in Table 1 do not consistently show flow-matching superiority even among the retrained methods (for example, FoldFlow-OT and FoldFlow-SFM have starred scTM values below 0.5 at length 500). The claim should be removed or explicitly reframed as a hypothesis, or supported by a controlled experiment linking the abstract metrics to full-model performance.","section":"Section 3 and Section 7.1"},{"comment":"The footnote to Table 1 states that '*' indicates quality below the scTM > 0.5 threshold, 'so Novelty and Diversity are excluded from the comparison.' However, the table still reports Novelty and Diversity values for starred rows (e.g., Genie1 at length 500, FoldFlow-OT at length 500, FoldFlow-SFM at length 500). Either the starred entries should have blank cells for Novelty and Diversity, or the footnote should be corrected to say that these values are reported but not used in the comparative ranking. As written, the table and its footnote are internally inconsistent.","section":"Table 1 footnote"}],"minor_comments":[{"comment":"The abstract's statement 'All integrated methods are fairly investigated with the same training dataset and evaluation metrics' should be qualified to reflect that RfDiffusion is evaluated with its official checkpoint, as acknowledged in Section 5.","section":"Abstract and Section 5"},{"comment":"There are several typographical errors: 'angel' should be 'angle,' 'Follwing' should be 'Following,' and 'Backend by Pytorch Lightning' should be 'Backed by PyTorch Lightning.'","section":"Section 2"},{"comment":"References [12] and [13] are the same paper (Hu, Lundgren, and Niemi, Physical Review E, 2011) and should be merged.","section":"References"},{"comment":"The text says 'Given 5 unconditionally generated protein structures with every length ranging from 60 to 320 (1,040 structures in total).' If lengths are sampled as integers from 60 to 320, five samples per length would give 1,305 structures, not 1,040; the sampling procedure should be clarified so the total is reproducible.","section":"Section 7.3"},{"comment":"The counts in the motif-scaffolding summary (FrameFlow 13/24, Genie2 7/24, RfDiffusion 6/24) sum to 26/24; the authors should clarify how ties are handled or report the counts without overlap.","section":"Section 7.2"},{"comment":"The axis labels in Figure 5 appear garbled with escaped unicode strings in the manuscript; the figure should be regenerated with readable labels before publication.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a useful and timely benchmarking need, and the released code/data are valuable. The central fairness claim, however, is load-bearing and is currently not supported because RfDiffusion is included in the main comparisons despite being an untrained reference checkpoint, and because training hyperparameters for the retrained methods are undisclosed. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would also encourage the editor to ensure that the final version makes the 'supplementary reference' status of RfDiffusion explicit in every table and qualitative conclusion where it appears."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the packaged, retrained framework: five SE(3) models in one PyTorch Lightning harness with a shared dataset and metric suite. That is worth having, and it goes beyond ProteinBench and Scaffold-Lab, which mostly evaluated released checkpoints. The abstract's math abstraction is mostly a careful restatement of Genie, FrameDiff, and FoldFlow equations—fine for teaching, not a new theory. The synthetic R3/SO(3) Wasserstein experiments support the claim that flow matching converges faster on toy MLP distributions, but the paper leans on that to explain Table 1, which is a stretch; toy behavior does not establish transfer to real proteins.\n\nThe main soft spot is the fairness claim. Section 5 says RfDiffusion has no retraining code, so they evaluate its official checkpoint as a \"supplementary reference,\" but Table 1 lists it as a full method and the text draws comparative conclusions from its rankings. The official checkpoint was trained on different data and configuration, so the \"same training dataset\" condition in the abstract is violated exactly for the method that dominates the quality columns. That is not fatal to the benchmark, but it needs to be a clearly marked reference row, not a fair-comparison entry. I would also want hyperparameters, noise schedules, optimizer settings, and compute budgets for the five retrained models; Table 2 shows epochs from 100 to 800, so training configurations vary enough to matter.\n\nThe \"first comprehensive benchmark built upon a unified training framework\" is overstated given Scaffold-Lab and ProteinBench are cited two pages earlier. The aligned-retraining twist is real, so this is fixable by rewording rather than a new dispute. Citation pattern is otherwise sound: all key baselines are cited, and the appendix faithfully credits prior derivations. No smoke-and-mirrors fitting, no invented results.\n\nWho is this for? People building or using SE(3) generators for scaffolding will get a convenient starting point and a reasonably broad metric suite. Theorists will not learn much from the abstraction section. The soft spots are addressable: disclose configs, demote RfDiffusion, soften the \"first\" claim. I would send it to peer review—it deserves a serious referee—but I would expect major revision before publication.","headline":"Useful packaged benchmark with a real fairness caveat—RfDiffusion is a guest row, and retraining configs are undisclosed.","tokens_in":16468,"tokens_out":1646,"would_cite":true,"duration_ms":21472,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark retrains six SE(3) protein generators under one setup for a fair test.","keywords":["protein structure design","SE(3) generative models","benchmark","diffusion models","flow matching","score matching","motif scaffolding","protein backbone generation"],"falsifier":"Retrain RfDiffusion from scratch in the same framework with disclosed hyperparameters and rerun the unconditional scaffolding tables; if its quality drops toward the other score-matching models, the flow-matching advantage is confirmed, whereas if it keeps its edge, the paper understates RfDiffusion's algorithmic strength. Releasing the exact training configurations and reproducing Table 1 would settle the same question.","tokens_in":15653,"feed_emoji":"🧬","tokens_out":9282,"duration_ms":100045,"temperature":0.7,"pith_summary":"Protein-$\\mathrm{SE}(3)$ retrains six existing generative protein backbone models from the DDPM, score-matching, and flow-matching families inside one modular training pipeline with the same datasets and evaluation metrics, so that design performance can be compared on equal footing. The central claim is that, once training is aligned this way, flow-matching methods converge most cleanly on synthetic translation and rotation alignment tasks and generally deliver strong designability, diversity, novelty, and inference speed, while the score-matching model RfDiffusion, evaluated with its official checkpoint, still reaches the highest self-consistency quality, especially at longer chain lengths. The benchmark also abstracts the shared mathematics of $\\mathrm{SE}(3)$ diffusion into a synthetic toolkit that trains on coordinates and rotation matrices alone, enabling rapid prototyping before protein-level training. The contribution is therefore a standardized arena: one training setup, one set of metrics, and a public codebase for comparing and building $\\mathrm{SE}(3)$-based protein structure generators.","feed_headline":"Retrained on one track: six protein generators face a fair test","feed_subtitle":"Flow-matching leads on speed and diversity; the official RfDiffusion checkpoint still sets the quality bar.","key_machinery":"The load-bearing object is the residue-frame decomposition of the protein backbone, in which each residue is a rigid transformation in $\\mathrm{SE}(3)=\\mathbb{R}^3\\rtimes\\mathrm{SO}(3)$; generation is split into a translation process on $\\mathbb{R}^3$ with Gaussian noise and a rotation process on $\\mathrm{SO}(3)$ with isotropic Gaussian noise $\\mathrm{IGSO}(3)$. The unifying mechanism is a shared training and evaluation pipeline that retrains all baselines on the same filtered monomeric dataset, and a mathematical abstraction toolkit that measures distribution alignment in both spaces with the first-order Wasserstein distance $W_1$. This decomposition is what allows DDPM, score-matching, and flow-matching models to share one codebase, and it is what lets the benchmark attribute performance differences to algorithmic choice rather than to dataset or training-scheme variation.","core_discovery":"The paper's central claim is that fair cross-comparison of $\\mathrm{SE}(3)$ protein generators is achievable when every method is retrained under one shared framework, meaning the same data filtering, same training backend, and same metrics, and that under this setup flow-matching models display the best convergence in the synthetic $\\mathbb{R}^3$ and $\\mathrm{SO}(3)$ alignment experiments and dominate diversity and efficiency, while the official-checkpoint RfDiffusion retains the best unconditional scaffolding quality in terms of self-consistency TM-score and RMSD. On the motif scaffolding benchmark, the flow-matching model FrameFlow produces the most designable scaffolds, with the highest average self-consistency TM-score in 13 of 24 tasks and the lowest motif RMSD in 19 of 24 tasks. The paper further claims that all methods degrade as chain length grows, and that flow-matching's use of ordinary differential equations explains its speed advantage.","pith_inferences":["Because RfDiffusion is the only model that was not retrained, its leading quality scores sit outside the benchmark's own fairness premise; retraining it once training code is available is the natural first experiment to test whether flow-matching's advantage is algorithmic.","The synthetic-space abstraction implies that Wasserstein convergence on $\\mathbb{R}^3$ and $\\mathrm{SO}(3)$ could be used as a cheap screening signal for new $\\mathrm{SE}(3)$ generative methods, but the paper does not show that this signal transfers to full protein design.","If flow-matching's efficiency advantage persists at larger scale, ODE-based models may become the default backbone for protein design, pushing the DDPM family toward architectural changes to escape its cubic-cost attention.","The benchmark's modular design makes it straightforward to add newly released $\\mathrm{SE}(3)$ generators to the same training track, so the ranking is a living comparison rather than a fixed verdict."],"forward_implications":["If the unified comparison is correct, flow-matching models are the stronger default for unconditional protein scaffolding when both designability and inference speed matter.","On motif scaffolding, FrameFlow yields the most designable scaffolds on 13 of 24 Design24 tasks and the lowest motif RMSD on 19 of 24, so flow matching is also competitive under structural constraints.","All compared methods lose quality as chain length grows from 100 to 500 residues, so longer protein design remains an unsolved gap independent of generative paradigm.","Flow-matching models sample with ODE integration and need far fewer steps than the DDPM and score-matching baselines, making them the most efficient in the benchmark.","The secondary-structure analysis shows that several retrained models generate overly helical backbones, and only a subset of methods match the natural distribution of real proteins."],"supporting_citations":[{"why":"Supplies the filtering protocol for the monomeric training set and the score-matching baseline it defines.","marker":"[35]"},{"why":"Supplies the Genie2 baseline and the random mask-sampling scheme adopted for motif scaffolding training.","marker":"[19]"},{"why":"Supplies the Design24 motif evaluation tasks and the score-matching model RfDiffusion, which appears only through its official checkpoint.","marker":"[32]"},{"why":"Supplies the FoldFlow flow-matching baseline and the SO(3) flow-matching formulations abstracted in the mathematical toolkit.","marker":"[4]"},{"why":"Supplies the FrameFlow baseline used in unconditional and motif scaffolding comparisons.","marker":"[36]"},{"why":"Supplies the Genie1 DDPM baseline.","marker":"[18]"},{"why":"Provides the inverse-folding model used to turn generated backbones into sequences for designability scoring.","marker":"[8]"},{"why":"Provides the structure-prediction model used to compute self-consistency between designed sequences and generated backbones.","marker":"[20]"},{"why":"Provides the structural alignment tool used to compute novelty as maximum TM-score against the reference set.","marker":"[29]"}],"fun_headline_variants":["Six protein generators, one fair test: who wins?","Flow-matching dominates speed and diversity in protein benchmark","RfDiffusion checkpoint still leads in quality, benchmark shows","Benchmark: flow-matching fastest, RfDiffusion most accurate","Retrained on one track: flow-matching wins, RfDiffusion excels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the premise that retraining every method under one framework isolates algorithmic quality; that premise is strained because Section 5 admits RfDiffusion is evaluated only with its official checkpoint and Section 7.4 does not report hyperparameters, learning rates, or noise schedules for the retrained models, so part of the observed gap could be training configuration rather than algorithm.","fun_headline_variants_meta":{"raw":{"variants":["Six protein generators, one fair test: who wins?","Flow-matching dominates speed and diversity in protein benchmark","RfDiffusion checkpoint still leads in quality, benchmark shows","Benchmark: flow-matching fastest, RfDiffusion most accurate","Retrained on one track: flow-matching wins, RfDiffusion excels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001544,"raw_usage":{"total_tokens":6169,"prompt_tokens":933,"completion_tokens":5236,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":5147}},"tokens_in":549,"tokens_out":5236,"duration_ms":42408,"temperature":1.0,"reasoning_tokens":5147,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:40:43.837751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain RfDiffusion from scratch in the same framework with disclosed hyperparameters and rerun the unconditional scaffolding tables; if its quality drops toward the other score-matching models, the flow-matching advantage is confirmed, whereas if it keeps its edge, the paper understates RfDiffusion's algorithmic strength. Releasing the exact training configurations and reproducing Table 1 would settle the same question.","supporting_citations":[{"cited_title":"Se (3) diffusion model with application to protein backbone generation","cited_arxiv_id":null,"evidence_quote":"Supplies the filtering protocol for the monomeric training set and the score-matching baseline it defines."},{"cited_title":"De novo design of protein structure and function with rfdiffusion","cited_arxiv_id":null,"evidence_quote":"Supplies the Design24 motif evaluation tasks and the score-matching model RfDiffusion, which appears only through its official checkpoint."},{"cited_title":"Bronstein, and Alexander Tong","cited_arxiv_id":null,"evidence_quote":"Supplies the FoldFlow flow-matching baseline and the SO(3) flow-matching formulations abstracted in the mathematical toolkit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FrameFlow baseline used in unconditional and motif scaffolding comparisons."},{"cited_title":"Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds","cited_arxiv_id":null,"evidence_quote":"Supplies the Genie1 DDPM baseline."}],"review_version":1}