{"id":"8b803b77-0112-42c0-b6e6-e423f1dd797e","arxiv_id":"2507.19755","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A segment-level transformer predicts enzyme temperature stability with RMSE 24.03 and Pearson 0.33, and guided a cutinase variant with 1.64-fold improved activity after heat treatment.","lead":"The authors introduce Segment Transformer, a neural network that predicts an enzyme's temperature stability from its amino acid sequence by treating short sequence fragments as tokens. The model posts modest correlations (Pearson 0.33) and was used to engineer a cutinase variant with 1.64-fold improved heat-treated activity, though the mutation-level prediction method is not fully specified.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The architecture is selected on the same test set used for final reporting, and the selected RMSE (24.03) is not below the BiLSTM baseline (23.96); the state-of-the-art claim rests on an unvalidated, test-set-driven comparison.","rationale":"The reader focused on the undefined mutation-temperature-score mapping. That is a real problem, but the more fundamental issue is that the quantitative model claim is already invalid as reported: the architecture is chosen on the test set that later produces the headline numbers. The text itself identifies Fig. 4 as test-set optimization, so this is not an inference about hidden practice. The selection among nine configurations is exactly the kind of multiple comparison that inflates apparent performance; the reported RMSE, MAE, and correlations are optimistic. Moreover, the selected model does not actually beat BiLSTM on RMSE (24.03 vs 23.96), so the paper's own primary metric contradicts the \"state-of-the-art\" summary. Because this concern attacks the central claim directly, it is more load-bearing than the mutation-mapping issue. A corrected evaluation could still salvage the dataset and architecture idea, but the current paper's central quantitative claim should not be accepted. I therefore keep the reader's REJECT verdict, though I locate the primary weakness in the evaluation protocol rather than in the application-study mapping.","tokens_in":18948,"tokens_out":4667,"duration_ms":58355,"concrete_test":"Redo the experiments with a clean protocol: keep the test set completely untouched; use only the validation split to select the number of scales and DGSA blocks, then evaluate the selected model exactly once on the test set; repeat over 5-10 random test partitions and report mean and bootstrap CIs. If the test-selected RMSE of 24.03 is not replicated or does not beat BiLSTM's 23.96 under this protocol, the state-of-the-art claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Segment Transformer reaches state-of-the-art accuracy depends on the reported test metrics being unbiased generalization estimates. The paper's architecture search violates this: Fig. 4 is explicitly an optimization of the Segment Transformer architecture on the temperature stability test set, ranking nine configurations (DGSA blocks × scales) by RMSE, MAE, Pearson, Spearman, and grouped MAE; the \"final design\" with two scales and two DGSA blocks is selected because it achieved the best overall test performance (RMSE 24.03, MAE 18.09, correlations 0.33). The same test set is then used in Fig. 6 to declare superiority over the baselines. Selecting a configuration from the test set and then reporting performance on that same test set makes 24.03 a selected minimum, not an honest estimate of generalization; the true gap to BiLSTM (RMSE 23.96, MAE 18.49, r=0.29) is small and could easily reverse. In fact, on the primary RMSE metric, the selected Segment Transformer is numerically worse than BiLSTM, so the \"state-of-the-art\" wording depends on correlations and grouped MAE that were also part of the test-set selection criterion. This invalidates the headline quantitative claim, which is the foundation for the paper's novelty and for the engineering case study.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces a curated enzyme temperature stability dataset (3,454 entries from BRENDA) and a 'Segment Transformer' that predicts whole-sequence temperature stability from ESM-2 embeddings via multi-scale segment features and dual grouped segment attention. It claims state-of-the-art test performance (RMSE 24.03, MAE 18.09, Pearson/Spearman 0.33) and reports a cutinase engineering study in which a model-selected mutation A78E increased relative activity 1.64-fold and half-life 3.9-fold.","tokens_in":19206,"tokens_out":7527,"duration_ms":90448,"significance":"If the claims were supported, the paper would offer a valuable sequence-only baseline for enzyme thermostability prediction and a practical mutation-selection tool. The curated dataset, public code/data links, and wet-lab validation are concrete assets. However, the headline numerical claims are undermined by test-set architecture selection and an undocumented per-mutation scoring mechanism; with these corrected, the contribution could be meaningful, but as presented the evidence is not reliable.","major_comments":[{"comment":"Figure 4 is explicitly an architecture search on the temperature stability test set, and the final configuration (two scales, two DGSA blocks) is selected because it achieved the best RMSE, MAE, correlations, and grouped MAE on that test set. The same test set is then used in Figure 6 to claim superiority over all baselines. Consequently, RMSE=24.03 is a selected minimum, not an unbiased generalization estimate, and the comparison to BiLSTM (RMSE 23.96, MAE 18.49) is invalid; on the primary RMSE metric the selected Segment Transformer is numerically worse. Architecture selection must be done on the validation set (or via nested resampling) and the test set used only once.","section":"Architecture Optimization (Fig. 4); Performance Evaluation (Fig. 6)"},{"comment":"The mutation-selection protocol relies on per-residue 'temperature scores' (Fig. 9a), but the Methodology defines only a whole-sequence regression target (Eqs. 18-22) and an averaged segment attention vector alpha-bar (Eq. 14). No equation or algorithm is given for converting segment-level attention into per-mutation temperature scores, and because segment lengths differ across scales, the average in Eq. (14) is not well-defined without an alignment or upsampling step. The thresholds (importance >20, temperature score >50) are therefore not reproducible, and the cutinase results cannot be attributed to the model's predictions. This is a load-bearing gap for the paper's proof-of-concept claim.","section":"Applications of segment transformer in Thermal Property-Related Enzyme Engineering; Eq. (14)"},{"comment":"The abstract and conclusion state that the model 'successfully guided' the engineering of a cutinase with 'only 17 mutations', but Fig. 9b shows that only two of the seventeen tested mutations (A78E, N79G) improved thermostability, with several others reducing activity, and the Conclusion itself concedes that 'predictions of mutation effects are not yet fully reliable.' Without a defined scoring function, a random or baseline selection comparison, or a prospective test, the engineering experiment does not validate the model; the 3.9-fold half-life improvement refers to a single mutation selected after screening, not to a designed variant produced directly from the model.","section":"Conclusion; Fig. 9b"}],"minor_comments":[{"comment":"The temperature intervals R_k and weights w_k in the weighted RMSE loss are never specified, so the loss function is not reproducible; an ablation of the weighting scheme would strengthen the claim that it mitigates imbalance.","section":"Experimental Setup and Loss Functions, Eq. (18)"},{"comment":"The 10% test split was drawn before MMseqs2 clustering, so there is no control for sequence similarity between test and training sequences; this can inflate all reported metrics and should be discussed or corrected.","section":"Data Collection and Curation, Table 1"},{"comment":"The best validation checkpoint is selected from metrics evaluated every 8 epochs, and only a single run is reported; providing multiple seeds with standard deviations would make the small differences in Fig. 6 (e.g., correlations 0.32 vs 0.33) more convincing.","section":"Results and Discussion, Fig. 5"},{"comment":"Several typos and broken references need correction: 'access model performance' in Evaluation Metrics, 'fluctation' in Fig. 3, 'Gouped' in the TOC, and the empty cross-reference 'see Section' in Performance Evaluation.","section":"Throughout"},{"comment":"The text reports a 3.9-fold increase in half-life, but the stated numbers 29.5 min divided by 6.8 min equal approximately 4.3; please reconcile this discrepancy.","section":"Application Study, Fig. 9c"},{"comment":"The phrase 'achieved through only 17 mutations' should be rephrased: 17 mutations were experimentally screened, and the beneficial variant was selected afterward, so the claim of 'only 17' somewhat understates the screening effort.","section":"Abstract and Conclusion"}],"recommendation":"reject","confidential_remarks":"The manuscript has useful resources (dataset, code, and a real wet-lab effort), and the segment-level idea is worth exploring. However, the architecture search on the test set is a fundamental methodological error that invalidates the headline metrics, and the per-mutation temperature score is undefined. I recommend rejection; a resubmission that selects architecture on validation, defines the mutation-scoring algorithm, and reports unbiased test statistics (ideally with multiple seeds and a proper baseline) would be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere is my read. The paper has two genuinely useful pieces: a curated sequence-only thermostability dataset (27,216 BRENDA entries cleaned to 3,454 with a sensible MMseqs2-based split) and a segment-level architecture that is a real variation on residue-level ESM-2 approaches. The wet-lab cutinase experiment (A78E, 1.64x relative activity, 3.9x half-life) is also interesting, and the authors put code and data on GitHub/Zenodo, which is more than many papers in this space do.\n\nThe soft spots are where the reader says they are. The load-bearing problem is in Fig. 4: the number of scales and DGSA blocks was chosen by ranking nine configurations on the test set, and the same test set is then used in Fig. 6 to declare superiority over baselines. That makes the reported RMSE 24.03 a selected minimum, not an unbiased generalization estimate. The circularity is explicit in the text, not a misreading. On the primary metric, BiLSTM gets 23.96, so the state-of-the-art claim rests on correlations and grouped MAE that were themselves part of the selection criterion. That has to be fixed with a separate architecture-search split or nested cross-validation before the numbers mean anything.\n\nThe second soft spot is the mutation-scoring mechanism. The methodology defines regression of whole-sequence temperature stability; it never defines how segment importance scores become per-residue temperature scores for individual substitutions. The conclusion honestly admits that mutation-effect predictions are not fully reliable, which is good, but it means the engineering case study is a screen guided by an unspecified heuristic, not a validated model output. That is a central application claim that cannot be checked as written.\n\nThe third issue is more moderate: even on the selected test set, the improvement over a BiLSTM is marginal. Segment-level features help in some ranges, but the architecture's practical value is not yet demonstrated.\n\nAll that said, this paper deserves a serious referee. The dataset and the segment-level idea are worth engaging with, and the code is available for someone to re-run with a corrected protocol. The faults are fixable, but they are load-bearing. If I were an editor, I would send it out with the expectation that reviewers demand an unbiased architecture selection, a formal definition of the mutation-scoring mapping, and honest reporting of the small margin over baselines. The paper is for people building thermostability predictors, not for people who need a ready-to-use tool. I would not reject without review; I would reject the current claims.\n\nBest.","headline":"A useful dataset and a genuine segment-level architecture, but the headline results are compromised by test-set-based architecture selection and an unspecified mutation-scoring procedure.","tokens_in":19811,"tokens_out":2441,"would_cite":true,"duration_ms":31050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A segment-level Transformer predicts enzyme temperature stability from sequence alone, outperforming existing predictors and guiding a cutinase engineering experiment with a 1.64-fold thermostability improvement.","keywords":["enzyme thermostability","temperature stability prediction","segment-level representation","transformer","protein language model","enzyme engineering","cutinase","sequence-based prediction"],"falsifier":"Apply the exact score-extraction procedure used in the cutinase study to a held-out panel of single-point mutants with experimentally measured thermostability changes; if the predicted per-mutation temperature scores fail to rank the mutants' measured effects better than chance, the engineering-guidance claim collapses even if whole-sequence prediction error remains low.","tokens_in":18715,"feed_emoji":"🌡️","tokens_out":8939,"duration_ms":93921,"temperature":0.7,"pith_summary":"This paper argues that enzyme temperature stability can be predicted from amino acid sequence by modeling short contiguous segments of the protein rather than individual residues. The authors curate a dataset of 3,454 unique enzyme temperature records and build the Segment Transformer, which converts residue-level embeddings from a pretrained protein language model into multi-scale segment features and applies grouped segment attention. They report that this model outperforms existing sequence-based predictors, reaching an RMSE of 24.03°C, an MAE of 18.09°C, and Pearson and Spearman correlations of 0.33 on a similarity-clustered test set. As a practical test, the model's segment importance and per-position temperature scores were used to select 17 mutations in a cutinase, yielding a 1.64-fold increase in relative activity after heat treatment and a 3.9-fold longer half-life. The paper itself states that mutation-effect predictions are not yet fully reliable, so the engineering result is presented as a screen, not a guaranteed design.","feed_headline":"Segments, not single residues, predict enzyme heat stability","feed_subtitle":"A transformer that reads proteins in chunks bests existing predictors and guided a 1.64-fold cutinase stability gain.","key_machinery":"The load-bearing mechanism is the Dual Grouped Segment Attention (DGSA) block, an attention layer that splits a sequence's segment-level features into short-range groups and long-range groups, applies attention independently to each group, and sums the two branches. It operates on segment representations produced by strided down-sampling, fixed-length sequence segmentation, and 2D convolution, so each token corresponds to a short contiguous stretch of amino acids rather than a single residue. The attention weights from multiple scales are averaged into a segment importance profile, which is what the engineering study uses to prioritize mutation sites.","core_discovery":"The central claim is that representing an enzyme as a sequence of short contiguous segments, rather than individual amino acids, captures the unequal contribution of different protein regions to thermal behavior, and that this representation makes a Transformer-based model the best available sequence-only predictor of temperature stability. On its curated test set the Segment Transformer achieves an RMSE of 24.03°C, an MAE of 18.09°C, and Pearson and Spearman correlations of 0.33, outperforming BiLSTM, CNN, Transformer, RNN, Light Attention, TemStaPro, Seq2Topt, and DeepET trained under the same reweighted loss. The paper further claims that the model can guide enzyme engineering: attention-derived segment importance combined with predicted per-position temperature scores identified 17 candidate mutations in a Humicola insolens cutinase, and the best variant, A78E, showed a 1.64-fold increase in relative activity after heat treatment and a half-life of 29.5 minutes versus 6.8 minutes for the wild type, a 3.9-fold improvement, with no loss of catalytic activity.","pith_inferences":["If segment-level attention truly marks thermostability-determining regions, fusing it with residue-level encoders could make single-mutation prediction accurate enough to replace parts of directed-evolution screening; the paper names this as future work.","The curated data are small (3,454 unique entries) and skewed toward 40–59°C enzymes, so the reported error and correlation numbers are likely a conservative floor; larger, balanced mutation-level datasets could raise them substantially.","A controlled comparison between similarity-clustered and random splitting would separate the architecture's contribution from the evaluation protocol's, since a stricter split tends to depress accuracy for all models.","The cutinase case is a single enzyme; the three additional cutinase validation panels suggest transferability within one enzyme family, but the value for unrelated enzyme families remains untested."],"forward_implications":["Enzyme heat tolerance can be ranked from sequence alone at roughly 24°C RMSE, without structure information or source-organism metadata.","Segment-level representations are more informative than residue-level tokens for thermostability, according to the ablation results, so similar hierarchical designs may help other protein property prediction tasks.","The model's fluctuation ranges and segment importance scores give researchers a visual, interpretable starting point for choosing mutation targets.","A small set of predicted mutations (17) produced a 1.64-fold gain in relative activity and a 3.9-fold longer half-life in one cutinase, suggesting the pipeline can shrink experimental screening panels.","Because mutation-effect predictions are not yet fully reliable, the model is best used to prioritize candidates for experimental validation rather than to certify individual mutations."],"supporting_citations":[{"why":"Supplies the temperature stability records and sequence accessions that form the curated dataset.","marker":"6,7"},{"why":"Provides the clustering algorithm used to split training and validation by sequence similarity.","marker":"28"},{"why":"Provides the pretrained protein language model that produces the initial residue-level embeddings.","marker":"29"},{"why":"Defines the DeepET baseline whose performance the Segment Transformer is compared against.","marker":"16"},{"why":"Defines the TemStaPro baseline, a sequence-representation thermostability predictor.","marker":"38"},{"why":"Defines the Seq2Topt baseline, a sequence-based optimum-temperature predictor.","marker":"39"}],"fun_headline_variants":["Protein segments beat single residues for heat-stability AI","Segment Transformer predicts enzyme heat stability better","Reading proteins in chunks yields better thermal predictions","Cutinase stability boosted 1.64x using segment-based AI","Segment-wise analysis improves enzyme temperature prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model is trained only to predict a whole enzyme's temperature stability, but the engineering experiment assumes that its segment importance scores and per-position temperature scores correctly identify which single amino acid changes will stabilize the enzyme, and the paper itself concedes those mutation-effect predictions are not fully reliable.","fun_headline_variants_meta":{"raw":{"variants":["Protein segments beat single residues for heat-stability AI","Segment Transformer predicts enzyme heat stability better","Reading proteins in chunks yields better thermal predictions","Cutinase stability boosted 1.64x using segment-based AI","Segment-wise analysis improves enzyme temperature prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1538,"prompt_tokens":977,"completion_tokens":561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":489}},"tokens_in":593,"tokens_out":561,"duration_ms":6251,"temperature":1.0,"reasoning_tokens":489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:02:48.486615+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the exact score-extraction procedure used in the cutinase study to a held-out panel of single-point mutants with experimentally measured thermostability changes; if the predicted per-mutation temperature scores fail to rank the mutants' measured effects better than chance, the engineering-guidance claim collapses even if whole-sequence prediction error remains low.","supporting_citations":[{"cited_title":"MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets","cited_arxiv_id":null,"evidence_quote":"Provides the clustering algorithm used to split training and validation by sequence similarity."},{"cited_title":"Science 2023, 379, 1123--1130","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained protein language model that produces the initial residue-level embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DeepET baseline whose performance the Segment Transformer is compared against."},{"cited_title":"TemStaPro: protein thermostability prediction using sequence representations from protein language models","cited_arxiv_id":null,"evidence_quote":"Defines the TemStaPro baseline, a sequence-representation thermostability predictor."},{"cited_title":"Seq2Topt: a sequence-based deep learning predictor of enzyme optimal temperature","cited_arxiv_id":null,"evidence_quote":"Defines the Seq2Topt baseline, a sequence-based optimum-temperature predictor."}],"review_version":1}