{"id":"9dde819c-9cf8-4d58-b842-792ef66d51fa","arxiv_id":"1908.08010","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"GP-PSM, a genetic programming scoring function, re-ranked PEAKS de novo peptide candidates and reduced the false-positive rate from 0.44 to 0.34 on 120 spectra.","lead":"A computer-trained scoring model re-ranks candidate peptide matches from mass spectrometry and cuts false identifications on a small proteomics test set. The method beats two standard machine-learning baselines, but the result rests on 120 spectra and lacks statistical validation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation set may overlap with learning set; no explicit disjointness in Section 4.1, and no code/data to verify separation.","rationale":"The reader's weakest assumption identified a transfer gap between Mascot-trained and PEAKS-applied scoring. That is a legitimate concern, but the more load-bearing issue is the lack of any stated disjointness between the learning and evaluation sets, which threatens the validity of the central empirical claim. If the sets overlap, the entire reported improvement is suspect; if they are disjoint, the transfer concern and small-sample issue remain but the core result may hold. The paper provides insufficient information to determine which case applies, so the verdict should be conditional on the authors demonstrating that the evaluation set is independent of the learning set and, ideally, providing a confidence interval or significance test for the 10% improvement. This is a concrete, checkable condition rather than a judgment of fraud or intent.","tokens_in":9643,"tokens_out":2658,"duration_ms":28648,"concrete_test":"Obtain from the authors the exact list of spectrum identifiers (raw file name, scan number, retention time, or equivalent) for the 10,000 learning spectra and the 120 evaluation spectra, then compute the intersection. If any of the 120 evaluation spectra appear among the 10,000 learning spectra, the experiment must be redone with a strictly disjoint evaluation set and the headline numbers recomputed. If the lists cannot be provided, an independent replication on the same benchmark dataset with a random, explicitly disjoint 120-spectrum holdout would settle whether the 10% improvement persists.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is an empirical improvement on the 120-spectrum evaluation set. Section 4.1 states that the 10,000-spectrum learning set and the 120-spectrum evaluation set are both selected from the same comprehensive full factorial benchmark dataset, but it never states that the evaluation spectra are disjoint from the learning spectra. This matters because the GP model is trained to predict Mascot scores for the true peptide of each training spectrum. If any evaluation spectrum appears in the learning set, the model has already seen the exact feature vector for its correct peptide and the corresponding high Mascot score, effectively memorizing the correct answer for that spectrum. In that case, the reported improvement of 12 additional spectra (10% of 120) could be entirely an artifact of training/evaluation overlap rather than a genuine ability to re-rank PEAKS candidate peptides. The paper also provides no list of spectrum identifiers, code, or data to rule this out. A secondary concern is that the best GP individual is chosen from 30 runs by training RSS, and the evaluation is a single 120-spectrum set with no confidence intervals or significance testing; this alone would limit certainty, but the potential overlap is more fundamental.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GP-PSM, a genetic-programming symbolic-regression scoring function for re-ranking peptide-spectrum matches. It extracts 11 features per PSM and regresses Mascot confidence scores on a learning set of 10,000 MS/MS spectra from the Wessels et al. benchmark, then applies the evolved function to re-score five candidate peptides per spectrum from PEAKS de novo sequencing on 120 evaluation spectra. The central empirical claim is that GP-PSM raises correct top-1 identifications from 67 to 79 of 120 spectra and lowers FPR from 0.44 to 0.34, outperforming RF and SVR re-ranking. The paper also analyzes the evolved tree for feature selection and interpretability.","tokens_in":9893,"tokens_out":5943,"duration_ms":62970,"significance":"If the reported 10-percentage-point FPR reduction is robust, the contribution is a useful, interpretable nonlinear PSM scoring function and a demonstration that GP-based re-ranking can outperform RF/SVR on a real benchmark. The use of a public benchmark with known ground truth, explicit feature definitions, comparison with two ML baselines, and interpretability analysis are strengths. The main limitations are statistical: the headline improvement rests on 120 spectra and a single GP individual selected from 30 runs, with no confidence intervals or significance tests, and the learning/evaluation split is not shown to be disjoint. These issues are fixable, but until addressed the central empirical claim is not established at the standard expected for a journal paper.","major_comments":[{"comment":"The learning set (10,000 spectra) and evaluation set (120 spectra) are both selected from the same full factorial benchmark dataset, but the text never states that the two sets are disjoint. Because the GP model is trained to predict the Mascot score of the known peptide for each training spectrum, any evaluation spectrum that also appeared in training would give the model direct access to the correct answer for that spectrum. This makes the abstract's 10% improvement claim potentially an artifact of training/evaluation overlap. Please state the disjointness explicitly and provide spectrum identifiers or a reproducible split so that the separation can be audited.","section":"Section 4.1 / Table 2"},{"comment":"No confidence intervals or significance tests are reported for the evaluation results. On n=120, the GP/SVR difference is 79 versus 72 correct spectra (FPR 0.34 versus 0.40), a gap well within plausible sampling variability; a paired test such as McNemar's test, or a bootstrap confidence interval on the FPR difference, is needed before 'GP-PSM outperforms RF and SVR' can be supported. In addition, Section 5.1 states that the best GP individual is selected from 30 runs by training RSS, so evaluating that chosen individual on a single 120-spectrum set inherits selection bias; please report the variance of the evaluation metrics across the 30 runs.","section":"Section 5.2 / Table 5"},{"comment":"The transfer assumption is load-bearing but untested. The scoring function is learned from PSMs produced by Mascot database search and applied to PSMs produced by PEAKS de novo sequencing; Section 4.3 only explains that the two sets differ, without any quantitative evidence that their feature distributions overlap. Please provide diagnostics such as feature-distribution comparisons or a cross-training experiment (e.g., train on PEAKS PSMs from a disjoint subset and evaluate), or temper the generalization claim accordingly.","section":"Section 4.3"},{"comment":"The evaluation lacks a baseline that re-ranks the same PEAKS candidate lists using a simple linear combination of the same features, or using raw Mascot scores where available. Such a baseline would isolate the contribution of the GP-evolved nonlinear form from the generic benefit of re-scoring with additional intensity features, and it is needed to support the claim that the GP representation itself is responsible for the observed improvement.","section":"Section 4.2 / Table 5"}],"minor_comments":[{"comment":"The sentence preceding Eq. (2) says that FPR is 'the ratio of the number of correct matches to the total number of MS/MS spectra', but the equation defines FPR as FP/N, i.e., the proportion of incorrect top-ranked PSMs. Please correct the sentence so that it matches the equation.","section":"Section 4.2 / Eq. (2)"},{"comment":"The text refers to 'SeqFix and SeqVar, (f11,f12)', but Table 1 lists f10 as SeqFix and f11 as SeqVar; the numbering and naming should be aligned consistently.","section":"Section 3.1"},{"comment":"Please specify the bin width for the fixed-length vectorization features f7-f10 and how the 4,000 bins are defined over the m/z range; without this detail the features are not fully reproducible.","section":"Section 3.1"},{"comment":"There are small typos: 'evolution set' should be 'evaluation set', and 'did not have the change' should be 'did not have the chance'.","section":"Section 5.2"},{"comment":"Please state explicitly whether 'FPR reduction after post-processing' is reported in absolute percentage points (e.g., 0.10 for GP-PSM) or as a relative reduction; the current text mixes '10% more spectra' with the absolute FPR change from 0.44 to 0.34.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of a computational-intelligence-meets-proteomics venue, and I see no reason to doubt the authors' good faith. However, the empirical support is thin: a single 120-spectrum evaluation set, a single selected GP individual, and no explicit statement that the learning and evaluation spectra are disjoint. All of these are fixable with additional experiments and reporting rather than fatal flaws. I would encourage the editor to ask for spectrum identifiers or data/code sharing, since the disjointness check and any significance testing cannot be independently audited without them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper applies GP symbolic regression to learn a PSM scoring function from Mascot-derived features, then uses it to re-rank PEAKS de novo candidates. The headline result is a drop in FPR from 0.44 to 0.34 on 120 spectra, driven mainly by recovering 20 of 25 previously missed correct peptides. That is a real empirical result, not a simulated one, and the comparison to RF and SVR is a reasonable sanity check.\n\nWhat is actually new is modest but real: GP-based symbolic regression for PSM rescoring, with an interpretable tree that discards redundant features. The workflow is clearly described, the feature set is sensible, and the authors are honest that for 28 of the 120 spectra no post-processing can help because the correct peptide is not in the candidate list. The use of an external benchmark dataset is a plus.\n\nThe soft spots are in the evaluation. First, the 120-spectrum evaluation set is small, and the paper reports no confidence intervals or significance tests. The difference between GP and SVR is 7 spectra, which could easily be noise. Second, the paper never states that the evaluation set is disjoint from the 10,000-spectrum learning set. Both are sampled from the same full factorial dataset, and without an explicit statement or a list of spectrum identifiers, the possibility of overlap cannot be ruled out. If overlap exists, the model would have seen the correct answer for some evaluation spectra, and the 10% improvement could be inflated. The authors likely intended them to be separate, but the burden is on them to say so. Third, the scoring function is trained on Mascot PSMs and applied to PEAKS PSMs; the paper discusses this transfer in Section 4.3 but does not test whether the feature distributions actually align. That is a known limitation, but it weakens the generality of the result.\n\nAlso, Equation 2 has a typo in the text: it says FPR is the ratio of correct matches, but the equation and the tables use false positives. Minor.\n\nNone of these flaws are fatal. The central mechanism is not circular: the regression target is Mascot score, and the evaluation is a downstream ranking task on PEAKS candidates. But the evidence is not strong enough to treat the 10% FPR reduction as a stable effect.\n\nWho is this for? Anyone working on post-processing for de novo peptide identification, or on GP applications in bioinformatics. It deserves a serious referee, but I would want the authors to address the disjointness and add error bars or at least a bootstrap.\n\nRecommendation: send to peer review, but flag the evaluation-set overlap and the lack of statistical validation as conditions.","headline":"A modest but real GP-based PSM rescoring paper; the 10% FPR claim needs better statistical backing and an explicit disjointness statement.","tokens_in":10416,"tokens_out":2918,"would_cite":false,"duration_ms":26486,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A genetic-programming scoring function re-ranks candidate peptide-spectrum matches and raises correct de novo identifications from 67 to 79 spectra out of 120.","keywords":["genetic programming","symbolic regression","peptide-spectrum match","de novo peptide sequencing","tandem mass spectrometry","scoring function","false positive rate","PSM re-ranking"],"falsifier":"Re-apply the learned function to a fresh, independently created set of MS/MS spectra with known peptide identities and count top-ranked correct matches; if the improvement over the original de novo rankings is not around 10 percentage points, the paper's central claim fails.","tokens_in":9455,"feed_emoji":"🧬","tokens_out":6861,"duration_ms":65363,"temperature":0.7,"pith_summary":"The paper proposes GP-PSM, a genetic-programming method that learns a scoring function for peptide-spectrum matches (PSMs) and uses it to re-rank the candidate peptides that a de novo sequencing algorithm proposes for each MS/MS spectrum. The authors aim to fix a weak point in de novo identification: the top-ranked candidate is often wrong, and existing scoring functions do not separate correct from incorrect matches well. They train a symbolic regression model on thousands of PSMs with known identities, using eleven features that describe how an experimental spectrum matches the theoretical spectrum of a candidate peptide. On an evaluation set of 120 spectra, re-ranking with the learned function raises correct identifications from 67 to 79 and lowers the false positive rate from 0.44 to 0.34, a 10-percentage-point improvement over the unprocessed de novo output. A careful reader would care because better PSM scoring directly improves confidence in peptide identifications without changing the sequencing instrument or algorithm.","feed_headline":"Learned re-ranker improves peptide identification by 10%","feed_subtitle":"A genetic-programming scorer re-ranks candidate peptides, lifting correct identifications on difficult MS/MS spectra.","key_machinery":"The central object is the GP-evolved scoring function, a tree-structured arithmetic expression over eleven PSM features: matched peak intensity, numbers of matched and unmatched peaks, precursor mass difference, consecutive b-ion and y-ion matches, cosine similarity, Euclidean distance, Hamming distance, and two SEQUEST-style fixed- and variable-length vector scores. The GP runs as symbolic regression: fitness is the relative sum of squared error between the evolved score and the known database-search confidence score, and the terminal set includes the eleven features plus constants, with protected arithmetic as the function set. The evolved tree re-scores each candidate peptide in a spectrum's output list; the highest-scoring candidate is then reported as the identification. Its work is to compress the eleven weak individual similarity signals into one ranking that separates the true peptide from near-synonymous decoys.","core_discovery":"The central claim is that a PSM scoring function evolved by genetic programming can serve as a post-processing re-ranker that outperforms the raw confidence scores of a de novo sequencing algorithm and also beats two regression baselines, Random Forest and Support Vector Regression. The learned function combines eleven spectrum-peptide similarity features into a single score, and when each spectrum's five candidate peptides are re-ranked by that score, the correct peptide reaches the top rank more often than it did under the original de novo score. Concretely, on 120 benchmark spectra the method identifies 79 spectra correctly versus 67 before re-ranking, identifies 20 of the 25 correct matches that the original tool had ranked below the top, and reduces the false positive rate from 0.44 to 0.34. The paper presents this as evidence that GP's nonlinear, feature-selecting symbolic expressions are better discriminators for full-length peptide assignments than hand-built or linear scoring functions.","pith_inferences":["A natural extension is to train the same re-ranker on decoy-based labels from database search, removing the need for a validated ground-truth set and allowing much larger training corpora.","Because the scorer never uses the original de novo confidence score, the same model could be applied to candidate lists from any de novo tool; testing on other tools would reveal whether the 10% gain is tied to one tool's candidate distributions.","The fact that most missed correct matches were recovered from lower ranks suggests the bottleneck in de novo identification may be ranking rather than candidate generation, a direction the paper does not explicitly pursue."],"forward_implications":["De novo peptide identification can be improved in a post-processing step, without altering the sequencing algorithm or acquisition parameters.","A 10-percentage-point false-positive-rate reduction on 120 spectra implies that re-ranking with a learned scorer can recover correct matches that the original de novo tool had ranked below the top.","The GP-evolved scoring function can be reused as a fitness function inside other de novo sequencing algorithms, not just as a post-hoc re-ranker.","The explicit feature selection in the evolved tree identifies which spectral properties matter most, making the scoring function interpretable rather than a black box."],"supporting_citations":[{"why":"Documents that fully correct de novo peptide predictions remain below 70%, motivating the need for a post-processing scorer.","marker":"[1]"},{"why":"Supplies the first six PSM features (matched intensities, peak counts, mass difference, terminal ion matches) that the GP scoring function combines.","marker":"[12]"},{"why":"Supplies the SEQUEST-style fixed- and variable-length vector features used to measure spectrum-peptide similarity.","marker":"[13]"},{"why":"Provides the evolutionary computation library used to implement and run the genetic programming search.","marker":"[14]"},{"why":"Provides the benchmark MS/MS spectra with known peptide identifications used to build the learning and evaluation sets.","marker":"[15]"},{"why":"Provides the database search method that establishes ground-truth peptide assignments and the target scores for regression.","marker":"[16]"},{"why":"Generates the five candidate peptides per spectrum that the learned scoring function re-ranks in the evaluation phase.","marker":"[17]"}],"fun_headline_variants":["GP re-ranker boosts correct peptide IDs by 10% on benchmark","Evolved scoring function re-ranks peptide matches for better ID","GP-PSM re-ranking lifts correct peptide calls by 10 percent","Post-hoc GP scorer trims false positives in peptide identification","Genetic programming re-ranker outdoes regression baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scoring function is trained on matches produced by a database search and then applied to matches produced by a different de novo sequencing algorithm; if those two kinds of matches score differently on the same features, the learned function will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["GP re-ranker boosts correct peptide IDs by 10% on benchmark","Evolved scoring function re-ranks peptide matches for better ID","GP-PSM re-ranking lifts correct peptide calls by 10 percent","Post-hoc GP scorer trims false positives in peptide identification","Genetic programming re-ranker outdoes regression baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1855,"prompt_tokens":979,"completion_tokens":876,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":787}},"tokens_in":595,"tokens_out":876,"duration_ms":8108,"temperature":1.0,"reasoning_tokens":787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:55:55.765493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-apply the learned function to a fresh, independently created set of MS/MS spectra with known peptide identities and count top-ranked correct matches; if the improvement over the original de novo rankings is not around 10 percentage points, the paper's central claim fails.","supporting_citations":[{"cited_title":"psite: Amino acid confidence evaluation for quality control of de novo peptide sequencing and modification site localization","cited_arxiv_id":null,"evidence_quote":"Documents that fully correct de novo peptide predictions remain below 70%, motivating the need for a post-processing scorer."},{"cited_title":"GA-Novo: De novo peptide sequencing via tandem mass spectrometry using genetic algorithm","cited_arxiv_id":null,"evidence_quote":"Supplies the first six PSM features (matched intensities, peak counts, mass difference, terminal ion matches) that the GP scoring function combines."},{"cited_title":"An approach to correlate tandem mass spectral data of peptides with amino acid sequences in a protein database","cited_arxiv_id":null,"evidence_quote":"Supplies the SEQUEST-style fixed- and variable-length vector features used to measure spectrum-peptide similarity."},{"cited_title":"DEAP: Evolutionary algorithms made easy.Jour- nal of Machine Learning Research , 13:2171–2175, 2012","cited_arxiv_id":null,"evidence_quote":"Provides the evolutionary computation library used to implement and run the genetic programming search."},{"cited_title":"A comprehensive full factorial lc-ms/ms proteomics benchmark data set","cited_arxiv_id":null,"evidence_quote":"Provides the benchmark MS/MS spectra with known peptide identifications used to build the learning and evaluation sets."},{"cited_title":"Probability-based protein identification by search- ing sequence databases using mass spectrometry data.electrophoresis, 20(18):3551– 3567, 1999","cited_arxiv_id":null,"evidence_quote":"Provides the database search method that establishes ground-truth peptide assignments and the target scores for regression."},{"cited_title":"Peaks: powerful software for peptide de novo sequencing by tandem mass spectrometry","cited_arxiv_id":null,"evidence_quote":"Generates the five candidate peptides per spectrum that the learned scoring function re-ranks in the evaluation phase."}],"review_version":1}