{"id":"277c6766-684c-44fb-9391-7b37c9aba7e4","arxiv_id":"2501.06669","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"A Transformer reaction predictor evaluated on document-, author-, time-, and reaction-class splits reveals that in-distribution benchmarks overstate real-world accuracy and that models only partially extrapolate to unseen reaction types.","lead":"This paper shows that standard random-split benchmarks make chemical reaction prediction models look better than they are: accuracy falls from 65% to 55% top-1 when moving to splits that respect author structure. It offers a panel of out-of-distribution evaluation tasks and analyses that give a more realistic picture for drug discovery and synthesis planning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NameRxn label noise may leave Suzuki-like reactions in the All Suzuki training set, inflating the reported zero-shot extrapolation accuracy.","rationale":"The paper's most distinctive contribution is the claim that reaction predictors retain some ability to extrapolate to entirely unseen reaction classes, as quantified by the NameRxn splits. The 'All Suzuki' result (51% top-1) is the headline number for this claim and is the most directly tied to the title's 'novel chemistry' framing. That number is only interpretable as zero-shot if the training set genuinely contains no Suzuki reactions. The paper's own description in S1.4 shows only a 74% NameRxn coverage and a reliance on removing '0.0' reactions; it does not address mislabeled reactions that remain under other class codes. Because NameRxn is a rule-based classifier, reactions can be assigned to a different named class when reagents are missing or conditions differ, leaving mechanistically equivalent examples in training. If even a modest fraction of true Suzuki reactions remain, the reported extrapolation accuracy could be substantially inflated. The proposed substructure-based retraining test directly quantifies this leakage. The reader's CONDITIONAL verdict is appropriate: the document/author and time splits are well designed and support the over-optimism claim, but the reaction-type extrapolation numbers should be treated as provisional until the NameRxn label assumption is checked. This concern does not move the overall verdict, but it sharpens the condition under which the paper should be fully trusted.","tokens_in":24980,"tokens_out":9148,"duration_ms":89136,"concrete_test":"Using the released cleaning and split code with Pistachio 2022Q4, identify all reactions in the All Suzuki training split that contain both a boronic acid/ester and an aryl/vinyl halide or sulfonate (Suzuki-compatible partners) by RDKit substructure search, regardless of NameRxn label, and manually inspect a random sample of 100 such flagged reactions. Then remove all such reactions from training, retrain the same BART model with the same hyperparameters, and evaluate on the same held-out All Suzuki test set. If top-1 accuracy drops substantially (e.g., >10 points from 51%) or the flagged count is large, the zero-shot extrapolation claim is inflated by label leakage; if accuracy is essentially unchanged and the count is negligible, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 and S1.4 define reaction-type splits by removing the named NameRxn classes and all uncategorized '0.0' reactions. The validity of the 'All Suzuki' split as a zero-shot test depends on NameRxn labels being both complete and accurate: every Suzuki coupling must be assigned to class 3.1. If some Suzuki reactions are mislabeled as another named class (e.g., a generic cross-coupling or a different named reaction), they remain in the 1M-reaction training set. The paper states that NameRxn covers only 74% of the cleaned dataset (S1.4); removal of '0.0' reactions excludes uncategorized examples but does not address mislabeled ones. Consequently, the reported 51% top-1 accuracy on held-out All Suzuki reactions (Fig. 4, Table S2.4) may partly reflect training on near-identical Suzuki examples, inflating the apparent extrapolation ability. The same issue affects the Buchwald–Hartwig time-split test set (Section 2.2.1), which is built from NameRxn codes 1.3.1–1.3.4 and 1.9.43.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates a BART-based encoder-decoder transformer for reaction product prediction on a cleaned, deduplicated subset of the Pistachio patent dataset, comparing several out-of-distribution (OOD) split strategies against the standard random split. It reports that random splits overestimate top-1 accuracy (65%) relative to document-based (58%) and author-based (55%) splits; that time-based splits show accuracy declining with extrapolation distance from the training cutoff; and that reaction-type splits based on NameRxn classes give mixed evidence about zero-shot extrapolation, with Suzuki subtypes being easier than Grignard ester or Heck reactions. The authors further analyze why certain classes are harder, such as the need for multi-step reasoning in Grignard double additions and stereochemical/regiochemical sensitivity in Heck reactions, and they provide a fingerprint-distance analysis of the distribution shifts induced by each split type. The paper explicitly positions itself as a benchmark and evaluation study, releasing code for data cleaning and model training.","tokens_in":25113,"tokens_out":10103,"duration_ms":94293,"significance":"If the findings hold, the paper makes a useful and timely contribution by quantifying how much standard random splits overstate reaction-prediction performance and by proposing a suite of complementary OOD splits (document, author, time, reaction-type) that better reflect deployment. The careful dataset cleaning, the controlled training-size comparisons, and the fingerprint-based analysis of split-induced distribution shifts are strengths, as is the public release of code. The time-split protocol with multiple cutoffs and the Buchwald–Hartwig case study are particularly nice. However, the reaction-type extrapolation results—which underpin the paper's statements about reaction discovery—depend on the accuracy and completeness of NameRxn labels, and the paper does not validate those labels. The study also uses a single model and single training runs, so the quantitative accuracy gaps should be interpreted with appropriate caution.","major_comments":[{"comment":"The reaction-type splits, especially the 'All Suzuki' zero-shot evaluation, are not robust to NameRxn label noise. The paper removes only uncategorized reactions (NameRxn '0.0') before forming splits, explicitly acknowledging this is to 'avoid inadvertently training on reactions that might be similar to those that we are trying to exclude.' However, NameRxn covers only 74% of the cleaned dataset (stated in S1.4), so a Suzuki reaction mislabeled as, say, a generic cross-coupling or another named class would remain in the 1M-reaction training set. This could inflate the reported extrapolation accuracy (e.g., 51% top-1 for All Suzuki) and weaken the paper's conclusion that models possess some ability to extrapolate to unseen reaction types. The same issue affects the Buchwald–Hartwig test set in S1.3, which is constructed from NameRxn codes 1.3.1–1.3.4 and 1.9.43; BH reactions mislabeled under other codes could appear in the training sets of the time-split models. To support the central claims of Section 2.3 and Section 2.2.1, the authors should validate NameRxn assignments on a representative sample, or use an independent reaction classifier (e.g., template matching or mechanistic rules) to detect and remove reactions of the held-out type from the training set, then re-evaluate. A sensitivity analysis that relaxes the definition of the held-out class would also help establish how much the results depend on label completeness.","section":"S1.4, Fig. 4, Table S2.4"},{"comment":"The headline accuracy numbers are reported from single training runs with no uncertainty intervals. The gap between random, document, and author splits (65%, 58%, 55%) is large enough that it is probably real, but the time-split results in Figure 3A show substantial inter-year variability, and the paper itself attributes part of this to 'variability from the random selection of test sets.' Without multiple seeds, standard errors, or confidence intervals, it is difficult to assess whether smaller differences—for example between the document- and author-based splits, or between adjacent time cutoffs—are meaningful. Since the paper's central claim is that random splits overestimate performance, and since it makes quantitative claims such as 'the drop of ~10% accuracy,' the authors should provide at least bootstrap confidence intervals over the test set, or ideally repeat training with a few seeds for the main splits. This would not change the qualitative direction of the results, but it would make the reported magnitudes more interpretable.","section":"Section 2.1, Fig. 2, Table S2.2; Section 2.2, Fig. 3"}],"minor_comments":[{"comment":"The phrase 'a model trained and evaluated on an on reactions split' is grammatically awkward and should be rephrased, e.g., 'a model trained and evaluated on a reaction-level (random) split.'","section":"Section 2.1, first paragraph"},{"comment":"The description of the dashed line is confusing: the caption says 'the dashed line indicates model performance when the model is extrapolating,' but in the figure the dashed line appears to be a boundary separating interpolative and extrapolative regions. Please clarify what the dashed line represents and how the reader is to interpret the region to its right.","section":"Figure 3A caption"},{"comment":"The small SMILES structures and the labels 'True' and 'False' are not self-explanatory; the reader has to infer that 'True' is the correct product and 'False' is an incorrect model prediction. A legend or a sentence in the caption would help.","section":"Figure 5B and 5C"},{"comment":"The reaction fingerprint is defined as the difference between product and reactant fingerprints, but the text then refers to 'cosine distance' between these difference vectors; it would be clearer to state explicitly that the cosine distance is computed on these difference vectors.","section":"S1.6"},{"comment":"The paper notes that the NameRxn system is not the only possible classification scheme and that different schemes may lead to different results; this is a good caveat, but it would be even more informative to include one such alternative (e.g., RXNO or template-based classes) for at least one held-out class, to test the sensitivity of the extrapolation conclusions to the choice of taxonomy.","section":"S1.4"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical evaluation paper whose central message—that random splits overestimate reaction-prediction performance—is well supported and likely to be of interest to the community. The main concern is the reliance on NameRxn labels for the reaction-type and Buchwald–Hartwig analyses without validating label accuracy; this is especially salient because the paper itself acknowledges the 74% coverage limitation. The paper's own limitation statements do not fully address the risk that mislabeled reactions remain in the training set and inflate the zero-shot extrapolation numbers. I would encourage the editor to require the authors to address this issue before publication, either by adding a validation study or by clearly reframing the reaction-type results as conditional on NameRxn accuracy. The single-run reporting is a secondary but real weakness that should also be corrected. The paper is otherwise well within scope and the authors' framing is appropriately modest."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core claim holds up: random splits overstate how well reaction predictors generalize. The 65/58/55 top-1 pattern across reaction-, document-, and author-based splits is consistent and the fingerprint distance analysis supports the interpretation that related reactions leak across the train/test boundary. This is a genuinely useful contribution, and I don't know of another single study that puts document, author, time, and reaction-class splits side by side with mechanistic probes. The double-addition Grignard reframing and the Heck stereo/regio analysis are nice pieces of evidence that go beyond just reporting accuracy numbers.\n\nThe soft spots are real but not load-bearing. Headline numbers come from single runs without error bars; one model family; proprietary data. The stress-test concern about NameRxn label noise is fair: coverage is only 74%, and removing '0.0' reactions does not catch mislabeled Suzuki couplings that stay in the training set. So the reported 51% top-1 on All Suzuki may be inflated. But this caveat applies to the quantitative strength of the reaction-type extrapolation results, not to the broader qualitative conclusion that in-distribution splits are over-optimistic. The time-split Buchwald–Hartwig analysis could be similarly affected, though the monotonic relationship between training counts and accuracy is robust either way.\n\nI would read the reaction-type accuracy numbers with caution, but the paper itself is transparent about many limitations—patent-only data, non-SOTA model, the arbitrariness of cleaning thresholds. The code is public, which counts for something even if the underlying data are proprietary. This is a paper for people building reaction prediction benchmarks and for practitioners who want a more honest estimate of deployed model performance. It deserves a serious referee, and the path to acceptance is clear: add multi-seed runs or confidence intervals, and either quantify NameRxn label noise or soften the zero-shot claims accordingly. I'd take it to the reading group and would likely cite it.","headline":"A careful empirical demonstration that random splits flatter reaction predictors; the NameRxn label-noise caveat is real but does not sink the central conclusion.","tokens_in":25717,"tokens_out":1474,"would_cite":true,"duration_ms":16288,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Random splits make reaction-prediction models look about 10 accuracy points better than they are, and the paper's split panel shows which reaction classes the models can and cannot extrapolate to.","keywords":["reaction prediction","out-of-distribution generalization","data split design","document-level split","time-based split","reaction-type extrapolation","NameRxn","transformer sequence model"],"falsifier":"Re-run the All Suzuki holdout with an independent, mechanism-based classifier that identifies Suzuki couplings by bond changes and catalyst rather than by the vendor label; if the model still scores near 51% top-1 after every mechanistically equivalent reaction is provably absent from training, the extrapolation claim stands, whereas a sharp accuracy jump after purging mislabeled or unclassified examples would show the result was partly leakage.","tokens_in":24729,"feed_emoji":"🧪","tokens_out":13980,"duration_ms":104033,"temperature":0.7,"pith_summary":"This paper argues that the standard way of benchmarking machine-learning reaction predictors—randomly splitting a reaction database into training and test sets—inflates reported accuracy, because a single patent or research team often contributes many near-identical reactions that end up on both sides of the split. To get realistic numbers, the authors evaluate a SMILES-based transformer on a cleaned 2.8-million-reaction version of the Pistachio database using splits that respect the data's actual structure: by document, by author, by publication year, and by entire reaction class. A model that scores 65% top-1 accuracy on a random split drops to 58% when whole documents are held out and 55% when whole authors are held out, and accuracy decays as the time gap between training data and test data widens. When entire reaction classes are withheld, the model extrapolates well to some classes (74–83% top-1 for specific Suzuki subtypes) and poorly to others (7–10% for Grignard ester and Heck), and the paper traces those failures to step composition and to stereochemistry and regiochemistry rather than to missing chemical knowledge. The paper's point is that reaction prediction is not as close to solved as leaderboard numbers suggest, and that this family of out-of-distribution evaluations should become the standard way to measure progress toward reaction discovery.","feed_headline":"65% to 55%: realistic splits shrink reaction-model accuracy","feed_subtitle":"Patent documents and research teams cluster similar reactions, so realistic splits expose where current models fall short.","key_machinery":"The carrying object is a sequence of split constructions over a cleaned, deduplicated version of the Pistachio patent reaction database, each evaluated with the same encoder-decoder transformer (BART architecture with SMILES tokenization, closely related to the Molecular Transformer). Three split mechanisms do the principal work: document- and author-based splits, which stop a patent's series of closely related scope or structure-activity reactions from straddling the train/test boundary; a ladder of time cutoffs from 1996 to 2020 paired with held-out test sets for each year, making extrapolation distance a measurable axis; and NameRxn reaction-class splits, which hold out complete classes (Grignard ester, Heck, Chloro Suzuki, Triflyloxy Suzuki, and all Suzuki) and contrast them with a baseline given 1,000 in-class examples, separating a class's intrinsic difficulty from its extrapolation difficulty. A diagnostic supplement computes fingerprint distances from each test reaction to its five nearest training neighbors in reactant space and in reaction-transform space, which distinguishes splits that shift the substrate distribution from splits that shift the transform distribution and correlates with the observed difficulty.","core_discovery":"The central claim, on the paper's own terms, is that accuracy figures reported on standard in-distribution benchmarks are over-optimistic, and that a panel of out-of-distribution splits—by document, by author, by time, and by reaction class—gives a more faithful measure of how reaction predictors behave in deployment and whether they can extrapolate to novel chemistry. A BART-based encoder-decoder transformer, close in design to the Molecular Transformer, reaches 65% top-1 accuracy on a random reaction-level split of Pistachio patent data but only 58% on a document-level split and 55% on an author-level split. On time-based splits, accuracy declines as the extrapolation distance grows, yet a model trained only on reactions reported through 1996 still makes correct top-1 predictions some 25 years out; on a static held-out set of Buchwald–Hartwig reactions its top-1 accuracy climbs from 5% to above 60% as later training cutoffs add more in-class examples. On reaction-type splits, held-out Chloro Suzuki and Triflyloxy Suzuki classes are extrapolated with less than a 10-point drop (74% and 83% top-1), largely because other Suzuki reactions remain in training, whereas removing the whole Suzuki family cuts All Suzuki accuracy to 51%. Grignard ester and Heck holdouts are far harder (10% and 7% top-1), and the paper shows these failures are partly compositional and representational: iterating the Grignard double addition as two single steps raises accuracy from 4.5% to 49.7%, and ignoring stereochemistry and then regiochemistry raises the Heck extrapolation score from 7% to 42%.","pith_inferences":["The split construction is author- and document-dense by design, so the 10-point gap probably understates the inflation of random splits on datasets with broader author diversity, such as academic literature or high-throughput experimentation collections.","The Grignard double-addition result suggests a general design principle the paper leaves implicit: reaction predictors may gain more from a step-decomposition objective, in which the model predicts one elementary transform and feeds the product forward, than from more training data.","The nearest-neighbor fingerprint distances offer a cheap pre-registration rule for future benchmarks: large reaction-transform distances should predict behavior like the Grignard ester, Heck, and All Suzuki cases, while large reactant distances should predict behavior like the document and author splits.","A natural next experiment not performed here is to run the same split panel on graph-based or pretrained-language-model reaction predictors; the ordering of which classes extrapolate well could serve as a fingerprint of a model's inductive bias."],"forward_implications":["Random-split leaderboard numbers overstate deployment accuracy by roughly 7–10 top-1 points, so future reaction-prediction papers should report document- and author-level splits alongside them.","Time-based evaluation bounds a model's useful lifetime: accuracy decays with the gap between training cutoff and test year, but the non-zero accuracy of a 1996-trained model decades later shows future reactions are partly predictable from past chemistry.","Extrapolation to an unseen reaction class works when a sibling family stays in training (Chloro and Triflyloxy Suzuki) and largely fails when the whole family is removed (All Suzuki, 51% top-1), which means claims about reaction discovery must be tested with full-family holdouts.","Much of the observed failure is recoverable: iterating the Grignard double-addition prediction as two single steps raises accuracy from 4.5% to 49.7%, and evaluation that ignores stereochemistry and then regiochemistry raises Heck extrapolation from 7% to 42%.","The three split families answer different questions—retrospective deployment, prospective use, and reaction discovery—so they should be reported together rather than treated as interchangeable benchmarks."],"supporting_citations":[{"why":"Pistachio patent reaction database; supplies the reactions whose document, author, year, and NameRxn structure defines every split in the study.","marker":"[52, 49]"},{"why":"Molecular Transformer; the benchmark model family the paper's BART implementation is compared with, and whose reported benchmark accuracies are the numbers the paper argues are over-optimistic.","marker":"[65]"},{"why":"BART encoder-decoder architecture; the model class used for all experiments.","marker":"[44]"},{"why":"SMILES tokenization scheme from the original sequence-to-sequence reaction model; defines the input representation of the transformer.","marker":"[64]"},{"why":"NameRxn hierarchical reaction classification; its named classes define the held-out reaction-type splits and the Buchwald–Hartwig test set.","marker":"[53, 40]"},{"why":"First reports of the Buchwald–Hartwig reaction; anchor the case study that ties prediction accuracy to the number of in-class reactions in training.","marker":"[17, 26, 54]"},{"why":"Time-split cross-validation method; the precedent the paper generalizes into a ladder of time cutoffs with per-year held-out test sets.","marker":"[71]"}],"fun_headline_variants":["Realistic splits cut reaction accuracy from 65% to 55%","Reaction models fail to generalize to novel reaction classes","Patent and author splits reveal overoptimistic reaction benchmarks","Iterative steps rescue Grignard extrapolation 4.5% to 49.7%","Time splits show reaction models lag on future reactions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every reaction-type result assumes NameRxn tagging is complete, so that removing the named classes plus all uncategorized reactions strips every training example of the held-out transform; if reactions that are mechanistically the same survive in training under another label, the reported extrapolation accuracies such as 51% top-1 on all Suzuki are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Realistic splits cut reaction accuracy from 65% to 55%","Reaction models fail to generalize to novel reaction classes","Patent and author splits reveal overoptimistic reaction benchmarks","Iterative steps rescue Grignard extrapolation 4.5% to 49.7%","Time splits show reaction models lag on future reactions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1857,"prompt_tokens":1107,"completion_tokens":750,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":723,"completion_tokens_details":{"reasoning_tokens":661}},"tokens_in":723,"tokens_out":750,"duration_ms":64946,"temperature":1.0,"reasoning_tokens":661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:55:31.531323+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the All Suzuki holdout with an independent, mechanism-based classifier that identifies Suzuki couplings by bond changes and catalyst rather than by the vendor label; if the model still scores near 51% top-1 after every mechanistically equivalent reaction is provably absent from training, the extrapolation claim stands, whereas a sharp accuracy jump after purging mislabeled or unclassified examples would show the result was partly leakage.","supporting_citations":[],"review_version":1}