{"id":"24f2c2ea-77b1-41bc-a3ca-c9467bd48759","arxiv_id":"2504.18340","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A chemistry fine-tuned LLM, Chemma, claims top benchmark results on retrosynthesis and yield prediction, and helped find working conditions for an unreported Suzuki coupling in 15 experiments.","lead":"Chemma, a fine-tuned large language model trained on 1.28 million chemistry question-answer pairs, answers questions about reactions: predicting products, planning retrosyntheses, recommending conditions, and estimating yields. In a wet-lab test on an unreported coupling reaction, model suggestions helped chemists find a 67% yield within 15 runs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The wet-lab success is confounded: the solvent switch that produced 67% yield was made by chemists, not Chemma, and the paper's own limitations state Chemma failed without chemist feedback, so the 15-run claim does not isolate the model's contribution.","rationale":"I agree with the reader's weakest assumption. This is the single most load-bearing issue because the title and abstract's distinctive claim is acceleration of organic synthesis, and the only wet-lab evidence for open-space exploration is Fig. 5D. The paper itself contains the admissions that make the concern precise: round 0 failed, and the solvent change was a chemist's suggestion. This is not a disagreement with consensus or a stylistic issue; it is an internal inconsistency between the stated 'autonomous exploration' capability and the described human-AI workflow. If a controlled ablation had been included and still showed Chemma's ligands were necessary, the verdict could be ACCEPT. Without it, CONDITIONAL is appropriate. Secondary concerns, such as the anomalously high 72.2% top-1 retrosynthesis result compared with NAG2G's 55.1%, and the absence of released code or weights, reinforce the need for verification but do not replace the attribution problem as the central issue. I would keep the reader's CONDITIONAL verdict and recommend the concrete ablation plus artifact release as conditions for a higher verdict.","tokens_in":29117,"tokens_out":4674,"duration_ms":48156,"concrete_test":"Run a controlled ablation for the same (1-benzoylpiperidin-2-yl)boronic acid ester + 4-bromobiphenyl coupling: fix the round-0 expert conditions and the rule 'if all ligands are below 15%, switch solvent to 1,4-dioxane', but replace Chemma's three round-1 ligand suggestions with random draws (or GPT-4 suggestions) from the same ligand class. Repeat several independent 15-run campaigns. If a no-Chemma baseline also finds conditions giving isolated yield >=50% within 15 runs, the 67% result cannot be attributed to Chemma; if no baseline does, the attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the paper's flagship open-space demonstration does not establish that Chemma caused the acceleration. In 'Exploring and optimizing open reaction spaces with Chemma' (Fig. 5D), round 0 begins with conditions selected by chemists' experience; Chemma contributes only two of nine ligands, and all nine give below 15% yield. The improvement to 67% occurs only after the authors state: 'following the suggestion of chemists, we change the solvent from p-Xylene to 1,4-Dioxane.' Thus the decisive variable change was human, not model-generated. The Limitations section further concedes: 'For an unreported reaction, Chemma is unable to generate effective answers without chemists' feedback (round 0 in Fig. 5D).' Because no ablation or control replaces Chemma with random or expert-only ligand selection under the same human solvent rule, the observed 15-run success is compatible with Chemma adding little over the chemists' prior knowledge. The central claim of 'autonomous experimental exploration and optimization in open reaction spaces' therefore rests on an uncontrolled anecdote, not on a demonstrated causal contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Zhang et al. present Chemma, a fully fine-tuned LLaMA-2-7B model trained on 1.28 million reaction Q&A pairs for tasks including forward prediction, retrosynthesis, condition generation, yield and selectivity prediction, and reaction optimization. The paper claims state-of-the-art top-1 accuracy on USPTO-50k retrosynthesis (72.2%), competitive yield and selectivity predictions on HTE/ELN/literature datasets, improved Bayesian optimization when augmented with Chemma-generated yields (Chemma-BO), and an active-learning wet-lab demonstration in which an unreported Suzuki-Miyaura coupling of cyclic aminoboronates with aryl halides is optimized within 15 runs to 67% isolated yield.","tokens_in":29321,"tokens_out":7411,"duration_ms":73265,"significance":"If fully supported, the work would be a useful demonstration that a domain-adapted LLM can serve as a practical synthesis assistant. The paper has notable strengths: a large multi-task chemistry instruction dataset, evaluation on standard public benchmarks such as USPTO-50k and ORD-derived HTE data, a publicly accessible web service, and a real wet-lab demonstration on a previously unreported reaction type. However, the evidence as presented does not fully support the strongest claims. The wet-lab success is confounded by human expert decisions, and several benchmark comparisons lack statistical error bars or significance tests. The Chemma-BO results are in-silico on existing HTE datasets, and the evaluation protocol is not fully specified with respect to potential data leakage. The central contribution therefore needs either additional control experiments or ablations, or a careful reframing of the claims from autonomous exploration to human-AI collaboration.","major_comments":[{"comment":"The 15-run wet-lab demonstration does not establish that Chemma caused the observed acceleration. The text states that in round 0 \"the initial combinations of starting condition variables are selected by chemists' experience\" and that Chemma proposed only two of nine ligands, all of which gave yields below 15%. The improvement to 67% occurs only after the authors write \"following the suggestion of chemists, we change the solvent from p-Xylene to 1,4-Dioxane.\" The Limitations section further concedes that \"For an unreported reaction, Chemma is unable to generate effective answers without chemists' feedback (round 0 in Fig. 5D).\" Because no control experiment replaces Chemma with random or expert-only ligand selection under the same human solvent rule, the claim of \"autonomous experimental exploration and optimization in open reaction spaces\" (abstract) is not supported. The authors should either provide such an ablation or explicitly revise the claim to describe a human-AI collaborative case study rather than autonomous model-driven exploration.","section":"Exploring and optimizing open reaction spaces with Chemma; Fig. 5D and Limitations"},{"comment":"The headline claim that \"Chemma surpasses the best-known results\" is not supported by the reported statistics. The retrosynthesis comparison (Fig. 3A) is a single fixed test set with one model run; no standard deviations, repeated-seed results, or significance tests are reported for Chemma or the baselines. The yield-prediction comparisons (Fig. 3D) report R2 and RMSE for individual random splits, and while Extended Data Fig. 4 shows that performance varies across split strategies, no confidence intervals or statistical tests are given. For the yield claim, the manuscript does not actually name a \"best-known\" baseline against which Chemma is compared in the main text. Please add error bars or repeated-run statistics, and explicitly identify the state-of-the-art numbers being surpassed.","section":"Retrosynthesis; Yield prediction; Fig. 3A and Fig. 3D"},{"comment":"The Chemma-BO evaluation is at risk of circularity because the manuscript does not specify whether the Chemma yield model used to score the full reaction space was trained on the same HTE dataset from which the observed yields are drawn. If the yield predictor has already seen many of the evaluated conditions during training, the \"generated yields\" are not a fair prior for a new optimization campaign. The authors should state the training split of the yield model, confirm that BO validation conditions are disjoint from its training data, and report the variance of the ten repeated runs (e.g., shaded confidence bands) in Fig. 4E-F. As written, the claim that Chemma \"significantly improves\" BO is not quantitatively supported.","section":"Implementation of Chemma-BO; Fig. 4E-F"},{"comment":"The active-learning workflow as described is not fully algorithmic and is therefore difficult to reproduce. The text repeatedly relies on unspecified chemist interventions, including choosing when to change base-solvent combinations, deciding between zero-shot and in-context-learning prompts, selecting when to fine-tune Chemma, and judging when a ligand suggestion should be regenerated. For example, in the C-H arylation example the authors write \"we proceed to randomly select a new base-solvent combination\" without specifying a rule. To support the claim of an active-learning framework, the authors should provide a precise pseudocode description of the decision rules and prompt-selection procedure, or clearly label the workflow as human-mediated exploration with Chemma as a suggestion generator.","section":"Active learning framework of Chemma for reaction optimization; Figs. 5B-C"}],"minor_comments":[{"comment":"The text says Chemma generates AdBrettPhos in run 1 but then states \"The achieved yield for this generated ligand XPhos is 4.95%\"; the ligand name is inconsistent and should be corrected.","section":"Fig. 5C and Extended Data Fig. 8C"},{"comment":"The phrase \"start-of-art performance\" should be corrected to \"state-of-the-art performance,\" and the duplicated sentence in the first paragraph of the introduction should be removed.","section":"Discussion"},{"comment":"The text says 30% of reactions are used as an out-of-sample test set for the Suzuki-Miyaura dataset but then states that four of the 28 substrate combinations are selected as testing cases; the relationship between these two statements should be clarified.","section":"Fig. 3D and Performance of Chemma on open benchmark data"},{"comment":"The phrase \"All reactions for evaluation are posted in Extended Data Fig. 2\" should read \"presented\" or \"shown,\" and the figure callouts should be checked for consistency with the referenced panels.","section":"Performance of Chemma on open benchmark data"},{"comment":"Extended Data Fig. 10 contains typos such as \"chemcial assistant\" and \"specifc chemical reaction\"; the prompt templates should be carefully proofread.","section":"Supplementary prompts"}],"recommendation":"major_revision","confidential_remarks":"The paper's strongest claims, namely state-of-the-art performance across tasks and autonomous wet-lab exploration, are broader than the evidence they provide. The underlying model and the real wet-lab demonstration are of interest, but the current version overstates the model's autonomy and lacks statistical support for several benchmark claims. I would advise the editor that a major revision that adds a control or ablation for the wet-lab claim, clarifies the Chemma-BO evaluation protocol, and tempers the language from \"autonomous\" to \"human-AI collaborative\" would make the central claims defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The abstract overstates the open-space result. The benchmark work is real and deserves a serious referee.\n\nThe genuinely new piece is the combination: a fully fine-tuned 7B LLaMA on 1.28M reaction Q&A pairs, an RLHF reward model for condition ranking, and a two-stage regression head that extracts embeddings for yield/selectivity prediction. The single-step retrosynthesis number — 72.2% top-1 on USPTO-50k without reaction class or templates — beats the cited NAG2G baseline by a wide margin, and the yield predictions on published HTE datasets (R2 ~0.8) are credible without DFT descriptors. The Chemma-BO idea — using LLM-generated yields as a prior for GP-based Bayesian optimization — shows a consistent acceleration over plain BO on two datasets, with ten repeats each. That is a contribution.\n\nThe soft spot is the open-space wet-lab campaign. The paper itself notes that in round 0 Chemma's suggestions all gave <15% yield, and the solvent change from p-xylene to 1,4-dioxane — the decisive variable — came \"following the suggestion of chemists.\" The Limitations section concedes Chemma was \"unable to generate effective answers without chemists' feedback.\" So the 15-run success cannot be attributed to the model alone. The autonomous exploration claim goes further than the evidence supports. The other flaw is the absence of error bars or significance tests for the benchmark claims, and no code or model weights released, only a hosted service. Those need fixing before the claims are fully verifiable.\n\nThe paper is not sloppy; the limitations are stated openly, and the design of the training pipeline is sensible. The citation pattern is appropriate — the authors engage with Jablonka, Boiko, Shields, and the HTE literature. But the headline claim needs reining in, and the experimental section should either add a control or be presented as human-AI collaboration rather than autonomous model-driven search.\n\nWho this is for: anyone working on LLMs for chemistry or reaction optimization. It deserves peer review, not desk rejection. Send it out, and ask for error bars, a released artifact, and either a control experiment or softened language for the open-space claim.","headline":"Worth refereeing: the benchmark and Chemma-BO results are solid, but the 'autonomous open-space' claim is undercut by the chemists' solvent switch and the paper's own Limitations.","tokens_in":29944,"tokens_out":2178,"would_cite":true,"duration_ms":22144,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that a chemistry-tuned language model, trained on 1.28 million reaction Q&A pairs, outperforms prior methods on retrosynthesis and yield prediction and, in a 15-run human-AI campaign, found a 67%-yield condition for a…","keywords":["organic synthesis","large language models","retrosynthesis","yield prediction","active learning","Bayesian optimization","Suzuki-Miyaura coupling","SMILES"],"falsifier":"Run the same unreported coupling from the same round-0 starting conditions, but replace Chemma's ligand suggestions with random ligands and follow the same human solvent-switch rule; if this control reaches 67% isolated yield in 15 or fewer runs, the attribution of the acceleration to Chemma is falsified. A complementary check: pre-register Chemma's predicted yields for the round-1 candidates and compare them with observed yields; if predictions are not positively correlated with the 67% result, the model is not driving the success.","tokens_in":28878,"feed_emoji":"🧪","tokens_out":9889,"duration_ms":85498,"temperature":0.7,"pith_summary":"Chemma is a large language model built by fully fine-tuning a 7-billion-parameter decoder-only model on 1.28 million question-answer pairs about organic reactions, expressed as SMILES and natural language. The paper sets out to show that such a chemistry-tuned model can act as a synthesis assistant: it reports 72.2% top-1 accuracy on single-step retrosynthesis, yield and selectivity predictions with R2 values of 0.74–0.93 on several benchmark and high-throughput datasets, and faster reaction optimization when combined with Bayesian optimization and an active-learning loop. The decisive demonstration is a previously unreported Suzuki-Miyaura coupling of cyclic aminoboronates with aryl halides, where the model and chemists together found a working ligand and solvent in 15 wet-lab runs, reaching 67% isolated yield. If the claims hold, reaction knowledge can be extracted from text and SMILES sequences rather than from quantum-chemical calculations, lowering the barrier to AI-guided synthesis.","feed_headline":"Fine-tuned LLM finds 67% yield for unreported reaction in 15 runs","feed_subtitle":"Chemistry-tuned model also tops retrosynthesis and yield benchmarks without quantum-chemical descriptors.","key_machinery":"The load-bearing mechanism is Chemma, with three linked components. Chemma-SFT is a decoder-only language model fully fine-tuned on 1.28 million reaction Q&A pairs for forward prediction, retrosynthesis, and condition generation. Chemma-RM is trained with pairwise ranking data and reinforcement learning from experimental feedback, so it prefers conditions associated with higher yields or selectivities. For regression tasks, the authors take reaction embeddings from the last hidden layer of Chemma-SFT and pass them through a small multilayer perceptron that predicts yield or selectivity; the same embedding space feeds the active-learning loop and a modified Bayesian-optimization scheme in which a Gaussian process corrects the bias between Chemma-predicted and observed yields. What carries the argument is that all tasks share one learned representation of reactions, so knowledge acquired from large reaction corpora can be reused for wet-lab suggestions.","core_discovery":"The central claim is that one fully fine-tuned language model, trained only on reaction SMILES paired with instruction-style questions, can learn enough chemistry to outperform dedicated methods on standard predictive tasks and to propose working conditions for reactions outside its training set. On a standard 50,000-reaction retrosynthesis benchmark, Chemma reaches 72.2% top-1 accuracy, surpassing the best transformer baseline by 17.1 points. On high-throughput and literature-derived yield datasets, the model reaches R2 values between 0.74 and 0.86 without DFT descriptors, and on regioselectivity and enantioselectivity tasks it reports R2 values of 0.93 and 0.89. The paper's headline wet-lab result is an unreported Suzuki-Miyaura coupling of a cyclic aminoboronate with an aryl halide: in an active-learning campaign of 15 runs, PAd3 as ligand and 1,4-dioxane as solvent gave 67% isolated yield. The authors argue this shows open-space reaction exploration is possible without predefined condition pools and without quantum-chemical calculations.","pith_inferences":["I read the wet-lab result as evidence for a more modest mechanism than the paper's headline: Chemma contributes candidate diversity, but the chemists supplied the initial ligand pool and the solvent switch from p-xylene to 1,4-dioxane, so the 15-run count alone does not isolate the model's causal contribution.","Round 0 of the open-space campaign produced yields below 15% for all nine tested ligands, suggesting the model's zero-shot extrapolation to a genuinely new reaction is weak and that the fine-tuning-in-the-loop step is the likely source of improvement.","A natural testable extension would be to run the same active-learning protocol with the model's suggestions replaced by random or expert-only suggestions while keeping the human intervention fixed; the difference in runs-to-target would quantify the model's added value.","If the shared reaction embeddings are the reason for the transfer, the same two-stage training recipe could be applied to other reaction-performance endpoints, but each new reaction family would still require its own fine-tuning and validation."],"forward_implications":["If the 72.2% top-1 retrosynthesis result is reproducible, template-free LLM retrosynthesis is competitive with dedicated graph and transformer methods on the standard 50k benchmark.","Yield and selectivity prediction without DFT descriptors would make data-driven optimization applicable to electronic lab notebook and literature data at lower computational cost.","Adding Chemma-generated yields to Bayesian optimization is reported to reach high yields in fewer experimental batches, suggesting synthetic data can substitute for some real observations.","If the 15-run open-space result is attributed to the model, active learning with a fine-tuned LLM can discover conditions for reaction classes absent from the training corpus, a step toward autonomous synthesis."],"supporting_citations":[{"why":"provides the large reaction corpus from which the model's training Q&A pairs were built.","marker":"[43]"},{"why":"supplies the 50k-reaction benchmark used for retrosynthesis training and evaluation.","marker":"[44]"},{"why":"documents the extraction of the reaction data underlying the retrosynthesis benchmark.","marker":"[45]"},{"why":"supplies the DFT-descriptor random-forest baseline and the Buchwald-Hartwig high-throughput yield data.","marker":"[55]"},{"why":"provides the Bayesian-optimization framework that the paper modifies into Chemma-BO.","marker":"[38]"},{"why":"provides the electronic-lab-notebook yield dataset used for out-of-sample prediction.","marker":"[58]"},{"why":"supplies the Suzuki-Miyaura high-throughput dataset used for yield prediction and optimization tests.","marker":"[62]"}],"fun_headline_variants":["Fine-tuned LLM finds 67% yield for novel reaction in 15 runs","Chemma LLM: 67% yield in 15 runs, no quantum chemistry","AI chemist LLM discovers new coupling in 15 tries, 67%","Language model proposes working conditions: 67% yield in 15","LLM beats benchmarks, finds 67% yield without DFT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the wet-lab campaign's success—67% isolated yield in 15 runs—can be attributed to Chemma's suggestions rather than to the chemists' experience, since the solvent change and the initial ligand choices came from the human experts.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLM finds 67% yield for novel reaction in 15 runs","Chemma LLM: 67% yield in 15 runs, no quantum chemistry","AI chemist LLM discovers new coupling in 15 tries, 67%","Language model proposes working conditions: 67% yield in 15","LLM beats benchmarks, finds 67% yield without DFT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000518,"raw_usage":{"total_tokens":2562,"prompt_tokens":1049,"completion_tokens":1513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":1413}},"tokens_in":665,"tokens_out":1513,"duration_ms":11079,"temperature":1.0,"reasoning_tokens":1413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:18:41.781186+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same unreported coupling from the same round-0 starting conditions, but replace Chemma's ligand suggestions with random ligands and follow the same human solvent-switch rule; if this control reaches 67% isolated yield in 15 or fewer runs, the attribution of the acceleration to Chemma is falsified. A complementary check: pre-register Chemma's predicted yields for the round-1 candidates and compare them with observed yields; if predictions are not positively correlated with the 67% result, the model is not driving the success.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the large reaction corpus from which the model's training Q&A pairs were built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the 50k-reaction benchmark used for retrosynthesis training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"documents the extraction of the reaction data underlying the retrosynthesis benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the DFT-descriptor random-forest baseline and the Buchwald-Hartwig high-throughput yield data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the Bayesian-optimization framework that the paper modifies into Chemma-BO."},{"cited_title":"Saebi, et al., On the use of real-world datasets for reaction yield prediction, Chemical Science 14, 4997–5005 (2023)","cited_arxiv_id":null,"evidence_quote":"provides the electronic-lab-notebook yield dataset used for out-of-sample prediction."},{"cited_title":"Perera, et al","cited_arxiv_id":null,"evidence_quote":"supplies the Suzuki-Miyaura high-throughput dataset used for yield prediction and optimization tests."}],"review_version":1}