{"id":"cab83459-d6b5-492a-8f92-ca33f31c716c","arxiv_id":"2505.05616","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuned Llama-3.1 models can perform enzymatic reaction prediction tasks, and multitask learning improves forward and retrosynthesis over single-task training.","lead":"This paper fine-tunes Llama-3.1 language models on enzyme reaction data and reports that they can predict enzyme classes, reaction products, and substrates, with multitask training improving product and substrate prediction. It is a benchmark that shows where general-purpose LLMs stand against specialized chemistry models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multitask gains may be a data-volume artifact: MT trains on ~3x more examples than each ST model, so the +7.9/+5.3 improvements do not isolate shared biochemical knowledge from plain additional data.","rationale":"I read the paper as an empirical benchmark with a specific mechanistic claim: multitask fine-tuning improves forward and retrosynthesis by sharing biochemical knowledge. The 91.7% EC1 accuracy is a solid, reproducible result under this split, and the leakage-aware grouping in Section 2.1 is thoughtful. The strongest soft spot is the ST-versus-MT comparison in Table 3, because the two conditions differ in training data volume by a factor of three. This is not a disagreement with the field; it is a missing internal control. The reader's contamination concern is real but external and acknowledged; it affects all accuracy numbers and cannot be resolved from the paper alone. The data-volume confound is directly visible in the paper's own tables and can be settled by a re-training experiment. If it lands, the abstract's claim that 'multitask learning enhances... by leveraging shared enzymatic information' overstates what the data show. The paper should be accepted only conditionally, with the matched-volume control or a softened claim as the condition. I therefore keep the reader's CONDITIONAL verdict but for a different, more proximal reason.","tokens_in":15783,"tokens_out":6787,"duration_ms":74435,"concrete_test":"Retrain the single-task FS and RS models on a matched-volume dataset: concatenate the other two tasks' task-formatted training examples into the ST training set, or equivalently subsample the MT training set so each task contributes ~600 examples while keeping total size ~1800, and evaluate on the same test split. If the matched-volume ST model matches or exceeds the MT match rates (33.8% FS, 19.2% RS in Table 3), the claimed multitask synergy is not supported; if it remains clearly below, the synergy interpretation survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.2 describes the MT setup as 'the three ST datasets are merged together,' and Table 4 reports the default ST training size as ~1800 samples per task. The MT model therefore trains on roughly 5400 examples, three times the data of any single-task model. The headline multitask gains in Table 3 (+7.9 points forward-synthesis match, +5.3 points retrosynthesis match) are attributed in the abstract and conclusions to 'leveraging shared enzymatic information,' but the design never controls for training-set size. A single-task model exposed to the same number of formatted examples—e.g., by concatenating the other tasks' training sets—could plausibly show comparable gains simply from seeing more reaction SMILES and enzyme–molecule associations. This confound is structural, not speculative: the ST and MT conditions differ in data volume by construction. It specifically undermines the paper's most distinctive claim (multitask synergy), whereas the pretraining-contamination concern acknowledged in Section 3.5 applies equally to all numbers and is already disclosed. The leakage-aware splitting and the reporting of standard deviations are genuine strengths, but the MT comparison needs a matched-volume control before the synergy interpretation can be accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper fine-tunes Llama-3.1 8B and 70B models with LoRA adapters on a BRENDA subset of the ECREACT dataset for three enzymatic tasks: EC number prediction, forward synthesis, and retrosynthesis. The authors compare single-task (ST) and multitask (MT) fine-tuning, evaluate low-data regimes and different LoRA configurations, and report that the 70B model reaches 91.7% EC1 accuracy and that multitask learning improves forward-synthesis match accuracy by 7.9 percentage points and retrosynthesis by 5.3 points over single-task models (Table 3). The paper concludes that fine-tuned LLMs capture biochemical knowledge and that multitask learning leverages shared enzymatic information. The authors also acknowledge limitations including possible pretraining contamination, dataset size constraints, and computational costs.","tokens_in":16015,"tokens_out":3649,"duration_ms":42484,"significance":"If the claims hold, the paper provides a useful, reproducible demonstration that open-weight LLMs adapted with PEFT can perform enzyme-related prediction tasks, with strengths including leakage-aware data splitting (grouping reactions by shared product/EC or substrate/EC pairs), evaluation over three seeds, and release of code and data. The honest limitations section is a positive feature. However, the most distinctive claim—that multitask learning yields synergy through shared biochemical knowledge—is currently confounded by training-set size, and the acknowledged pretraining contamination risk limits the interpretation of the absolute accuracy numbers. The paper is a solid empirical study, but the central multitask claim needs a matched-volume control before it can be accepted as stated.","major_comments":[{"comment":"The multitask gains attributed to 'shared enzymatic information' are confounded by training-set size: the MT model is trained on the merged three ST datasets (~5400 examples), whereas each ST model is trained on ~1800 examples (Table 4). The +7.9 and +5.3 percentage-point improvements in Table 3 are therefore equally consistent with a pure data-volume effect. Please add a matched-volume control—for example, an ST model trained on a comparable number of examples drawn from the same union of tasks, or an MT model trained on a downsampled union—or explicitly rephrase the abstract and conclusions to stop claiming that the gains arise from leveraging shared enzymatic information.","section":"3.2, Tables 3 and 4"},{"comment":"The acknowledged possibility that Llama-3.1's pretraining corpus includes BRENDA/ECREACT reactions means the absolute accuracies (e.g., 91.7% EC1 in Table 1 and 33.8% FS Match in Table 3) are not interpretable as biochemical generalization from fine-tuning alone. Because the paper's central claim is that fine-tuned LLMs 'capture biochemical knowledge,' this is a load-bearing issue, not merely a caveat. Please provide a contamination analysis (e.g., probing whether test reactions are memorized, or evaluating on temporally held-out or newly curated reactions) or substantially soften the generalization claim.","section":"3.5, Limitations"},{"comment":"The XGBoost comparisons for forward and retrosynthesis are not apples-to-apples: XGBoost is evaluated on exact 256-bit fingerprint matching while the LLM is evaluated on canonical SMILES matching. This makes the Section 3.3 claim that 'our models outperform the XGBoost model' in the low-data regime difficult to interpret. Since the low-data section uses this comparison to argue that fine-tuning adds value, the evaluation protocols should be aligned (e.g., evaluate both methods on the same exact-match definition) or the claim should be qualified accordingly.","section":"3.3 and Appendix 5.6"}],"minor_comments":[{"comment":"There are several typos, including 'the the' in the first paragraph, 'uncorrect' instead of 'incorrect,' and 'settup' instead of 'setup'; please proofread.","section":"Section 3.1"},{"comment":"Table 3 does not report standard deviations or confidence intervals for the ST and MT values, despite Section 2.3 stating that all results are averaged over N=3 experiments with standard deviations reported where applicable; please add them or clarify why they are omitted.","section":"Table 3"},{"comment":"The caption says the model is given 'reactants and substrates,' but the EC prediction task is defined in Section 2.1 as predicting EC numbers from 'substrates and products'; the caption should say 'reactants and products.'","section":"Figure 6 caption"},{"comment":"References [12] and [13] both list the same title 'Improving language understanding by generative pre-training'; the second entry appears to be mis-titled and should be corrected.","section":"References [12] and [13]"},{"comment":"The phrase 'we explore two new LoRA setups' is confusing because these are not novel architectural proposals; consider rewording to 'two additional LoRA configurations.'","section":"Section 2.4"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the data-volume confound in the multitask comparison. If the authors can add a matched-volume control and the multitask gains persist, the paper would be a solid contribution; if the gains disappear, the synergy claim must be withdrawn. The pretraining contamination issue is acknowledged but should not be dismissed as a mere caveat, since it affects the core generalization claim. The paper otherwise contains sound internal comparisons and useful reproducible resources."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a carefully done empirical benchmark: LoRA-fine-tuned Llama-3.1 8B and 70B on three enzyme tasks (EC number, forward synthesis, retrosynthesis) using the BRENDA subset of ECREACT, with leakage-aware grouping, standard deviations over three runs, an XGBoost baseline, and code/data released. What's genuinely useful is the head-to-head comparison of model size, LoRA configuration, and data regime, plus the honest disclosure that pretraining contamination is possible and that SOTA comparisons are not apples-to-apples. The 91.7% EC1 accuracy for the 70B model is a reasonable number, but it is not the key contribution.\n\nThe stress-test concern lands. The multitask setup merges the three ST datasets, so the MT model trains on roughly three times as many examples (~5400 vs ~1800). The +7.9 and +5.3 point gains in forward/retrosynthesis match are attributed to 'shared enzymatic information,' but the design never isolates data volume from multitask synergy. A single-task model trained on the same number of formatted examples could plausibly show comparable gains. This is a structural confound, not a nitpick, and it directly undercuts the paper's most distinctive claim. It is fixable: add a matched-volume ST baseline or an ablation that concatenates other tasks' data without task labels.\n\nThe pretraining contamination issue is real but disclosed, and the authors are appropriately cautious in the limitations section. It means the absolute accuracies should be read as upper bounds, not as proof of biochemical generalization. The small BRENDA subset and the non-comparable SOTA numbers are also acknowledged, so I don't count those against the authors twice.\n\nThe paper is not a breakthrough, but it is a solid, reproducible empirical study that answers a useful question: can PEFT-adapted open LLMs handle enzyme tasks, and what design choices matter? It deserves a serious referee. The path to acceptance is clear: run a matched-volume control for MT, and ideally evaluate on reactions published after Llama-3.1's pretraining cutoff. I would cite this for its careful preprocessing and low-data results, and I'd bring it to a reading group focused on LLM fine-tuning for chemistry.","headline":"A clean, honestly reported empirical benchmark of LoRA-fine-tuned Llama-3.1 on enzyme tasks, held back from acceptance by an uncontrolled multitask data-volume confound.","tokens_in":16532,"tokens_out":1434,"would_cite":true,"duration_ms":18226,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned Llama-3.1 models, trained with LoRA adapters, can predict enzyme classes, forward synthesis products, and retrosynthetic substrates from SMILES strings, with the 70B model reaching 91.7% accuracy on the first EC digit.","keywords":["enzyme commission number prediction","forward synthesis","retrosynthesis","large language models","parameter-efficient fine-tuning","low-rank adaptation (LoRA)","multitask learning","EC number classification"],"falsifier":"Run the same fine-tuning pipeline on a test set of enzyme reactions published after Llama-3.1's training data cutoff, or otherwise verifiably absent from public web corpora. If EC1 accuracy on that set collapses toward the 29.6% zero-shot baseline while accuracy on the original test set stays near 91.7%, the central claim of learned biochemical knowledge is refuted.","tokens_in":15589,"feed_emoji":"🧬","tokens_out":6504,"duration_ms":67800,"temperature":0.7,"pith_summary":"The paper sets out to show that a general-purpose open-weights language model, fine-tuned cheaply with LoRA adapters, can take on three enzyme-related prediction tasks: assigning Enzyme Commission numbers to reactions, predicting reaction products, and predicting substrates. Using the BRENDA subset of the ECREACT database, the authors report that a fine-tuned Llama-3.1 70B model assigns the correct first EC digit in 91.7% of test reactions, and that multitask training on all three tasks raises match accuracy by 7.9 percentage points in forward synthesis and 5.3 percentage points in retrosynthesis relative to single-task fine-tuning. If these results hold, they make a case that LLMs are a viable starting point for enzyme reaction prediction, especially for low-data settings or when one model must serve several biochemical tasks. The paper also reports that exact matches still trail specialized models, and it flags that the public reaction data may have appeared in the model's pretraining.","feed_headline":"Fine-tuned Llama-3.1 hits 91.7% on enzyme class prediction","feed_subtitle":"Multitask training adds 7.9 points on forward synthesis and 5.3 on retrosynthesis for a 70B model.","key_machinery":"The machinery has two parts. First, parameter-efficient fine-tuning via Low-Rank Adaptation (LoRA): the pretrained Llama-3.1 weights stay frozen and only small low-rank update matrices are learned, so the model gains task-specific behavior while the base model's general capabilities are preserved. Second, a multitask setup: the three task datasets are merged and one model is trained to emit EC numbers, products, and substrates, so each prediction can draw on shared enzymatic knowledge. All inputs and outputs are represented as text, with SMILES strings for molecules and dot-separated EC digits for enzymes, turning each task into a next-token generation problem. The data preparation, which groups reactions sharing the same product-EC or substrate-EC pair and rotates groups across tasks and splits, is what makes the single-task and multitask comparisons fair.","core_discovery":"The paper's central claim is that fine-tuned Llama-3.1 models capture enough biochemical knowledge to predict enzyme classes and, to a lesser degree, enzymatic reaction products and substrates. On the BRENDA subset, the 70B model reaches 91.7% macro-averaged accuracy for the first EC digit and 49.2% for three-digit EC numbers, and it generates valid SMILES in nearly all cases. Multitask fine-tuning on all three tasks simultaneously improves forward-synthesis match accuracy from 25.9% to 33.8% and retrosynthesis from 13.9% to 19.2%, which the authors attribute to shared enzymatic information. They also report that fine-tuning with as few as 200 training samples nearly doubles zero-shot EC1 accuracy, from 29.6% to 55.3%. In their reading, the residual gap to specialized baselines is a data-scale and coverage issue rather than a sign that the approach is unsound.","pith_inferences":["If the pretraining-leakage concern is resolved and the multitask gains persist on truly unseen reactions, fine-tuned open LLMs could serve as cheap pre-screening tools for enzyme-substrate pairs, feeding candidate reactions to more expensive specialized predictors.","The pattern that retrosynthesis benefits less than forward synthesis from multitask learning suggests that substrate prediction may need more diverse chemistry or explicit reaction templates; a testable extension is to add enzyme sequence or mechanism information to the prompt.","A direct next experiment is a temporal split: train only on reactions recorded before a cutoff date and test on later ones. If accuracy holds, the biochemical generalization is real; if it collapses, the reported numbers would need recalibration."],"forward_implications":["Fine-tuned 70B Llama-3.1 predicts the correct enzyme main class in 91.7% of test reactions, compared with 29.6% for zero-shot prompting of the same model.","Multitask training is the strongest lever for synthesis tasks: it adds 7.9 percentage points for forward synthesis and 5.3 for retrosynthesis over single-task fine-tuning.","With only 200 training examples, fine-tuning still reaches 55.3% EC1 accuracy, suggesting practical value in low-data biocatalysis settings.","Fine-tuning on one enzyme task improves the model's zero-shot performance on the other two; for example, EC-only fine-tuning lifts forward-synthesis match accuracy to 12.9%.","The model rarely produces invalid or unparseable SMILES under 5% for both synthesis tasks, so its errors are mostly chemically meaningful alternatives rather than grammar failures."],"supporting_citations":[{"why":"Supplies the ECREACT dataset, specifically the BRENDA subset, and the forward and retrosynthesis state-of-the-art numbers the paper compares against.","marker":"[9]"},{"why":"Provides the state-of-the-art EC-number prediction accuracies used as the benchmark for EC1 through EC3.","marker":"[10]"},{"why":"Supports the choice of fine-tuning over in-context learning by showing that fine-tuning outperforms in-context learning at comparable model sizes.","marker":"[19]"},{"why":"Demonstrates that fine-tuned LLMs handle chemistry tasks well in low-data regimes, motivating the paper's data-regime experiments.","marker":"[25]"},{"why":"Introduces the multitask instruction-tuning approach that the paper adapts to enzyme reaction tasks.","marker":"[30]"},{"why":"Defines LoRA, the parameter-efficient fine-tuning method used for all model adaptations in the study.","marker":"[36]"},{"why":"Provides QLoRA, the quantized fine-tuning approach that lets the authors train 4-bit models within their computational budget.","marker":"[37]"},{"why":"Defines the Llama-3.1 8B and 70B base models that are fine-tuned throughout the study.","marker":"[38]"}],"fun_headline_variants":["Llama-3.1 fine-tuning predicts enzyme classes at 91.7% accuracy","Multitask LLMs boost enzyme reaction predictions","70B Llama-3.1 improves retrosynthesis with multitask learning","Low-data enzyme prediction: LLMs fine-tune from 200 samples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the test reactions are genuinely new to the model; because the public BRENDA and ECREACT data may have appeared in Llama-3.1's pretraining, the reported accuracies could reflect memorization or format familiarity rather than learned biochemistry.","fun_headline_variants_meta":{"raw":{"variants":["Llama-3.1 fine-tuning predicts enzyme classes at 91.7% accuracy","Multitask LLMs boost enzyme reaction predictions","70B Llama-3.1 improves retrosynthesis with multitask learning","Low-data enzyme prediction: LLMs fine-tune from 200 samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1449,"prompt_tokens":945,"completion_tokens":504,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":561,"tokens_out":504,"duration_ms":5429,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:00:34.465970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuning pipeline on a test set of enzyme reactions published after Llama-3.1's training data cutoff, or otherwise verifiably absent from public web corpora. If EC1 accuracy on that set collapses toward the 29.6% zero-shot baseline while accuracy on the original test set stays near 91.7%, the central claim of learned biochemical knowledge is refuted.","supporting_citations":[{"cited_title":"Biocatalysed synthesis planning using data-driven learning","cited_arxiv_id":null,"evidence_quote":"Supplies the ECREACT dataset, specifically the BRENDA subset, and the forward and retrosynthesis state-of-the-art numbers the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the state-of-the-art EC-number prediction accuracies used as the benchmark for EC1 through EC3."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates that fine-tuned LLMs handle chemistry tasks well in low-data regimes, motivating the paper's data-regime experiments."}],"review_version":1}