{"id":"85c93ad8-74ef-407f-affe-e28cc1a30758","arxiv_id":"2506.23056","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"K-MSE adds a substructure knowledge base, a learned molecule-spectrum scorer, and Monte Carlo tree search, lifting LLM exact-match accuracy on MolPuzzle from 3.7% to 27.3% (GPT-4o-mini) and from 27.8% to 57.8% (GPT-4o).","lead":"The authors built a plug-in system that helps large language models deduce molecular structures from spectral data, reporting large accuracy gains on a 216-molecule chemistry benchmark. It matters because automating structure elucidation could speed up chemical lab analysis, though the evaluation is small and several settings were tuned on the test set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MolPuzzle gains in Table 1 are computed after choosing k and Niter on the same 216-molecule test set, so the central >20% improvement claim is not yet established out-of-sample.","rationale":"I read the paper in good faith. The framework is coherent, the code is promised, and the reported gain magnitudes are large enough to be worth investigating. The reader's formal weakest assumption was scorer generalization, and that is a real mechanism-level risk. However, the manuscript partially addresses it with Figure 4, which shows a 0.53 correlation between the specialized scorer's reward and oracle fingerprint similarity on GPT-4o-mini, and with Figure 5, which shows improved retrieval hit rates. The test-set hyperparameter selection issue is not addressed anywhere: Section 5.3 and Appendix A.3 together reveal that k and Niter were chosen by inspecting performance on the same 216-molecule benchmark used for Table 1. With n=216, this is a direct threat to the central empirical claim, independent of whether the scorer generalizes. My proposed held-out validation check would settle whether the gains are genuine. If the gains persist, I would accept the central claim; if they collapse, the verdict should move toward rejection. Since the reader already assigned CONDITIONAL, my read does not change that verdict, but it sharpens the reason: the condition should include a properly split hyperparameter selection procedure.","tokens_in":17442,"tokens_out":8101,"duration_ms":91747,"concrete_test":"Split MolPuzzle's 216 molecules into a development set and a held-out test set (e.g., 108/108, stratified by formula complexity). For each base model, select k in {0,1,2,3} and Niter in {0,2,4,8,16} on the development set only, then freeze the chosen hyperparameters and run the full comparison on the held-out test set. Repeat over 5 random splits and report the mean and 95% CI for ACC. If K-MSE's ACC gain over CoT remains above 0.20 for GPT-4o-mini and GPT-4o on held-out molecules, the central claim stands; if the gain collapses or falls within noise, the Table 1 results are dominated by test-set hyperparameter selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim depends on the magnitude of the ACC improvements in Table 1, but those numbers are produced with hyperparameters selected on the evaluation set itself. Appendix A.3 reports k=1 for Llama-3.2-11B and GPT-4o-mini, k=2 for GPT-4o and GPT-o1, and Niter=8 for most models; Section 5.3 and Figure 6 justify these choices as the 'best performance' on MolPuzzle. MolPuzzle contains only 216 molecules, so selecting even a few hyperparameters on this set can fit noise rather than real signal. No held-out validation split, no cross-validation, and no confidence intervals are reported. The ACC jumps from 0.037 to 0.273 (GPT-4o-mini) and from 0.278 to 0.578 (GPT-4o) may therefore reflect test-set selection bias rather than a robust plugin gain. This is load-bearing because the headline assertion is precisely these numbers; if the gains shrink or vanish under honest out-of-sample evaluation, the central claim fails even if the underlying mechanism is sound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces K-MSE, a plug-in reasoning framework for molecular structure elucidation. It combines a molecular substructure knowledge base, a learned molecule-spectrum scorer used as both retriever and reward model, and Monte Carlo Tree Search with Self-Refine. Experiments on the MolPuzzle benchmark with four base LLMs report large gains, notably ACC increases of 0.236 for GPT-4o-mini and 0.300 for GPT-4o. The paper also provides ablations of the knowledge base, the critique inputs, and scorer components, plus an analysis of token consumption.","tokens_in":17678,"tokens_out":3843,"duration_ms":44327,"significance":"If the reported gains are robust, the work is a useful contribution to test-time scaling for scientific reasoning: it demonstrates a concrete way to inject domain knowledge and a trained reward model into LLM inference, and it releases code and detailed implementation details that support reproducibility. The paper is also careful to include a Limitations section and extensive appendices with prompts and hyperparameters. However, the central quantitative claim currently rests on an evaluation protocol with no held-out hyperparameter selection and no statistical significance testing, which limits confidence in the headline improvements.","major_comments":[{"comment":"The hyperparameters Niter and k appear to be selected on the MolPuzzle test set itself. Section 5.3 states that the best performance on GPT-4o-mini occurs at k=1 and on GPT-4o at k=2, and Appendix A.3 then fixes these values for the main results in Table 1. Because MolPuzzle contains only 216 molecules, tuning even two hyperparameters on the evaluation set can fit noise rather than real signal, and the headline ACC gains (e.g., 0.037 to 0.273 for GPT-4o-mini) may be inflated by selection bias. The paper should either use a held-out validation split or cross-validation for hyperparameter selection, or report the full sensitivity grid over Niter and k for all models and show that the conclusions are stable across reasonable choices.","section":"§5.3, Appendix A.3, Figure 6"},{"comment":"The abstract and Section 5.2 use the word \"significantly\" to describe the performance gains, but the evaluation reports no error bars, confidence intervals, or significance tests. With only 216 benchmark molecules, a single-run point estimate does not support the claim of statistical significance. The authors should report bootstrap confidence intervals for the metrics, a paired significance test (e.g., McNemar for ACC), or at least variance across multiple runs or random seeds. This is load-bearing because the central assertion is precisely that the gains are substantial and reliable.","section":"Table 1, §5.2"},{"comment":"The molecule-spectrum scorer is trained on 9,000 simulated ZINC spectra and then used for both knowledge-base retrieval (Eq. 8) and MCTS reward (Eq. 12). The manuscript's evidence that this scorer transfers to real MolPuzzle spectra is limited to a validation-loss curve and the correlation shown in Figure 4 for GPT-4o-mini. Since the same scorer drives both retrieval and answer selection, a systematic bias in the scorer could be reinforced through the search. The authors should provide a more direct evaluation of scorer quality on MolPuzzle, such as the correlation between scorer reward and structural correctness for the candidate molecules actually encountered during MCTS, or an ablation replacing the scorer reward with an oracle similarity measure. This would strengthen the causal interpretation of the gains.","section":"§4.2, Appendix A.2, Eq. (12)"}],"minor_comments":[{"comment":"The loss in Eq. (7) is labeled NT-Xent, but the denominator as written sums only over negative cross-modal pairs and omits the positive pair; the standard NT-Xent form includes the positive in the denominator. Please clarify whether this is a typo or an intentional variant, and define the exact loss used in training.","section":"Eq. (7)"},{"comment":"The phrase \"more than 20% improvement\" is ambiguous because it could mean absolute percentage points or relative improvement. The reported numbers (e.g., ACC from 0.278 to 0.578) are absolute increases; please state this explicitly.","section":"Abstract, §5.2"},{"comment":"For GPT-o1, the table reports only CoT and K-MSE, omitting the Self-R, Self-C, MAD, and MCTSr baselines that are shown for the other models. This makes the comparison incomplete for the strongest base model.","section":"Table 1"},{"comment":"The left panels of Figures 4 and 5 display numeric values without labeled axes or a clear description of which metric is shown. Adding axis labels and a legend would make the figures interpretable without referring to the caption multiple times.","section":"Figures 4 and 5"},{"comment":"The problem formulation includes IR spectra as input, but the scorer uses only C-NMR and H-NMR. The manuscript should clarify that IR is handled by the LLM prompt only and is not part of the scorer's reward or retrieval signal.","section":"§4.2, §3"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the evaluation protocol: hyperparameters appear to be tuned on the test set, and there are no significance tests. These issues are fixable with additional experiments or a more cautious framing, but they currently prevent me from endorsing the headline 'significant boost' claim. If the authors can provide out-of-sample validation or a sensitivity analysis, I would be supportive of publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a serious, well-built attempt to make LLMs solve structure elucidation with test-time search, and the reported gains are large enough that the approach deserves a careful look. The novel part is the integration: a 593-substructure knowledge base, a contrastively trained molecule-spectrum scorer that doubles as retriever and reward model, and MCTS with Self-Refine. The ablation work is honestly done: removing the KB or the critique inputs hurts, and the specialized scorer clearly beats an LLM-as-scorer (0.53 vs 0.03 correlation with oracle similarity). The authors also ship code; that counts.\n\nThe main soft spot is real: Niter and k are chosen on the same 216-molecule MolPuzzle test set, and Figure 6 explicitly calls them \"best performance.\" With no held-out split and no error bars, I would not take the precise >20% ACC gains at face value. But this is a common flaw in this literature, and it is not fatal: the gains are huge, several ablations use fixed settings, and the mechanism is plausible. The scorer is trained on simulated ZINC spectra, so generalization to experimental spectra is unverified; the paper admits the benchmark limitation. That is a fair caveat, not a hidden defect.\n\nThe bigger question is whether the evaluation methodology can be fixed in a revision, and it can: a validation split, significance tests, or cross-validation would settle it. The core idea is good enough that I would not desk-reject. A referee should focus on the evaluation and on probing how sensitive the gains are to k and Niter. The citation pattern is fine, with proper credit to MCTSr and MolPuzzle.\n\nWho is this for? Anyone building LLM copilots for spectral interpretation, and people working on test-time scaling for scientific reasoning. It deserves a serious referee: I would send it out, with a request to tighten the evaluation. My own verdict is between \"promising but needs validation\" and \"conditional accept.\"","headline":"A well-built integration of knowledge base, learned scorer, and MCTS for structure elucidation that reports large gains, but the headline numbers rest on test-set hyperparameter selection, so they need out-of-sample verification before being taken at face value.","tokens_in":18236,"tokens_out":1590,"would_cite":true,"duration_ms":17300,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that augmenting an LLM with a molecular substructure knowledge base, a contrastively trained molecule–spectrum scorer, and Monte Carlo Tree Search raises exact-match structural accuracy on MolPuzzle from 3.7% to 27.3%…","keywords":["molecular structure elucidation","LLM reasoning","Monte Carlo Tree Search","test-time scaling","molecule-spectrum scorer","chemical substructure knowledge base","contrastive learning","MolPuzzle"],"falsifier":"Measure the rank correlation between the K-MSE reward score and the ground-truth fingerprint similarity on all candidate molecules the search generates across the full MolPuzzle set; if the correlation drops to near zero on novel structures, as the authors find for the LLM-based scorer (0.03), the gains should vanish. A direct ablation that replaces R(a') with random or inverted scores, while keeping the search structure, knowledge base, and rewrite steps intact, would show whether the scorer's signal actually causes the 0.236/0.300 ACC jumps.","tokens_in":17254,"feed_emoji":"🧪","tokens_out":8323,"duration_ms":74605,"temperature":0.7,"pith_summary":"Molecular structure elucidation—deducing a molecule from its NMR, IR, and formula data—remains hard for LLMs even as they excel at many analytic tasks. The paper identifies the cause as missing chemical knowledge in two places: LLMs do not cover the sheer variety of substructures, and they cannot judge whether a predicted molecule actually matches the observed spectra. K-MSE addresses both with an external knowledge base of 593 substructures and a contrastively trained molecule–spectrum scorer that supplies a reward signal for the search. Plugging this into a Monte Carlo Tree Search loop, exact-match accuracy rises from 3.7% to 27.3% on GPT-4o-mini and from 27.8% to 57.8% on GPT-4o, with gains across all four tested base models. If the claim holds, it means a specialized verifier plus retrieval can make LLMs usable for automated spectral interpretation without retraining the language model.","feed_headline":"Chemical knowledge and tree search double LLM structure ID accuracy","feed_subtitle":"Adding substructure knowledge and a spectral reward model lifts exact-match accuracy from 27.8% to 57.8% on MolPuzzle","key_machinery":"The load-bearing mechanism is the molecule–spectrum scorer, a dual-encoder model whose cosine similarity $\\mathrm{sim}(g_m(m), g_s(n))$ serves double duty: as the reward $R(a')$ that evaluates each new tree node (Eq. 12), and as the retrieval score that selects top-$k$ substructures from the knowledge base given the query spectrum (Eq. 8). The spectrum encoder $g_s$ discretizes the chemical shift, splitting pattern, and coupling constant of each carbon and hydrogen signal into token embeddings and runs them through a two-layer Transformer (Eqs. 5–6); the molecule encoder $g_m$ embeds the molecular graph with a 5-layer GIN and the Morgan+MACCS+RDK fingerprint with an MLP, then fuses them (Eq. 4). Training on 9,000 ZINC molecules with simulated C-NMR and H-NMR spectra uses the NT-Xent loss (Eq. 7). This scorer is the component that converts a generic self-refine loop into guided search, and the paper's ablations show that removing any of its inputs (graph, fingerprint, splitting pattern, or coupling constant) raises validation loss.","core_discovery":"On its own terms, the paper's discovery is that the bottleneck for LLMs in molecular structure elucidation is not reasoning effort but two specific knowledge deficits, and that both can be patched at inference time. First, an automatically constructed molecular substructure knowledge base, extracted from a 4M-molecule database with LLM-generated descriptions, widens the model's coverage of chemical space. Second, a specialized molecule–spectrum scorer—a molecule encoder (GIN over the molecular graph plus MLP over concatenated fingerprints) and a spectrum encoder (Transformer over tokenized chemical shifts, splitting patterns, and coupling constants)—is trained with the NT-Xent contrastive loss on 9,000 simulated NMR spectra from ZINC. The cosine similarity between molecule and spectrum embeddings becomes the reward in a Monte Carlo Tree Search whose expansion step critiques the current candidate using its rendered image and formula, then rewrites it. The paper reports that this combined system outperforms CoT, Self-Refine, Self-Consistency, multi-agent debate, and a prior MCTS self-refine method on the MolPuzzle benchmark, with the scorer's reward correlating at 0.53 with ground-truth fingerprint similarity, versus 0.03 for an LLM-based scorer. The knowledge base alone also improves every baseline method it is added to, indicating the components are independently useful.","pith_inferences":["A natural extension is to treat the scorer's differentiable spectrum–molecule similarity as a training signal for the LLM itself (e.g., reinforcement learning or best-of-n distillation), which could compound the reported gains; the paper only uses the scorer at inference.","The retrieval-plus-verifier pattern should transfer to other spectroscopic modalities (MS, UV-Vis, crystallography) wherever a simulator can generate paired spectra and structures, reusing the same two-encoder scorer and MCTS plugin without changing the LLM.","The non-monotonic effect of retrieval count k hints that retrieved substructures can mislead the LLM when they conflict with the evidence; a testable refinement is to gate how many substructures the model sees by the scorer's retrieval confidence, or to allow a 'none of these' option.","The near-zero correlation (0.03) between LLM self-evaluation and oracle similarity suggests that, without an external verifier, self-guided spectral reasoning by LLMs is close to its ceiling; beating the paper's 0.53 scorer correlation becomes a concrete benchmark for future verifiers."],"forward_implications":["The same plugin lifts weaker open models: Llama-3.2-11B's exact-match ACC goes from 0.014 to 0.111, showing the gains are not exclusive to frontier LLMs.","The knowledge base alone is a transferable ingredient: adding it to CoT, Self-Refine, Self-Consistency, MAD, and MCTSr on GPT-4o-mini improves every one of them, with CoT+KB more than tripling baseline ACC (0.037→0.111).","Test-time compute is a direct dial: increasing MCTS iterations from 0 to 8 steadily improves accuracy before saturating, while retrieval count k has an optimum (k=1 on GPT-4o-mini, k=2 on GPT-4o) beyond which irrelevant substructures add noise.","The specialized scorer beats an LLM-based retriever (BM25 over generated descriptions) by 0.120 ACC and 0.169 retrieval hit rate, so a learned spectrum–molecule embedding is also a better knowledge-base bridge.","Since the scorer's reward correlates with oracle similarity at 0.53 while the LLM scorer is at 0.03, the paper ties its gains to fixing a self-affirmation bias in LLM self-evaluation."],"supporting_citations":[{"why":"Supplies the MolPuzzle benchmark and the task formulation (inputs x_ir, x_cnmr, x_hnmr, x_formula) that all experiments measure against.","marker":"Guo et al., 2024"},{"why":"Provides the MCTSr approach—MCTS with self-refine—that K-MSE builds on and includes as a baseline.","marker":"Zhang et al., 2024a"},{"why":"Supplies the Self-Refine critique-and-rewrite loop used in the tree expansion step.","marker":"Madaan et al., 2023"},{"why":"Gives the NT-Xent contrastive objective used to train the molecule-spectrum scorer.","marker":"Chen et al., 2020"},{"why":"ZINC database provides the 9,000 molecules whose simulated C-NMR and H-NMR spectra train the scorer.","marker":"Sterling and Irwin, 2015"},{"why":"MOSES database provides the 4M molecules from which the knowledge base's 593 substructures are extracted.","marker":"Polykovskiy et al., 2020"},{"why":"GIN (Graph Isomorphism Network) is the graph encoder used in the molecule encoder of the scorer.","marker":"Xu et al., 2019"},{"why":"Chain-of-Thought prompting is the primary baseline that each model group is compared against.","marker":"Wei et al., 2022"}],"fun_headline_variants":["Knowledge base plus tree search doubles LLM molecule ID accuracy","Substructure know-how and MCTS reward lift LLM spectral ID by 20%","LLM structure elucidation boosted by chemical knowledge and tree search","Tree search reasoning with spectral scorer improves LLM molecule ID","Knowledge-enhanced MCTS sharpens LLM molecular structure parsing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire gain rests on the molecule-spectrum scorer's cosine similarity being a trustworthy proxy for structural correctness on molecules the search encounters; if the scorer, trained only on simulated ZINC spectra, does not transfer to the benchmark's measured spectra, both knowledge retrieval and final answer selection are guided by noise and the reported improvements should disappear.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge base plus tree search doubles LLM molecule ID accuracy","Substructure know-how and MCTS reward lift LLM spectral ID by 20%","LLM structure elucidation boosted by chemical knowledge and tree search","Tree search reasoning with spectral scorer improves LLM molecule ID","Knowledge-enhanced MCTS sharpens LLM molecular structure parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1551,"prompt_tokens":1017,"completion_tokens":534,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":445}},"tokens_in":633,"tokens_out":534,"duration_ms":5924,"temperature":1.0,"reasoning_tokens":445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:51:31.470133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the rank correlation between the K-MSE reward score and the ground-truth fingerprint similarity on all candidate molecules the search generates across the full MolPuzzle set; if the correlation drops to near zero on novel structures, as the authors find for the LLM-based scorer (0.03), the gains should vanish. A direct ablation that replaces R(a') with random or inverted scores, while keeping the search structure, knowledge base, and rewrite steps intact, would show whether the scorer's signal actually causes the 0.236/0.300 ACC jumps.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Self-Refine critique-and-rewrite loop used in the tree expansion step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the NT-Xent contrastive objective used to train the molecule-spectrum scorer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ZINC database provides the 9,000 molecules whose simulated C-NMR and H-NMR spectra train the scorer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MOSES database provides the 4M molecules from which the knowledge base's 593 substructures are extracted."}],"review_version":1}