{"id":"c18c8450-50ae-4b4f-ae7b-32e76f4100a0","arxiv_id":"2608.16082","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FIRMPO, a preference-optimization framework that re-ranks candidate molecules using molecular-formula matching and infrared-spectrum similarity, substantially improves top-1 structure elucidation accuracy across three datasets.","lead":"This paper introduces a training method that uses chemical rules, matching molecular formulas and comparing infrared spectra, to push AI models toward more chemically reasonable molecule predictions. It reports large gains in top-1 accuracy for structure elucidation from infrared spectra, which matters for automated chemistry and drug discovery.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation's random 85/5/10 splits likely allow near-duplicate molecules across train and test, so the reported FIRMPO gains may reflect memorization rather than improved structure elucidation.","rationale":"I re-read the paper's central claim and its experimental support. The strongest claim is that FIRMPO consistently improves molecular accuracy across datasets and backbones, especially at top-1 (Section 4.2). The largest reported effect is IRtoMol top-1 on NIST WebBook, rising from 36.51% to 55.94%. For this claim to hold, test-set accuracy must reflect generalization to unseen structures, not memorization of near-duplicate molecules. The paper specifies only 'randomly split' in Section 4.1 and Appendix B, with no scaffold split, canonicalization, or duplicate filtering. In chemistry benchmarks, random splits are known to create train/test overlap via shared scaffolds, so generated SMILES can be retrieved from memory. This is especially dangerous for a preference-optimization method that explicitly re-ranks candidates from a beam: if the correct SMILES is memorized, its presence in the beam is guaranteed, and the optimization only needs to learn to move it to rank 1. The reported gains, e.g., top-1 approaching the base model's top-10 accuracy, are consistent with this re-ranking-from-memorization story. I do not claim this is what happened; I claim the current evidence cannot rule it out. A scaffold split would settle it. I also note the paper lacks error bars, which makes small gains like PatchIR QM9S 0.28 to 0.29 uninterpretable, but the leakage issue is the primary concern. Since the reader's verdict was already CONDITIONAL on additional experiments, my read does not change the verdict.","tokens_in":11744,"tokens_out":9089,"duration_ms":86035,"concrete_test":"Reproduce the main comparison (at least IRtoMol Base vs +FIRMPO on NIST WebBook and IBM) using a Bemis-Murcko scaffold split with exact duplicate removal after canonicalization (e.g., RDKit canonical SMILES or InChI), ensuring the same molecule never appears in both train and test. If the Acc@1 gap between Base and +FIRMPO shrinks by more than 50% or falls below one standard error across three seeds, the reported gains are largely memorization artifacts. Additionally, report train/test Tanimoto similarity distributions to confirm that no near-neighbor leakage remains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on random 85/5/10 splits (Section 4.1 and Appendix B) with no scaffold-based separation, canonicalization, or duplicate removal. Molecular datasets such as QM9S, IBM, and NIST WebBook contain many stereoisomers, tautomers, and closely related scaffolds; under random splitting, a substantial fraction of test molecules can share a Bemis-Murcko scaffold or even an exact SMILES with training molecules. For a sequence generation model, exact/near-duplicate memorization is a well-known confound: both the base model and the FIRMPO-tuned model can be boosted by memorizing the target SMILES, and preference optimization is precisely designed to amplify already-generated correct candidates. The paper reports no experiments under scaffold split, no Tanimoto-similarity-controlled split, and no error bars, so the magnitude of the 36.51% to 55.94% NIST top-1 jump cannot be attributed to generalization. This is the most load-bearing weakness because if the gains shrink or invert under leakage-controlled splits, the claim that FIRMPO 'significantly improves molecular structure elucidation accuracy' (Section 4.2) is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes FIRMPO (Formula- and IR-Matched Preference Optimization), a preference-optimization framework for molecular structure elucidation from molecular formulas and infrared spectra. For each training input, candidate structures are generated by a supervised backbone model via beam search and re-ranked using a lexicographic key that combines three signals: identity with the ground-truth SMILES, exact matching of the implied molecular formula, and cosine similarity between the Chemprop-IR-predicted theoretical spectrum and the observed spectrum. A listwise preference loss with tier-aware weighting is then used to fine-tune the backbone. Experiments on QM9S, IBM, and NIST WebBook with IRtoMol, PatchIR, and IR-Bench report improvements in top-k molecular accuracy, formula accuracy, and IR similarity. The method is model-agnostic and changes only the training objective.","tokens_in":11991,"tokens_out":4645,"duration_ms":43735,"significance":"If the reported results hold, FIRMPO would be a practically useful and conceptually clean extension of preference optimization to chemistry: it is a plug-and-play training objective, the tiered preference structure is chemically motivated, and the paired comparison against supervised baselines across three datasets is a reasonable evaluation design. The paper also ships explicit mathematical definitions of the loss and the preference construction algorithm, which is a strength. However, the current experimental support is not yet convincing because the random data splits may allow near-duplicate molecules between training and test, the preference key includes the ground-truth identity as the highest-priority term without an ablation that removes it, and no uncertainty estimates or forward-model quality checks are reported. The central claim of consistent improvement is therefore plausible but not yet established at the level required for a definitive publication.","major_comments":[{"comment":"The evaluation uses random 85/5/10 splits with no scaffold-based separation, canonicalization, or near-duplicate removal. Molecular datasets such as QM9S, IBM, and NIST WebBook contain many stereoisomers, tautomers, and closely related scaffolds, and random splitting can place near-identical molecules in both training and test sets. For a SMILES-generation model, this is a known confound: the reported gains, including the NIST IRtoMol top-1 jump from 36.51% to 55.94%, may reflect memorization of molecular scaffolds rather than generalization of the preference-optimization method. The authors should repeat the main experiments under a scaffold split or a Tanimoto-similarity-controlled split, and report the number of test molecules that share a Bemis-Murcko scaffold with any training molecule.","section":"Section 4.1 / Appendix B"},{"comment":"The preference key kx(y) assigns highest priority to mMol(y,y*), i.e., whether the candidate is identical to the ground-truth SMILES. This means the preference ordering is not purely driven by the chemical feedback of formula matching and IR consistency; it is partly a direct supervised re-ranking toward the ground-truth answer. The ablations in Section 4.4 remove either the IR term or the formula term, but never remove the mMol term, so the reported improvements cannot be attributed to the chemical feedback alone. An additional ablation with kx(y) = (m_hatF(y,F), sIR(y,X)) is necessary to quantify the contribution of the chemical signals versus the ground-truth identity term.","section":"Section 3.3, Eq. (6)-(7)"},{"comment":"All results are reported as point estimates without standard deviations or multiple seeds. Many of the claimed improvements are very small, for example QM9S PatchIR Acc@1 from 0.28 to 0.29 and QM9S IR-Bench Acc@10 unchanged at 22.65. Without error bars or significance testing, it is impossible to assess whether these differences are meaningful. The authors should provide means and standard deviations over at least three random seeds, and ideally a paired significance test for the main comparisons.","section":"Section 4.2, Table 1"},{"comment":"The IR spectral consistency signal relies on Chemprop-IR as the fixed forward model g(y), but the manuscript does not evaluate the accuracy or reliability of this forward model on the datasets used. If Chemprop-IR produces inaccurate theoretical spectra, the IR-consistency preference may be systematically misleading, and the reported gains in IR similarity may not reflect true chemical consistency. The authors should report the accuracy of Chemprop-IR predictions on held-out spectra from these datasets, or include an ablation using an oracle forward model or an alternative spectral predictor.","section":"Section 3.3, Eq. (5)"}],"minor_comments":[{"comment":"The conclusion states that gains persist at top-5 and top-10, but Table 1 shows several unchanged top-10 values, including QM9S IR-Bench Acc@10 (22.65 to 22.65), QM9S PatchIR Acc@10 (2.43 to 2.43), IBM PatchIR Acc@10 (11.75 to 11.75), and NIST PatchIR Acc@10 (47.28 to 47.28). This statement should be qualified to say that top-1 gains are consistent while top-k gains are dataset- and backbone-dependent.","section":"Section 5 / Table 1"},{"comment":"For several FIRMPO variants, the formula accuracy is identical across all k, for example 99.88 at AccF@1, AccF@5, and AccF@10 on QM9S IRtoMol. This pattern is surprising and should be explicitly explained; it may indicate that formula-matched candidates are always placed before formula-mismatched candidates, but the reader needs to know whether the metric is redundant at k>1 for these models.","section":"Section 4.2, Table 1"},{"comment":"The hyperparameter values are not reported: the manuscript states that the learning rate, batch size, and beta are tuned on the validation set, but gives no concrete values or ranges for any of the three datasets. A short table with the chosen values would make the experiments reproducible.","section":"Appendix B"},{"comment":"The notation K2(x) is used both for the tier index and for the count of candidates in the union of tiers K1 and K2. This is confusing; renaming the counts to, e.g., n1(x), n2(x), n3(x) would improve readability.","section":"Section 3.4, Eq. (9)"},{"comment":"The comparison with KPO is only performed on IRtoMol. Since the main paper claims that FIRMPO is backbone-agnostic and consistently better than generic listwise PO, the comparison should be repeated for at least one additional backbone, or the claim should be narrowed accordingly.","section":"Appendix C"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know before reading this paper: FIRMPO is a clean, well-written preference-optimization framework for IR-based structure elucidation, and the reported top-1 gains are large. But the evaluation has some unaddressed weaknesses that keep me from fully trusting those numbers yet.\n\nWhat's actually new: the tiered preference scheme — ground-truth identity first, then formula match, then IR similarity — with the weighting terms in Eq. (9) is a thoughtful adaptation of listwise PO to a structured-generation problem where chemical validity matters. The paper is honest about putting ground-truth identity at the top of the lexicographic key (Eq. 6-7), so the method is essentially supervised re-ranking with a chemical tie-breaker. That's not a flaw; it's a reasonable design. The ablations (w/o IR, w/o Formula) show each signal contributes, and the model-agnostic framing means the idea could be applied to other backbones.\n\nWhere it gets soft:\n\n1. The random 85/5/10 split (Section 4.1, Appendix B) is a real concern. Molecular datasets like QM9S and IBM contain many near-duplicate scaffolds; without scaffold-based separation or at least Tanimoto-controlled split, the reported gains could reflect memorization rather than generalization. The paired comparison (Base vs +FIRMPO) partially mitigates this, since the base also benefits from memorization, but preference optimization is specifically designed to amplify already-generated correct candidates, so the extra gain could be memorization on top of memorization.\n\n2. No error bars, no multiple seeds. Some numbers in Table 1 are suspiciously constant — formula accuracy identical across k for the FIRMPO variants, which begs an explanation. It could be a rounding artifact, but it needs a footnote or a separate verification.\n\n3. The forward model Chemprop-IR is used both to construct preferences and to compute the Sim@k evaluation metric. If that model is biased on these datasets, the IR-consistency improvements are partly circular. The paper does not validate Chemprop-IR's accuracy on QM9S/IBM/NIST, and that should be a required experiment.\n\n4. The missing ablation is the one where ground-truth identity is removed from the preference key. That's the ablation that would tell us whether FIRMPO does anything beyond memorizing correct answers from the training set. The current w/o IR and w/o Formula ablations still keep the ground-truth tier, so they don't isolate the method's contribution.\n\nThese are fixable with additional experiments, so the paper deserves a serious referee — but not acceptance as is. The target audience is people working on ML for spectroscopy or preference optimization for structured outputs; they'll find the method worth engaging with. Send it to peer review with a clear expectation of a major revision addressing the leakage-controlled split and the forward-model validation.","headline":"Sensible preference-optimization method for IR-based structure elucidation, but the reported gains rest on splits that likely allow memorization and on an unvalidated forward model; needs a leakage-controlled re-run before I'd trust the numbers.","tokens_in":12520,"tokens_out":2661,"duration_ms":25444},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Preference optimization with chemical feedback raises top-1 accuracy of IR-to-structure models consistently across three datasets and three backbones.","keywords":["Infrared spectroscopy","molecular structure elucidation","preference optimization","chemical feedback","molecular formula matching","IR spectral consistency","SMILES generation","top-1 accuracy"],"falsifier":"Re-run the identical FIRMPO pipeline on a scaffold-separated split (no test molecule shares its ring scaffold with any training molecule) or after removing every test molecule with a near-identical training analogue; if the top-1 accuracy gains vanish or shrink to a few points, the central claim of consistent improvement fails. A second check is to measure the gain separately on test molecules whose nearest training neighbor is close versus distant, since the claim predicts roughly equal gains in both groups.","tokens_in":11566,"feed_emoji":"🧪","tokens_out":11702,"duration_ms":87147,"temperature":0.7,"pith_summary":"Infrared-based machine learning models often include the correct molecule somewhere in a beam of candidates yet rank chemically impossible isomers at the top. FIRMPO is a post-training preference-optimization step that turns two pieces of chemical feedback into ranking preferences: exact matching of the candidate's molecular formula to the input formula, and cosine similarity between the candidate's theoretical IR spectrum and the observed spectrum. Under a tiered lexicographic ordering (ground-truth structure first, then formula-matched candidates sorted by IR similarity, then the rest), the method re-ranks the candidate beam and re-trains the model to prefer that order. Across three IR datasets and three backbone models, the paper reports consistent gains that are largest at top-1, for example IRtoMol's accuracy on the NIST Chemistry WebBook rising from 36.51% to 55.94%. The reason this matters is that the approach injects hard chemical constraints into any spectrum-to-structure model without changing its architecture, narrowing the gap between top-1 and top-k accuracy that limits automated structure elucidation.","feed_headline":"Chemical feedback lifts top-ranked IR structure accuracy 19 points","feed_subtitle":"A plug-and-play preference step makes IR-based molecule models rank chemically consistent candidates first.","key_machinery":"The load-bearing object is the tiered lexicographic preference key\n$$k_x(y) = (m_{\\mathrm{Mol}}(y,y^*),\\ m_{\\hat{F}}(y,F),\\ s_{\\mathrm{IR}}(y,X)),$$\nwhere $m_{\\mathrm{Mol}}$ is 1 when the candidate SMILES is identical to the ground-truth structure, $m_{\\hat{F}}$ is 1 when the candidate's implied molecular formula equals the input formula, and $s_{\\mathrm{IR}}$ is the cosine similarity between the theoretical IR spectrum predicted from the candidate by a fixed learned forward model and the observed spectrum. This key converts qualitative chemical feedback into a total order over the beam: ground-truth candidates first, then formula-matched candidates sorted by IR similarity, then remaining candidates sorted by IR similarity. The FIRMPO loss in Eq. (9) is a listwise preference objective with task-specific weights that upweight comparisons anchored at ground-truth candidates, assign weaker comparison strength within the formula-matched region, and assign stronger strength to contrasts against formula-mismatched candidates, so the model learns cross-tier separation rather than an arbitrary ordering within a tier.","core_discovery":"The paper's central claim is that the top-ranked outputs of supervised IR-to-molecule models are unreliable largely because of a ranking failure, not a generation failure: the correct structure is often present in the beam but out-ranked by candidates that violate chemical constraints. FIRMPO treats those constraints as preference signals, computing for each candidate a lexicographic key $k_x(y)$ whose first component is whether the candidate equals the ground-truth structure, whose second is whether the candidate's implied molecular formula exactly equals the input molecular formula, and whose third is the cosine similarity between the candidate's theoretical IR spectrum and the observed spectrum. Sorting the beam by this key produces a chemically preferred list, and a listwise preference-optimization loss trains the model to place top-tier, formula-matching, spectrum-consistent candidates at the front. On QM9S, IBM, and the NIST Chemistry WebBook, FIRMPO improves molecular accuracy at top-1, top-5, and top-10 across the IRtoMol, PatchIR, and IR-Bench backbones, with the largest reported jump being top-1 accuracy on the experimental NIST set from 36.51% to 55.94%.","pith_inferences":["The mechanism suggests the same feedback scheme should transfer to other spectroscopy-to-structure tasks, such as MS/MS or NMR-guided elucidation, where analogous constraints (elemental composition, substructure matches) are known; the paper does not test this.","Because the preference signal is built from a fixed theoretical-IR predictor, FIRMPO inherits that predictor's systematic errors; jointly training or ensembling the spectrum predictor could make the preference signal more robust to noisy experimental spectra.","The random-split evaluation leaves open whether the gains are genuine generalization or partly memorization of common scaffolds; a scaffold-separated split is the sharpest untested check of the central claim.","The method's gain should scale with beam diversity: a candidate beam that never contains the correct molecule cannot be rescued by any preference signal, so measuring FIRMPO's improvement as a function of beam width would quantify the ceiling identified in the paper's limitations."],"forward_implications":["FIRMPO is model-agnostic: the same preference-construction procedure can be attached to any existing IR-to-SMILES backbone, so published supervised models can be upgraded without changing their architecture or decoding procedure.","Because the preference signals are chemical constraints, the optimized models report higher formula accuracy and higher IR spectral similarity at the top ranks, not only higher molecular hit rates.","The gap between top-1 and top-k molecular accuracy narrows, attacking the specific failure mode—correct molecule generated but ranked too low—that limits practical use in chemical analysis.","Ablations show the two feedback signals are complementary: dropping formula matching lowers formula accuracy, dropping IR consistency weakens the ranking among formula-correct candidates, and the full tiered signal performs best.","The gains persist at top-5 and top-10, indicating that FIRMPO improves overall ranking quality rather than only the first position."],"supporting_citations":[{"why":"Supplies the IRtoMol base model and the conditional generation task (molecular formula + IR spectrum to SMILES) that FIRMPO is applied to.","marker":"[2]"},{"why":"Provides the fixed forward model g(y) that predicts theoretical IR spectra from SMILES, used to compute the spectral-consistency preference signal.","marker":"[15]"},{"why":"Contributes the QM9S synthetic IR dataset used as one of the three evaluation benchmarks.","marker":"[26]"},{"why":"Contributes the IBM synthetic multimodal spectroscopy dataset used as the second evaluation benchmark.","marker":"[3]"},{"why":"Contributes the NIST Chemistry WebBook experimental IR spectra used as the third evaluation benchmark and the source of the largest reported gain.","marker":"[13]"},{"why":"Provides the IR-Bench backbone, one of the three supervised baselines FIRMPO is evaluated on.","marker":"[4]"},{"why":"Provides the PatchIR backbone, one of the three supervised baselines FIRMPO is evaluated on.","marker":"[25]"},{"why":"Supplies the generic listwise preference optimization baseline KPO, compared in Appendix C to show FIRMPO's improvement is not merely generic preference optimization.","marker":"[5]"}],"fun_headline_variants":["Chemical feedback re-ranks IR molecule candidates, boosting top-1 accuracy 19 points","Chemical feedback lifts IR molecule ranking: top-1 jumps 19 points on NIST","FIRMPO: chemical feedback re-ranks IR structures for top-1 accuracy gains","Chemical feedback improves IR molecule ranking: 19-pt top-1 accuracy jump"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the random 85/5/10 train/test splits make the test set chemically independent of the training set; the paper does not separate molecules by scaffold or remove near-duplicates, so if chemically similar structures appear on both sides, the reported gains could reflect memorization of common scaffolds rather than genuine generalization of the preference optimization.","fun_headline_variants_meta":{"raw":{"variants":["Chemical feedback re-ranks IR molecule candidates, boosting top-1 accuracy 19 points","Chemical feedback lifts IR molecule ranking: top-1 jumps 19 points on NIST","FIRMPO: chemical feedback re-ranks IR structures for top-1 accuracy gains","Chemical feedback improves IR molecule ranking: 19-pt top-1 accuracy jump"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000779,"raw_usage":{"total_tokens":3478,"prompt_tokens":1012,"completion_tokens":2466,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2374}},"tokens_in":628,"tokens_out":2466,"duration_ms":16527,"temperature":1.0,"reasoning_tokens":2374,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T20:42:53.441921+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-run the identical FIRMPO pipeline on a scaffold-separated split (no test molecule shares its ring scaffold with any training molecule) or after removing every test molecule with a near-identical training analogue; if the top-1 accuracy gains vanish or shrink to a few points, the central claim of consistent improvement fails. A second check is to measure the gain separately on test molecules whose nearest training neighbor is close versus distant, since the claim predicts roughly equal gains in both groups.","supporting_citations":[{"cited_title":"A deep learning model for predicting selected organic molecular spectra.Nature Computational Science, 3(11):957–964, 2023","cited_arxiv_id":null,"evidence_quote":"Contributes the QM9S synthetic IR dataset used as one of the three evaluation benchmarks."},{"cited_title":"The nist chemistry webbook: A chemical data resource on the internet.Journal of Chemical & Engineering Data, 46(5):1059–1063, 2001","cited_arxiv_id":null,"evidence_quote":"Contributes the NIST Chemistry WebBook experimental IR spectra used as the third evaluation benchmark and the source of the largest reported gain."},{"cited_title":"Setting new benchmarks in ai-driven infrared structure elucidation.Digital Discovery, 4(7):1936–1943, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the IR-Bench backbone, one of the three supervised baselines FIRMPO is evaluated on."},{"cited_title":"Transformer-based models for predicting molecular structures from infrared spectra using patch-based self-attention","cited_arxiv_id":null,"evidence_quote":"Provides the PatchIR backbone, one of the three supervised baselines FIRMPO is evaluated on."},{"cited_title":"K-order Ranking Preference Optimization for Large Language Models","cited_arxiv_id":"2506.00441","evidence_quote":"Supplies the generic listwise preference optimization baseline KPO, compared in Appendix C to show FIRMPO's improvement is not merely generic preference optimization."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}