REVIEW 5 major objections 6 minor 16 references
A new training objective aligns the hidden states of chemically equivalent SMILES prefixes, resolving trajectory divergence in autoregressive molecular models.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 05:39 UTC pith:R4NJBLVY
load-bearing objection Same-suffix contrastive alignment is a real idea worth refereeing, but the abstract promises much more than the body shows, and the invariance mechanism is asserted more than proven. the 5 major comments →
SIGMA: Semantic Identifier Grouping for Molecular Autoregression
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that explicit token-level alignment of shared future trajectories induces geometric invariance in the latent space of a causal molecular language model. The authors construct positive pairs from two randomized SMILES traversals that are verified, by a structure oracle, to reconstruct the same molecule when completed with an identical suffix, and a negative prefix that completes to a different structure. A projection head separates syntactic features (used by the language-model objective) from structural semantics (used by the contrastive loss), and the loss is averaged over every suffix token position. The paper reports a Trajectory Invariance Score of 0.041—down
What carries the argument
The central object is the dense trajectory alignment objective: an InfoNCE-style loss computed per suffix token position between two projected hidden-state sequences that share a certified identical future, with a finite relative margin separating a structurally non-equivalent prefix. The projection head is a two-layer bottleneck (768 to 256 to 128) that decouples syntax from semantics, so the language-model objective retains full syntactic detail while the contrastive loss operates on structure-only representations. The theoretical bridge is the claim that gradient flow through the shared suffix attention layers enforces Encoder(p_u) ≈ Encoder(p_v), propagating invariance backward from the
Load-bearing premise
The stated mechanism assumes that aligning hidden states at shared suffix positions push the earlier prefix representations together through backpropagation, even though the loss never directly supervises those prefix states.
What would settle it
Run SIGMA with a stop-gradient applied to all prefix hidden states so that suffix-align losses cannot update the prefix encoder; if the Trajectory Invariance Score still falls to roughly 0.04, the paper's gradient-flow explanation is not the cause and the invariance must arise elsewhere.
If this is right
- Sequence-based molecular generators can reach distributional fidelity competitive with graph-based models (FCD 0.752) while keeping near-perfect validity (>99.8%).
- The invariance transfers across string representations: the objective improves FCD in all four SELFIES blocks and in two of four SMILES blocks, so the approach is not tied to SMILES-specific syntax.
- Under a fixed RL optimizer, SIGMA's prior yields 20-40% more unique scaffolds on tasks like Osimertinib at comparable peak scores, indicating less mode collapse.
- IsoBeam raises the number of unique scaffolds discoverable at beam width 50,000 to roughly double that of standard beam search, by removing isomorphic duplicates.
- Compute-matched ablations on full-corpus blocks show that chemically correct state correspondence—not contrastive pressure alone—is the effective ingredient.
Where Pith is reading between the lines
- The same suffix-alignment recipe should apply to any autoregressive model over a one-to-many serialization—for instance, code with equivalent compiler-normalized forms, or RNA secondary structures—where a cheap oracle can certify that two prefixes share a common future.
- If the gradient-flow bridge is confirmed, it suggests a general design pattern: enforce permutation invariance without architectural changes by aligning latent trajectories at points where the future is shared, which could be cheaper than explicit graph encoders at large scale.
- A natural testable extension is to relax the oracle certification to purely syntactic validity; if that suffices, the method could be applied to unlabelled string corpora where no structure-checking tool exists.
- The probe-suffix protocol (appending chemically stable caps to incomplete prefixes) is an inventive workaround for validation of partial molecules, and one could test whether a learned completion model could replace hand-crafted caps.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SIGMA, a token-level contrastive objective for autoregressive chemical language models. SIGMA constructs positive pairs of SMILES prefixes that, while syntactically different, share a common valid suffix and correspond to the same molecular subgraph; it aligns the projected hidden states of these suffix tokens with an InfoNCE-style loss, and repels structural negatives. The authors also propose IsoBeam, a beam-search variant that prunes candidates whose partial sequences parse to already-seen molecular graphs. Experiments on ZINC-250k report improved FCD (0.752) and higher scaffold diversity in PMO optimization, with additional visualizations of latent-space invariance. The abstract, however, claims far broader validation across four datasets, eight representation-dataset blocks, and six property-prediction benchmarks, none of which appear in the body.
Significance. If the central mechanism were established, dense trajectory alignment would be a useful idea for reducing the SMILES ambiguity problem in autoregressive molecular models, and the reported ZINC-250k FCD improvement is potentially interesting. The paper provides detailed hyperparameters and a clear description of the view-construction protocol, which is a strength. However, the significance is undercut by (i) the large gap between the abstract's claims and the experiments actually reported, (ii) the unsupported theoretical bridge connecting suffix alignment to prefix invariance, and (iii) a fundamental flaw in the IsoBeam pruning criterion. The current evidence does not support the general claims made in the abstract.
major comments (5)
- [Abstract and §4] The abstract claims FCD reductions across 'eight representation-dataset blocks' (four datasets under SMILES and SELFIES) and mean predictive improvements on 'all six molecular property benchmarks', with paired 95% confidence intervals against every control. The body's experiments (§4) report only ZINC-250k unconditional generation and PMO optimization; no SELFIES, QM9, or property-prediction experiments appear anywhere in the manuscript or appendices. This is a load-bearing omission: the headline results are not reproducible from the submitted text. The authors must either include the promised experiments or rewrite the abstract to describe only the ZINC-250k and PMO results.
- [§3.4, Eq. (7)] The claim that minimizing the suffix contrastive loss enforces Encoder(p_u) ≈ Encoder(p_v) via gradient flow is asserted without proof. The loss directly supervises only suffix-token hidden states; the effect on prefix encoders is an indirect consequence that is not derived. Since this bridge is the stated mechanism for resolving prefix trajectory divergence, it is load-bearing. A formal derivation or a controlled experiment (e.g., measuring prefix-representation alignment with versus without the suffix objective, holding data and architecture fixed) is needed before the invariance claim can be accepted.
- [§3.2, Structural Negatives] The negative-sampling condition H(Mol(p_neg ⊕ s)) ≠ H(G) is not well-defined for incomplete SMILES prefixes. If p_neg is not a valid SMILES fragment, Mol(p_neg ⊕ s) may fail; the paper does not specify how such cases are handled. If negatives are predominantly invalid fragments, the contrastive loss may not be learning the intended chemical discrimination, and the reported gains could partly reflect a trivial filtering artifact. The construction of negatives should be described precisely, including how incomplete prefixes are made parseable (e.g., by the probe-suffix protocol) and what fraction of sampled negatives are actually usable.
- [Algorithm 1 (IsoBeam)] The pruning criterion is based on the InChIKey of the fully parsed valid molecule for a partial sequence. A partial SMILES such as 'c1ccccc1' is parsed by RDKit as benzene, but when it appears as a prefix of 'c1ccccc1C(=O)O' it represents a phenyl fragment with an open valence, not a complete benzene molecule. Thus two partial paths that are syntactically different but both parse to the same complete molecule may be incorrectly considered isomorphic, while two paths with different open growth points may be mistakenly deduplicated. This undermines the claim that IsoBeam 'prunes equivalent paths' and could artificially inflate the scaffold-diversity results in §4.4. The algorithm needs to encode the open growth point (e.g., using dummy-atom or partial-graph canonicalization) rather than relying on the InChIKey of the parsed molecule.
- [§4.1 and Appendix F.3 (TIS)] The Trajectory Invariance Score (TIS) measures cosine distance between hidden states of equivalent prefixes, which is very close to the objective that SIGMA optimizes (via the claimed backpropagation into prefix encoders). Reporting TIS as the main evidence of 'geometric invariance' is therefore partly circular. To convince readers that the method improves structural awareness beyond the training objective, the paper should include external evaluations that were not directly optimized for, such as property-prediction accuracy, scaffold-hopping success on held-out targets, or a retrieval-based analysis of latent space.
minor comments (6)
- [Title] The arXiv title 'SIGMA: Semantic Identifier Grouping for Molecular Autoregression' differs from the full-text title 'Structure-Invariant Generative Molecular Alignment...'. Please unify.
- [§3.3 and Appendix B.1] The projection head dimension is inconsistent: Section 3.3 states d_proj = 128, while Appendix B.1 says the head maps to 256. Clarify which was used.
- [Appendix D, Eq. (8)] The auxiliary objective in Appendix D is written with hidden states h, whereas the main text Eq. (5) uses projected states z. This inconsistency should be resolved, since the projection head is described as essential for the syntax-semantics trade-off.
- [Figures] Figures 4–8 are referenced in the text, but the manuscript as provided contains only captions without the actual images. Please ensure all figures are included.
- [References] There are minor reference errors (e.g., 'Ar-Pous' should be 'Arús-Pous'; the SimCTG citation in §2 appears to reference a TACL paper with a different title).
- [Table 1] The PMO table does not clearly state the baseline configuration. The checkmark column presumably indicates SIGMA, but the baseline model and training details should be explicit.
Circularity Check
TIS invariance evidence is close to the training objective, but headline FCD and scaffold-diversity results are externally grounded.
specific steps
-
fitted input called prediction
[Section 3.4 (Eqs. 5-7); Section 4.2/Table 2; Appendix F.3]
"we average this objective over the entire suffix length L_suf: L_SIGMA = 1/L_suf Σ_{t=1}^{L_suf} L^(t)_SIGMA (6) ... effectively enforcing: Encoder(pu)≈Encoder(pv) (7). Appendix F.3: TIS is defined as the expected cosine distance over the test set: TIS=E_{M∼D_test}[1−sim(h(s1),h(s2))]"
The training loss is a cosine-similarity contrastive objective on latent states of equivalent strings; TIS is the complementary cosine-distance metric between latent states of equivalent prefixes. Section 3.4 explicitly claims the loss 'effectively enforces' that prefix equivalence, so reporting TIS reductions as evidence of geometric invariance partly re-measures the training objective rather than providing an independent test. The circularity is partial because TIS uses backbone h rather than the projection z and evaluates prefixes rather than suffix tokens; moreover the FCD and scaffold-diversity results do not reduce to the loss.
full rationale
SIGMA's central derivation chain is mostly self-contained. The headline FCD=0.752 is computed against test-set ChemNet activations, and the PMO scaffold-diversity numbers come from a fixed REINVENT protocol, so they are external to the L_SIGMA objective. IsoBeam's pruning rule is an explicit algorithmic definition, not a hidden input, and its larger #Scaf is a designed consequence rather than a circular derivation. I found no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled by citation. The only near-circular element is the TIS / state-correspondence evidence: the paper trains with cosine-similarity alignment of equivalent latent states and then evaluates a cosine-distance metric on equivalent latent states, with Section 3.4 asserting the loss 'effectively enforces' the very prefix equivalence TIS measures. This makes part of the invariance analysis self-confirmatory, but it does not undermine the externally grounded FCD and optimization results. Separately, the abstract promises four datasets and six property benchmarks while the body reports only ZINC-250k and PMO; that is a missing-support/correctness issue, not a circularity issue, so it does not affect the score.
Axiom & Free-Parameter Ledger
free parameters (4)
- contrastive temperature tau =
0.1
- loss weight lambda =
not reported
- projection head output dimension d_proj =
128 (Section 3.3) vs 256 (Appendix B)
- probe suffix sprobe =
e.g., methyl cap or ring closure
axioms (5)
- standard math SMILES strings are valid serializations and the autoregressive factorization p_theta(S)=prod_t p_theta(x_t|x_<t) is the correct generative model
- domain assumption InChIKey hash equality certifies molecular graph isomorphism (Eq. 3 and Algorithm 1)
- ad hoc to paper Aligning hidden states at shared suffix tokens backpropagates through the causal Transformer to make prefix encoders equivalent (Eq. 7)
- domain assumption A partial string that parses as a valid molecule can be treated as a complete molecule for uniqueness pruning (Algorithm 1)
- ad hoc to paper H(Mol(p_neg + s)) != H(G) is a sufficient negative signal
invented entities (1)
-
Dummy atom / wildcard anchor at cut sites
no independent evidence
read the original abstract
Autoregressive molecular models assign probability to molecular serializations even though chemical identity is invariant to serialization. Equivalent serializations can therefore represent a common molecular identity yet induce inconsistent next-token decisions. Randomized strings broaden exposure, but do not reveal which intermediate decisions should agree. We introduce SIGMA, a dense suffix-position objective built from chemically certified same-suffix triplets: two equivalent histories, one non-equivalent history, and a shared suffix. SIGMA aligns corresponding pre-token hidden states along the continuation and separates the negative to a finite relative margin, leaving the language-model objective, decoder, and inference procedure unchanged. We compare SIGMA with canonical training, randomized-serialization training, and last-token alignment across four datasets under SMILES and SELFIES. Across the eight representation-dataset blocks, SIGMA yields clear test-reference Frechet ChemNet Distance reductions in six: all four SELFIES domains and QM9 and ZINC under SMILES, with paired 95% confidence intervals below zero against every control. Position-wise analyses show improved state correspondence, chemical discrimination, and next-token agreement while preserving between-molecule information. On two full-corpus ZINC blocks, compute-matched ablations identify chemically correct state correspondence as the effective ingredient. Beyond generation, SIGMA improves mean predictive performance on all six molecular property benchmarks and reduces sensitivity to equivalent molecular serializations on every task.
Figures
Reference graph
Works this paper leans on
-
[3]
Table 3.Hyperparameter Settings for SIGMA Training. Hyperparameter Value Optimization Optimizer AdamW Peak Learning Rate 5×10 −4 Weight Decay 0.01 Beta parameters (0.9,0.999) Gradient Clipping (Norm) 1.0 Batch Size 64 Training Epochs 50 Warmup Steps 2000 Contrastive Learning Temperature (τ) 0.1 Augmentation Views (N) 2 Projection Head Dim 256 Architecture...
2000
-
[6]
Fabian, B., Edlich, T., Gaspar, H., Segler, M., Meyers, J., Fiscato, M., and Ahmed, M. Molecular representation learning with language models and domain-relevant aux- iliary tasks.arXiv preprint arXiv:2011.13230,
Pith/arXiv arXiv 2011
-
[7]
Lost in translation: Chemical language models and the misunderstanding of molecule structures
Ganeeva, V ., Sakhovskiy, A., Khrabrov, K., Savchenko, A., Kadurin, A., and Tutubalina, E. Lost in translation: Chemical language models and the misunderstanding of molecule structures. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 12994– 13013,
2024
-
[8]
Honda, S., Shi, S., and Ueda, H. R. Smiles transformer: Pre- trained molecular fingerprint for low data drug discovery. arXiv preprint arXiv:1911.04738,
Pith/arXiv arXiv 1911
-
[9]
Strategies for pre-training graph neural networks.arXiv preprint arXiv:1905.12265,
Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V ., and Leskovec, J. Strategies for pre-training graph neural networks.arXiv preprint arXiv:1905.12265,
Pith/arXiv arXiv 1905
-
[12]
Roberta: A robustly optimized bert pretraining approach
Liu, Y ., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V . Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692,
Pith/arXiv arXiv 1907
-
[13]
Shi, C., Xu, M., Zhu, Z., Zhang, W., Zhang, M., and Tang, J. Graphaf: a flow-based autoregressive model for molec- ular graph generation.arXiv preprint arXiv:2001.09382,
Pith/arXiv arXiv 2001
-
[2005]
Mistral 7b.arXiv preprint arXiv:2310.06825,
Jiang, A., Sablayrolles, A., Roux, A., et al. Mistral 7b.arXiv preprint arXiv:2310.06825,
-
[2015]
Su, Y ., Liu, F., Meng, Z., Lan, T., Shu, L., Shareghi, E., and Collier, N. Tacl: Improving bert pre-training with token-aware contrastive learning.arXiv preprint arXiv:2111.04198,
-
[2017]
M., Cox, S., Schilter, O., Baldassari, C., White, A
Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., and Schwaller, P. Chemcrow: Augmenting large- language models with chemistry tools.arXiv preprint arXiv:2304.05376,
-
[2019]
Bagal, V ., Aggarwal, R., Vinod, P., and Priyakumar, U. D. Molgpt: molecular generation using a transformer- decoder model.Journal of chemical information and modeling, 62(9):2064–2076,
2064
-
[2020]
Bert: Pre-training of deep bidirectional transformers for lan- guage understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. InProceedings of the 2019 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186,
2019
-
[2021]
Bjerrum, E. J. Smiles enumeration as data augmentation for neural network modeling of molecules.arXiv preprint arXiv:1703.07076,
-
[2022]
Wang, Z., Shi, J., Heess, N., Gretton, A., and Titsias, M. K. Learning-order autoregressive models with ap- plication to molecular graph generation.arXiv preprint arXiv:2503.05979,
-
[2023]
Chithrananda, S., Grand, G., and Ramsundar, B. Chemberta: large-scale self-supervised pretraining for molecular prop- erty prediction.arXiv preprint arXiv:2010.09885,
Pith/arXiv arXiv 2010
-
[2024]
Druggpt: a gpt-based strategy for designing potential ligands targeting specific proteins.bioRxiv, pp
Li, Y ., Gao, C., Song, X., Wang, X., Xu, Y ., and Han, S. Druggpt: a gpt-based strategy for designing potential ligands targeting specific proteins.bioRxiv, pp. 2023–06,
2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.