Pith. sign in

REVIEW 3 major objections 5 minor 62 references

CROP: Integrating Topological and Spatial Structures via Cross-View Prefixes for Molecular LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CROP's claim: jointly resampling molecular graphs and images into short SMILES-guided prefixes improves molecule captioning, IUPAC naming, and property prediction over sequence-only and graph-only baselines.

desk verdict The cross-view prefix resampler is a real idea, but the reported gains are muddied by what looks like test-set selection of the LLM partition. read the letter →

arxiv 2508.06917 v1 pith:WW6FMLEI submitted 2025-08-09 q-bio.QM cs.AI

Jianting Tang , Yubo Wang , Haoyu Cao , Linli Xu This is my paper
classification q-bio.QMcs.AI
keywords MultimodallargelanguagemodelsfusionMoleculargraphimageCross-viewprefixesMoleculecaptioningIUPACnamepredictionProperty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chemical language models that read only SMILES strings miss the molecule's geometry and atom-level connectivity. CROP tries to fix that by feeding the language model two extra views—the molecular graph (which atoms are bonded to which) and the molecular image (the molecule's spatial shape)—and compressing both into short prefixes before they enter the model. The compression is guided by the model's own SMILES representation, so the structural information is selected with chemical knowledge rather than blindly. If CROP is right, sequence-plus-structure beats sequence-alone on generative and predictive chemistry tasks, and it does so with small extra cost: ten prefix tokens replace hundreds of raw embeddings. Across molecule captioning, IUPAC name prediction, and property prediction, the paper reports consistent gains over baselines that use only sequences or only graphs.

What carries the argument

The SMILES Guided Resampler (SGR) and the Structural Embedding Gate (SEG). SGR is a stack of transformer cross-attention layers in which the query stream is the SMILES guidance $Z_S$ produced by the language model's lower layers, and the key/value stream is the concatenation $[Z_G, Z_I, Z_S]$ of graph, image, and SMILES embeddings; the output $Z = \mathrm{SGR}(Z_S, [Z_G, Z_I, Z_S])$ is a fixed $b \times w \times d$ tensor. SEG then produces, for every upper-layer $i$, a prefix $\hat{Z}_i = P_i Z$, where $P_i$ is a softmax-weighted vector over the $b$ groups, so the model can have any lower/upper split and can inject structural information at many layers. The mechanism's two claimed jobs are

What would settle it

For each partition $(b,u)$ in Figure 7, compute both validation and test BLEU-2 on PubChem324k captioning; if the partition that maximizes test BLEU-2 (the one reported) is not the one that maximizes validation BLEU-2, then the reported 44.9 versus 38.7 gain over MolCA is inflated by selecting on the test set. Repeating the comparison with partitions frozen by validation would settle it.

Watch

Extended reading notes

Core claim

CROP partitions a chemistry-pretrained language model (Galactica) into a lower segment that reads the SMILES string and an upper segment that receives both SMILES and structural prefixes. The lower segment produces a fixed-length set of hidden 'SMILES guidance' vectors; a cross-attention resampler then uses those vectors as queries against the concatenated graph and image embeddings as keys and values, returning a compact structural embedding $Z \in \mathbb{R}^{b \times w \times d}$. A gating module converts this into layer-wise prefixes. The paper's central claim is that this jointly resampled, SMILES-guided prefix route delivers the best of both structural views: on PubChem324k molecule ca

Load-bearing premise

The headline comparisons assume the per-task choice of how the language model's layers are split—how many handle only the SMILES and how many also see the structural prefixes—was made on a validation set rather than by peeking at test results; the paper does not say so.

Editorial extensions

If this is right

  • CROP's three-view variant ($S+G+I$) outperforms its own two-view variants on nearly every metric in Tables 1-3, so the paper claims topological and spatial structure are complementary rather than redundant for these tasks.
  • Adding a new structural view is claimed to be cheap: its embeddings are concatenated into SGR's key/value stream, and the prefix length stays fixed at $w=10$.
  • Because prefixes are injected into every layer of the upper segment, the structural information is processed throughout the model, not only at the input; the SEG ablation supports that this deep contact helps.
  • The SMILES-guided resampler is credited with faster convergence during fine-tuning, not just better final scores; the training curves in Figure 6 (Right) show it reaching optimum earlier than a resampler with random queries.
  • The efficiency analysis says the full architecture trains in 5.53 hours on PubChem324k captioning, close to the 5.37 hours of Galactica alone, so the structural gains are presented as nearly context- and compute-neutral.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the complementarity claim generalizes, other spatial views the paper mentions—3D conformers, grids, or stereo descriptors—could be dropped into the same key/value stream without lengthening the prefix; a natural next experiment is to test whether three or four views continue to add value or saturate.
  • The SMILES guidance mechanism depends on a causal language model with segmentable layers; the paper notes decoder-only models other than Galactica could be used, which suggests the benefit might transfer to larger general-purpose LLMs if they are given enough chemical pretraining.
  • A cleaner test of whether the image view truly supplies spatial information would be to vary image rendering (atom colors, bond wedges, coordinate layout) while keeping SMILES and graph fixed; if scores change substantially, the image encoder may be exploiting rendering style rather than genuine molecular shape.
  • The per-task choice of partition $(b,u)$ and prefix length $w$ is empirical; if those choices are task- and dataset-dependent, a practical recipe for choosing them without a validation set would be needed before the architecture is used on new molecule families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CROP, a multimodal molecular LLM architecture that augments SMILES-based LLMs with molecular graph and image views. The core idea is to partition the LLM (Galactica 1.3B) into a lower segment that processes SMILES and produces SMILES-guided queries, and an upper segment that receives fixed-length cross-view prefixes. A SMILES Guided Resampler (SGR) jointly resamples graph and image embeddings using cross-attention, and a Structural Embedding Gate (SEG) converts the resulting embeddings into prefixes for all upper-segment layers. The authors evaluate CROP on molecule captioning (PubChem324k, CheBI-20), IUPAC name prediction, and molecular property prediction (six MoleculeNet datasets), with ablations on SGR, SEG, LLM partitions, and efficiency.

Significance. If the empirical claims hold, CROP is a useful architectural contribution. It provides a parameter-efficient method for injecting multiple structural views into LLMs, uses the LLM's own SMILES hidden states as guidance, and demonstrates that molecular images can improve generative molecular tasks. The efficiency comparison in Table 5 and the SGR/SEG ablations are valuable and give the reader insight into where the gains come from. However, the paper's central claim of 'significant performance gains' is currently not fully supported because the evaluation protocol appears to involve hyperparameter selection on test data, and the generative-task results lack uncertainty estimates. These issues need to be resolved before the reported superiority over baselines can be accepted.

major comments (3)
  1. [§4.1, Fig. 7 (Right), Appendix C] The paper does not establish that the LLM partition (b,u) and prefix length w were selected on a validation split. Section 4.1 states that the authors 'identify the optimal partition for each task,' and Figure 7 (Right) is explicitly labeled 'test results of four molecular tasks.' Appendix C (Figure 12) plots BLEU-2 versus w on PubChem324k without stating the split. The only validation-based selection described is for the model checkpoint. If these hyperparameters were chosen using test-set results, the reported improvements in Tables 1–3 (e.g., PubChem324k BLEU-2 44.9 vs MolCA 38.7; IUPAC BLEU-2 81.5 vs MolCA 75.0) are inflated relative to baselines, which did not receive equivalent test-set tuning. Please move hyperparameter selection to a validation split, or explicitly demonstrate that the results are robust to the choice of (b,u) and w.
  2. [Tables 1, 2, 4] The generative-task results are reported without variance or significance tests. The introduction claims 'significant performance gains,' but Table 1 and Table 2 give single point estimates, and Table 4 (ablations) also reports single numbers. Several reported gaps are small (e.g., PubChem324k CROP(S+G+I) vs CROP(S+G): 44.9 vs 43.4 BLEU-2; IUPAC CROP(S+G+I) vs CROP(S+G): 81.5 vs 80.8 BLEU-2). Without standard errors, paired tests, or at least multiple seeds, the reader cannot assess whether the differences are meaningful. Please report uncertainty for the captioning/IUPAC results as is done for property prediction in Table 3.
  3. [Tables 1–3] Most baselines are not re-run under the authors' protocol: Table 2 says 'Baseline results are from [32]' and Table 3 says 'Baseline results are from their original papers.' Because CROP's fine-tuning uses 100 epochs and validation-based checkpoint selection (and, per the first comment, possibly test-based hyperparameter selection), comparisons with cited numbers may reflect different training budgets and selection protocols rather than architectural gains. Please state explicitly which baselines were re-run and which were cited, and re-run at least the closest baselines (MolCA, GIT-Mol) under the same protocol, or justify why the cited numbers are directly comparable.
minor comments (5)
  1. [§3.3 vs Appendix D] Section 3.3 says Stage 2 unfreezes the graph/image encoders and uses LoRA for the LLM, but Appendix D says 'On molecule property prediction task, LoRA is not utilized.' Clarify which setting applies to each task, and why property prediction differs.
  2. [Table 3] Table 3 does not list TrainableParams, making it hard to compare parameter efficiency across property prediction models. Add this column or state that the same trainable-module setup is used.
  3. [Figure 2 / Table 5] The variants CROParch1, CROParch2, and CROParch3 are used in Figure 2 and Table 5 but not formally defined in the methodology. Define these variants explicitly, including what components are trained in each.
  4. [§4.2] The phrase 'large margin' for CheBI-20 CROP(S+G) vs MolCA is not supported by the 1.8 BLEU-2 difference; consider softening the wording.
  5. [References] References [6] and [7] are the same MolT5 paper; consolidate to avoid duplication.

Circularity Check

1 steps flagged · score 4.0 of 10

Reported test numbers are selected maxima over test-set partition choices; otherwise no derivation-level circularity.

  1. fitted input called prediction [Section 4.3 'Impact of Different LLM Partitions', Figure 7 (Right); headline results in Tables 1-3]
    "We compare five CROP variants with (b,u) set to (1, 23), (6, 18), (12, 12), (18, 6), and (23, 1), respectively. As illustrated in Figure 7, CROP b=12,u=12 performs best on the PubChem324k and CheBI-20 molecule captioning datasets. Additionally, CROP b=6,u=18 performs best on the PubChem324k IUPAC name prediction dataset. ... Figure 7: The experimental results of different partition settings for the LLM. (Left) The pre-training loss curves. (Right) The test results of four molecular tasks."

    The optimal partition (b,u) is identified from Figure 7, whose right panel is explicitly labeled 'test results'. The same test-set metrics are then reported as CROP's performance in Tables 1-3, so the reported numbers are selected maxima over five partitions rather than independent predictions. The paper only states that the model checkpoint is selected on the validation set ('The best-performing model on the validation set is selected for testing'), not that the partition hyperparameter was chosen on validation. Baselines are taken from prior papers and did not receive equivalent test-set tuning, so the headline 'significant performance gains' (e.g., BLEU-2 44.9 vs MolCA 38.7) conflate model quality with selection on evaluation data.

full rationale

The paper has no equation-level derivation that reduces to its own inputs: the SMILES-guided resampler (Eqs. 4-6) uses SMILES hidden states as queries and concatenated graph/image/SMILES embeddings as keys/values, then the Structural Embedding Gate (Eqs. 7-9) turns the resampled embeddings into prefixes. This is a self-conditioning design, not circular evidence. No load-bearing self-citations are present; the external bases (Galactica, moleculeSTM, ImageMol, MolCA) are independent, and the internal ablations (CROP vs CROPw/o sgr, CROP vs CROPw/o seg, and S/S+G/S+I/S+G+I variants) provide controlled comparisons. The one significant circularity concern is evaluation-based: the LLM partition (b,u) is explicitly chosen after inspecting 'test results' (Fig. 7 Right), and those same test numbers are then presented as CROP's reported performance. This makes the headline superiority over baselines partially an artifact of test-set selection. The prefix-length w selection (Appendix C, Fig. 12) is also not reported as validation-based, adding uncertainty but without the explicit 'test results' label. Because the central architectural claim retains independent support from controlled ablations, the circularity is partial, not total. Score 4.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central architecture relies on standard ML components (transformers, GIN, ResNet, LoRA); free parameters are task-specific prefixes and partition choices. No new physical or mathematical entities are introduced. The main domain assumptions concern the value of 2D images as spatial structure and the utility of Galactica's SMILES guidance.

free parameters (4)
  • Prefix length w = 10
    Chosen by validation experiments on PubChem324k molecule captioning (Appendix C, Figure 12); affects the amount of structural information presented to the LLM.
  • LLM partition (b,u) = (12,12) for captioning/property; (6,18) for IUPAC
    Selected per task based on performance; appears to be evaluated on test results (Section 4.1, Figure 7 Right), which can inflate results.
  • SGR layers = 4
    Number of transformer layers in the SMILES Guided Resampler; set without an ablation study.
  • LoRA rank/alpha = r=32, alpha=64
    LoRA hyperparameters for fine-tuning the LLM (Appendix D); not extensively searched.
assumptions (5)
  • domain assumption GIN and ResNet18 encoders provide useful molecule representations
    Graph encoder initialized from moleculeSTM, image encoder from ImageMol (Section 4.1). The paper assumes these frozen or fine-tuned encoders capture complementary structure.
  • domain assumption Galactica's SMILES hidden states carry chemical prior knowledge
    The SMILES guidance is derived from Galactica's lower segment (Section 3.2); the paper claims this enriches the resampling, an untested assumption about the internal representations.
  • domain assumption 2D RDKit images encode spatial configuration and overall shape
    Appendix B renders images with RDKit's MolToImage; the paper equates this 2D layout with the 'spatial view'. The independence of this information from the graph is not validated against 3D conformations.
  • ad hoc to paper The modified attention mask in the lower segment is a valid way to let prefixes perceive SMILES tokens
    Equation (3) defines a custom mask M' for prefix tokens; this is a design choice without theoretical justification, though it is consistent with causal attention.
  • domain assumption Benchmark datasets and scaffold splits are appropriate for comparing molecular LLMs
    Property prediction uses MoleculeNet with scaffold splits; captioning and IUPAC use PubChem324k and CheBI-20. The paper assumes these splits are comparable across baselines.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CROP: Integrating Topological and Spatial Structures via Cross-View Prefixes for Molecular LLMs." pith.science (2026). https://pith.science/paper/WW6FMLEI

@misc{pith2026250806917,
  author       = {Pith},
  title        = {Pith review of: CROP: Integrating Topological and Spatial Structures via Cross-View Prefixes for Molecular LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WW6FMLEI}},
  note         = {Machine review of arXiv:2508.06917}
}
read the original abstract

Recent advances in molecular science have been propelled significantly by large language models (LLMs). However, their effectiveness is limited when relying solely on molecular sequences, which fail to capture the complex structures of molecules. Beyond sequence representation, molecules exhibit two complementary structural views: the first focuses on the topological relationships between atoms, as exemplified by the graph view; and the second emphasizes the spatial configuration of molecules, as represented by the image view. The two types of views provide unique insights into molecular structures. To leverage these views collaboratively, we propose the CROss-view Prefixes (CROP) to enhance LLMs' molecular understanding through efficient multi-view integration. CROP possesses two advantages: (i) efficiency: by jointly resampling multiple structural views into fixed-length prefixes, it avoids excessive consumption of the LLM's limited context length and allows easy expansion to more views; (ii) effectiveness: by utilizing the LLM's self-encoded molecular sequences to guide the resampling process, it boosts the quality of the generated prefixes. Specifically, our framework features a carefully designed SMILES Guided Resampler for view resampling, and a Structural Embedding Gate for converting the resulting embeddings into LLM's prefixes. Extensive experiments demonstrate the superiority of CROP in tasks including molecule captioning, IUPAC name prediction and molecule property prediction.

Figures

Figures reproduced from arXiv: 2508.06917 by the authors.

Figure 1
Figure 1. An overview of the strengths and weaknesses of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of three MLLM architectures with dif [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The architecture of CROP. The SMILES Guided Resampler utilizes SMILES guidance [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The architecture of the Structural Embedding Gate [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The captions generated by CROP(𝑺+𝑮) and CROP(𝑺+𝑮+𝑰 ) on example molecules. CROP(𝑺+𝑮+𝑰 ) provides more accurate descriptions of substituent positions, types, and the connectivity of branched structures in molecules. task aims at predicting IUPAC name strings from other …
Figure 6
Figure 6. Figure 6: (Left) The distinct BLEU-2 scores in different [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The experimental results of different partition set [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The respective lengths of directly concatenated and [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: (Left) Directly concatenate molecular SMILES, graph [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 11
Figure 11. Figure 11: Pseudo-code for building the molecular image. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: The performance of CROP with different prefixes [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 14
Figure 14. Figure 14: Prompt for evaluating the performance of GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 38 canonical work pages

  1. [32]

    Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. 2023. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. arXiv preprint arXiv:2310.12798 (2023)

    arXiv 2023

  2. [1]

    A Patrícia Bento, Anne Hersey, Eloy Félix, Greg Landrum, Anna Gaulton, Francis Atkinson, Louisa J Bellis, Marleen De Veij, and Andrew R Leach. 2020. An open source chemical structure curation pipeline using RDKit. Journal of Cheminfor- matics 12 (2020), 1–16

    work page 2020

  3. [2]

    He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. 2023. Instructmol: Multi- modal integration for building a versatile and reliable molecular assistant in drug discovery. arXiv preprint arXiv:2311.16208 (2023)

    arXiv 2023

  4. [3]

    Alex D Casey, Steven F Son, Ilias Bilionis, and Brian C Barnes. 2020. Prediction of energetic material properties from electronic structure using 3D convolutional neural networks. Journal of Chemical Information and Modeling 60, 10 (2020), 4457–4473

    work page 2020

  5. [4]

    Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2020. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence , Vol. 34. 3438–3445

    work page 2020

  6. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    arXiv 2024

  7. [7]

    Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. 2022. Translation between molecules and natural language. arXiv preprint arXiv:2204.11817 (2022)

    arXiv 2022

  8. [8]

    Ray F Egerton et al . 2005. Physical principles of electron microscopy . Vol. 56. Springer

    work page 2005

Show all 62 references
  1. [9]

    Junfeng Fang, Shuai Zhang, Chang Wu, Zhiyuan Liu, Sihang Li, Kun Wang, Wenjie Du, Xiang Wang, and Xiangnan He. 2024. Moltc: Towards molecular relational modeling in language models. arXiv preprint arXiv:2402.03781 (2024)

    2024 arXiv

  2. [10]

    Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. 2022. Geometry-enhanced molec- ular representation learning for property prediction. Nature Machine Intelligence 4, 2 (2022), 127–134

    2022

  3. [11]

    Henri A Favre and Warren H Powell. 2013. Nomenclature of organic chemistry: IUPAC recommendations and preferred names 2013 . Royal Society of Chemistry

    2013

  4. [12]

    Zubatyuk, Maksim Kulichenko, Nicholas Lubbers, Justin S

    Nikita Fedik, Roman I. Zubatyuk, Maksim Kulichenko, Nicholas Lubbers, Justin S. Smith, Benjamin Tyler Nebgen, Richard A. Messerly, Ying Wai Li, Alexander I. Boldyrev, Kipton Barros, Olexandr Isayev, and Sergei Tretiak. 2022. Extending machine learning beyond interatomic potent...

    2022

  5. [13]

    Michael Fernandez, Fuqiang Ban, Godwin Woo, Michael Hsing, Takeshi Yamazaki, Eric LeBlanc, Paul S Rennie, William J Welch, and Artem Cherkasov. 2018. Toxic colors: the use of deep learning for predicting toxicity of compounds merely from their graphic images. Journal of chemic...

    2018

  6. [14]

    Garrett B Goh, Charles Siegel, Abhinav Vishnu, Nathan O Hodas, and Nathan Baker. 2017. Chemception: a deep neural network with minimal chemistry knowledge matches the performance of expert-developed QSAR/QSPR models. arXiv preprint arXiv:1706.06689 (2017)

    2017 arXiv

  7. [15]

    Thomas A Halgren. 1996. Merck molecular force field. I. Basis, form, scope, pa- rameterization, and performance of MMFF94. Journal of computational chemistry 17, 5-6 (1996), 490–519

    1996

  8. [16]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  9. [17]

    Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2024. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Pro- ceedings of the AAAI Conference on Artificia...

    2024

  10. [18]

    Nicolas Keriven. 2022. Not too little, not too much: a theoretical analysis of graph (over) smoothing. Advances in Neural Information Processing Systems 35 (2022), 2268–2281

    2022

  11. [19]

    Johannes Klicpera, Florian Becker, and Stephan Günnemann. 2021. Gemnet: Universal directional graph neural networks for molecules. arXiv e-prints (2021), arXiv–2106

    2021

  12. [20]

    Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. 2020. Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation. Machine Learning: Science and Technology 1, 4 (2020), 045024

    2020

  13. [21]

    Denis Kuzminykh, Daniil Polykovskiy, Artur Kadurin, Alexander Zhebrak, Ivan Baskov, Sergey Nikolenko, Rim Shayakhmetov, and Alex Zhavoronkov. 2018. 3D molecular representations based on the wave transform for convolutional neural networks. Molecular pharmaceutics 15, 10 (2018)...

    2018

  14. [22]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  15. [23]

    Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao-Yong Wei, Hui Liu, Jiliang Tang, and Qing Li. 2024. Empowering molecule discovery for molecule-caption transla- tion with large language models: A chatgpt perspective. IEEE Transactions on Knowledge and Data Engineering (2024)

    2024

  16. [24]

    Zhen Li, Mingjian Jiang, Shuang Wang, and Shugang Zhang. 2022. Deep learning methods for molecular representation and property prediction. Drug Discovery Today 27, 12 (2022), 103373

    2022

  17. [25]

    Ziyao Li, Shuwen Yang, Guojie Song, and Lingsheng Cai. 2021. Hamnet: Conformation-guided molecular representation with hamiltonian neural net- works. arXiv preprint arXiv:2105.03688 (2021)

    2021 arXiv

  18. [26]

    Youwei Liang, Ruiyi Zhang, Li Zhang, and Pengtao Xie. 2023. DrugChat: towards enabling ChatGPT-like capabilities on drug molecule graphs. arXiv preprint arXiv:2309.03907 (2023)

    2023 arXiv

  19. [27]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning. arXiv preprint arXiv:2304.08485 (2023)

    2023 arXiv

  20. [28]

    Pengfei Liu, Yiming Ren, Jun Tao, and Zhixiang Ren. 2024. Git-mol: A multi- modal large language model for molecular science with graph, image, and text. Computers in Biology and Medicine 171 (2024), 108073

    2024

  21. [29]

    Shuai Liu, Jie Li, Kochise C Bennett, Brad Ganoe, Tim Stauch, Martin Head- Gordon, Alexander Hexemer, Daniela Ushizima, and Teresa Head-Gordon. 2019. Multiresolution 3D-DenseNet for chemical shift prediction in NMR crystallogra- phy. The journal of physical chemistry letters 1...

    2019

  22. [30]

    Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Animashree Anandkumar. 2023. Multi-modal mol- ecule structure–text model for text-based retrieval and editing. Nature Machine Intelligence 5, 12 (2023), 1447–1457

    2023

  23. [31]

    Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. 2021. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728 (2021)

    2021 arXiv

  24. [33]

    Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie-Yan Liu. 2023. Molxpt: Wrapping molecules with text for generative pre-training. arXiv preprint arXiv:2305.10688 (2023)

    2023 arXiv

  25. [34]

    Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He

  26. [35]

    OpenAI. 2024. Hello GPT-4o

    2024

  27. [36]

    Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. 2023. Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. arXiv preprint arXiv:2310.07276 (2023)

    2023 arXiv

  28. [37]

    Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. 2020. Self-supervised graph transformer on large-scale molecular data. Advances in neural information processing systems 33 (2020), 12559–12571

    2020

  29. [38]

    Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE transactions on neural networks 20, 1 (2008), 61–80

    2008

  30. [39]

    Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. 2022. A molecular multimodal foundation model as- sociating molecule graphs with natural language. arXiv preprint arXiv:2209.05481 (2022)

    2022 arXiv

  31. [40]

    Jocelyn Sunseri and David R Koes. 2020. Libmolgrid: graphics processing unit accelerated molecular gridding for deep learning applications.Journal of chemical information and modeling 60, 3 (2020), 1079–1084

    2020

  32. [41]

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Sto- jnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 (2022)

    2022 arXiv

  33. [42]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  34. [43]

    Ngoc Tran, Daniel Kepple, Sergey Shuvaev, and Alexei Koulakov. 2019. Deep- Nose: Using artificial neural networks to represent the space of odorants. In International Conference on Machine Learning . PMLR, 6305–6314. MM ’25, October 27–31, 2025, Dublin, Ireland. Jianting Tang,...

    2019

  35. [44]

    Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2019. Deep graph infomax. ICLR (Poster) 2, 3 (2019), 4

    2019

  36. [45]

    David Weininger. 1988. SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of chemical information and computer sciences 28, 1 (1988), 31–36

    1988

  37. [46]

    Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Ge- niesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. 2018. MoleculeNet: a benchmark for molecular machine learning.Chemical science 9, 2 (2018), 513–530

    2018

  38. [47]

    Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. 2022. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations

    2022

  39. [48]

    Hongxin Xiang, Shuting Jin, Xiangrong Liu, Xiangxiang Zeng, and Li Zeng. 2023. Chemical structure-aware molecular image representation learning. Briefings in Bioinformatics 24, 6 (2023), bbad404

    2023

  40. [49]

    Liangxu Xie, Lei Xu, Shan Chang, Xiaojun Xu, and Li Meng. 2020. Multitask deep networks with grid featurization achieve improved scoring performance for protein–ligand binding. Chemical biology & drug design 96, 3 (2020), 973–983

    2020

  41. [50]

    Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018)

    2018 arXiv

  42. [51]

    Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi-Xiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu. 2024. Secap: Speech emotion captioning with large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19323–19331

    2024

  43. [52]

    Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. 2019. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling 59, 8 (2...

    2019

  44. [53]

    Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian P

    Kevin Yang, Kyle Swanson, Wengong Jin, Connor W. Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian P. Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, T. Jaakkola, Klavs F. Jensen, and Regina Barzilay

  45. [54]

    Jiacai Yi, Chengkun Wu, Xiaochen Zhang, Xinyi Xiao, Yanlong Qiu, Wentao Zhao, Tingjun Hou, and Dongsheng Cao. 2022. MICER: a pre-trained encoder– decoder architecture for molecular image captioning. Bioinformatics 38, 19 (2022), 4562–4572

    2022

  46. [55]

    Journal of Chemical Information and Modeling 59 (2019), 3370 – 3388

    Analyzing Learned Molecular Representations for Property Prediction. Journal of Chemical Information and Modeling 59 (2019), 3370 – 3388. https: //api.semanticscholar.org/CorpusID:198986021

    2019

  47. [56]

    Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021. Do transformers really perform badly for graph representation? Advances in neural information processing systems 34 (2021), 28877–28888

    2021

  48. [57]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 (2023)

    2023 arXiv

  49. [58]

    Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2022. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications 13, 1 (2022), 862

    2022

  50. [59]

    Xiangxiang Zeng, Hongxin Xiang, Linhui Yu, Jianmin Wang, Kenli Li, Ruth Nussi- nov, and Feixiong Cheng. 2022. Accurate prediction of molecular properties and drug targets using a self-supervised image representation learning framework. Nature Machine Intelligence 4, 11 (2022),...

    2022

  51. [60]

    Shifa Zhong, Jiajie Hu, Xiong Yu, and Huichun Zhang. 2021. Molecular image- convolutional neural network (CNN) assisted QSAR models for predicting con- taminant reactivity toward OH radicals: Transfer learning, data augmentation and model interpretation. Chemical Engineering J...

    2021

  52. [61]

    Jiaxin Zhang, Wentao Yang, Songxuan Lai, Zecheng Xie, and Lianwen Jin. 2024. Dockylin: A large multimodal model for visual document understanding with efficient visual slimming. arXiv preprint arXiv:2406.19101 (2024)

    2024 arXiv

  53. [63]

    "" Figure 13: Prompt for evaluating the performance of GPT-4o and Llama38B-Instruct on the molecule captioning task. prompt =

    Dan-Hao Zhu, Xin-Yu Dai, and Jia-Jun Chen. 2021. Pre-train and learn: Preserving global information for graph neural networks. Journal of Computer Science and Technology 36 (2021), 1420–1430. CROP: Integrating Topological and Spatial Structures via Cross-View Prefixes for Mole...

    2021

  54. [2019]

    In Proceedings of the AAAI conference on artificial intelligence , Vol

    Molecular property prediction: A multilevel quantum interactions modeling perspective. In Proceedings of the AAAI conference on artificial intelligence , Vol. 33. 1052–1060

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.