REVIEW 4 major objections 5 minor 27 references
Small language models predict molecular properties far more accurately when their zero-shot prompts are enriched with graph-derived context from a GNN expert, with relative accuracy gains of over 25% in most settings and up to 74% on Tox21.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 06:18 UTC pith:2PU3OKZD
load-bearing objection Useful framework, honest writing, but the headline Tox21 gains mostly come from the expert label, not the graph explanation, and the evaluation needs error bars. the 4 major comments →
Improving Molecular Property Prediction in Small Language Models Using Graph-based Tools
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that a small language model prompted only with a SMILES string is structurally blind to molecular topology, and that this blindness can be substantially mitigated by giving the model graph-derived context via tool use. The context consists of a GNN expert's prediction with confidence, an explanatory subgraph extracted by GNNExplainer and rendered as a SMILES fragment, and a natural-language rationale generated by the SLM itself. Across MUTAG and a 1000-molecule Tox21 subset, the ALL CONTEXT configuration produces the largest gains, raising DeepSeek's Tox21 accuracy from 38.50% to 67.00% (a 74% relative improvement) and yielding 75.44% on MUTAG. Still, the best to
What carries the argument
The central mechanism is a context-augmented prompting loop, an agentic tool-use architecture. Given a SMILES string, the SLM can call three tools: ExpertPredict, which returns the GNN expert's label and confidence; ExplainSubgraph, which runs GNNExplainer to produce an edge-importance mask, thresholds it at the top 10% of edges, and returns the resulting subgraph as a SMILES fragment; and ChemReason, which prompts the SLM to write a short paragraph about why that substructure might be toxic or mutagenic. The final prompt under ALL CONTEXT concatenates the original SMILES, the expert hint, the subgraph SMILES, and the rationale. This graph-to-text conversion is the load-bearing mechanism: it
Load-bearing premise
The reported accuracy gains rest on a single 70/30 train/test split per dataset, evaluated once per configuration with no repeated seeds or significance testing, so the observed improvements—especially DeepSeek's 74% relative gain on a roughly 300-molecule Tox21 test set—could be sampling noise.
What would settle it
Re-run the full evaluation across multiple random 70/30 splits (e.g., 10 repeats) and require the ALL CONTEXT configuration to beat SMILES-only on a paired comparison with non-overlapping confidence intervals or a paired significance test on the same test molecules. If the pooled gain is not significant, or the 38.50% to 67.00% Tox21 result does not replicate, the paper's central claim fails.
If this is right
- Zero-shot SLM property prediction improves substantially when graph-derived context is added to the prompt, even with no language-model fine-tuning.
- The GNN's explanatory subgraph, not just its label prediction, contributes actionable structural evidence; the necessity test shows that GNNExplainer's top edges concentrate decision-critical information for the expert.
- Tool-augmented prompting offers a modular, low-cost alternative to architectural integration when full graph encoders or joint training are unavailable.
- The gains are model- and dataset-dependent: code-tuned models like DeepSeek and Qwen benefit the most, while the smallest general-purpose model (Llama 3.2) shows inconsistent gains, suggesting the benefit depends on instruction-following ability.
- A measurable gap to the specialized GNN expert remains, indicating a boundary of purely text-conditioned reasoning for molecular structure.
Where Pith is reading between the lines
- If the accuracy gains replicate across repeated random splits, the same tool-use pattern could extend to regression targets (e.g., solubility, binding affinity) since the mechanism is target-agnostic; that would be a natural next experiment.
- A direct ablation not reported in the paper—comparing HINT-only vs ALL CONTEXT—would isolate how much of the improvement comes from the subgraph and rationale versus the expert prediction alone, and could reveal the cheapest sufficient context.
- The framework assumes the GNN expert's confidence is well-calibrated; separately calibrating the expert's probabilities could distinguish gains from hint content from gains from confidence information.
- A testable extension is to apply the same graph-to-text tool loop to other graph-structured scientific tasks (e.g., protein or materials property prediction) where linear string formats under-specify topology.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a context-augmented prompting framework in which a frozen small language model (SLM) can call a GNN expert to obtain a prediction/confidence, a GNNExplainer-derived subgraph, and an SLM-generated rationale before answering a binary molecular property question. The framework is evaluated on MUTAG and a 1,000-compound Tox21 subset with three SLMs (DeepSeek, Qwen, Llama) under five prompt configurations. The authors report relative accuracy improvements over SMILES-only prompting of up to 74% and conclude that graph-derived context mitigates structural blindness, while acknowledging a persistent gap to the specialized GNN expert.
Significance. If the reported gains are robust, the framework is a useful, low-cost way to inject graph knowledge into frozen SLMs without retraining, and the component ablation could guide agentic tool design. Strengths: the approach is modular, uses openly available components, includes five prompt-configuration ablations, and honestly reports that SLMs remain below GNN experts. The edge-drop intervention is a reasonable faithfulness check. However, the quantitative claims are not yet established due to the absence of error bars or repeated runs, the small test sets, and the confounding of the expert hint with the explanatory subgraph on Tox21. The causal interpretation in the conclusion also goes beyond what the experiments show.
major comments (4)
- [§4.1, Table 2] The central quantitative claims rest on a single 70/30 split with no repeated runs, no seeds, and no confidence intervals. The MUTAG test set has only ~56 molecules and the Tox21 test set ~300. A shift of a few molecules changes accuracy by several points; for instance, the DeepSeek MUTAG ALL CONTEXT gain of 19.30 points (56.14→75.44) is about 11 molecules, and the 74.03% relative Tox21 gain is a 28.50-point movement on ~300 molecules. Without standard errors, bootstrapped confidence intervals, or a McNemar test, the headline improvements cannot be distinguished from sampling noise. Please re-run with multiple seeds or provide bootstrap estimates.
- [§4.3, Observation 1 / Table 2] The Tox21 improvements are not attributable to the explanatory graph components: for DeepSeek, SMILES+HINT already gives 67.00 and ALL CONTEXT gives 67.00 (zero gain); for Qwen, ALL CONTEXT is 68.50 vs 67.50 for HINT (+1.0); for Llama, ALL CONTEXT is 63.50 vs 64.00 for HINT (−0.5). Thus on Tox21 the subgraph and reasoning components add essentially nothing. Moreover, the DeepSeek SMILES baseline of 38.50% is far below the always-negative majority baseline (63.0%, since the positive rate is 37%), indicating a degenerate baseline. The '74% improvement' is therefore mostly the model adopting the GNN expert's hint, not recovering graph structure. Please report majority-class accuracy and AUROC/balanced accuracy, and provide a component-wise incremental analysis.
- [§4.4, Fig. 2, Conclusion] The edge-removal experiment shows that edges ranked by GNNExplainer are more important for the trained GNN expert than random edges. This is a faithfulness check for the expert model, not a demonstration that the highlighted motifs are chemically necessary or causally relevant to the property. The conclusion's statement that the results offer 'a causal grounding perspective' overstates the evidence. Please rephrase as an explanation-faithfulness diagnostic.
- [§4.1, Table 1] The Tox21 setup is under-specified: the 1,000-molecule subset is described as 'random' without a seed or stratification, and collapsing the multi-label Tox21 benchmark to 'any positive' creates a heterogeneous binary label that may not reflect a meaningful biological endpoint. Results on this subset may not generalize to standard Tox21 benchmarks. Please provide the sampling procedure, a reproducible seed, and ideally per-task or AUROC results on the full benchmark.
minor comments (5)
- [General] Figures 1–3 are referenced in the text but not included in the submitted manuscript text; ensure the final PDF contains the actual figures.
- [§2.3] Reference [24] (ChemBERTa) is cited as evidence of an LLM 'SMILES-to-structure gap,' but ChemBERTa is a pretrained molecular language model, not a general-purpose LLM. Please cite a study that actually evaluates general-purpose LLMs on SMILES-to-graph reasoning.
- [§3.3, Appendix A] For SMILES + REASONING and ALL CONTEXT, the rationale is generated by the same SLM that later receives it. The paper should state whether the rationale was generated once and reused, and whether the model had already seen the full SMILES when generating it. This could introduce a self-confirmation confound.
- [§1] DeepSeek-Lite-Code has 16B total parameters (2.4B active). Calling it 'small' should be justified or qualified as 'small relative to frontier models.'
- [Throughout] Minor typos and stylistic issues: 'an GNNExplainer' (Section 3.2), 'use them as a hint' (Section 1), and inconsistent spacing around citations. A careful proofread is needed.
Circularity Check
No central circularity; the empirical comparison is self-contained, but the 'necessity' validation is tautological with GNNExplainer's objective.
specific steps
-
self definitional
[Section 4.4 (necessity test, Figure 2) and Section 3.2 (GNNExplainer subgraph selection)]
"To assess whether the extracted motif is merely correlational or behaves like a necessary substructure for the expert’s prediction, we perform a targeted perturbation test on the expert model: we progressively remove edges from 𝐺 either: (i) in decreasing importance order according to GNNExplainer (GNNExplainer), or (ii) at random (Random Score), and measure the expert accuracy as a function of the number of removed edges."
The explanatory subgraph is defined in Section 3.2 as the top-10% edges of GNNExplainer's importance mask m, and the test measures how much removing exactly those ranked edges degrades the same GNN's accuracy. GNNExplainer is designed to find edges whose removal maximally disrupts the prediction, so the observed steeper drop vs random removal is the explainer's own objective replayed as validation. It confirms explainer fidelity, not an independent causal property; the conclusion's 'supporting a causal grounding perspective' overstates the test. This is minor and does not affect the main empirical comparison.
full rationale
The paper's main claimed result is an empirical accuracy comparison across prompt configurations on held-out test splits. The GIN expert is trained on the 70% training split only; SLMs are zero-shot and never see test labels. The five configurations are proper ablations, and the paper includes a GNN expert row as a reference, so the 'ALL CONTEXT' gains are not a renamed fit. The concern that Tox21 gains are explained by the HINT (expert label) channel is an attribution/interpretation issue, not circularity: Table 2 itself shows SMILES+HINT equals or nearly equals ALL CONTEXT on Tox21, and the paper does not hide this. Statistical fragility (single split, no seeds) is a correctness/robustness risk, not a circularity. No load-bearing self-citations or imported uniqueness theorems appear. The only mild circularity is the necessity-based edge-drop test in Section 4.4, which validates GNNExplainer by the same objective GNNExplainer optimizes; this is a tautological sanity check, not a derivation of the central result. Therefore the central claim has independent content; score 2 reflects the minor circular validation only.
Axiom & Free-Parameter Ledger
free parameters (4)
- GNN expert hyperparameters =
hidden_dim=64, lr=2e-3, epochs up to 100, patience=10
- GNNExplainer edge threshold =
top 10% of edge importance (dynamic tau)
- Tox21 subset and binarization =
1,000 random molecules; any-positive label
- Train/test split ratio =
70/30
axioms (5)
- domain assumption SMILES sequence representations under-specify graph-topological cues, causing 'structural blindness' in LLMs.
- domain assumption The GNNExplainer top-10% subgraph is a functionally relevant (causal) substructure for the prediction.
- domain assumption The SLM's self-generated rationale (ChemReason) adds no external knowledge bias and is chemically sound enough to help inference.
- domain assumption The random 1,000-molecule Tox21 subset with 'any positive label' binarization is a representative binary toxicity task.
- domain assumption Zero-shot accuracy on a single test split is a stable estimator of SLM performance.
read the original abstract
Small language models (SLMs) have shown promise for zero-shot molecular property prediction from SMILES strings, yet they often suffer from structural blindness because sequence representations under-specify key graph-topological cues. We propose a modular Context-Augmented Prompting framework that enables agentic tool use at inference time: a trained GNN expert model provides a predictive hint with confidence, and a GNN extracts an instance-specific explanatory subgraph (e.g., a subgraph SMILES and an accompanying explanatory paragraph). We evaluate three commonly used SLMs on MUTAG and Tox21 under five prompting configurations ranging from SMILES-only to using all available tools at hand. Across two datasets, enriching prompts with graph-derived context yields substantial accuracy gains, often exceeding 25% relative improvement and up to 74% on Tox21. We further validate the functional relevance of the extracted motifs via a necessity-based edge-drop intervention. Despite the observed gains, a persistent gap remains to specialized GNN models, highlighting both the value and limits of text-conditioned reasoning for molecular structure.
Figures
Reference graph
Works this paper leans on
-
[1]
Gilmer, S
J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, G. E. Dahl, Neural message passing for quantum chemistry, in: D. Precup, Y. W. Teh (Eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 ofProceedings of Machine Learning Research, PMLR, 2017, pp. 1263–1272. URL: htt...
2017
-
[2]
E. Heid, K. P. Greenman, Y. Chung, S. Li, D. E. Graff, F. H. Vermeire, H. Wu, W. H. G. Jr., C. McGill, Chemprop: A machine learning package for chemical property prediction, J. Chem. Inf. Model. 64 (2024) 9–17. URL: https://doi.org/10.1021/acs.jcim.3c01250. doi:10.1021/ACS.JCIM.3C01250
-
[3]
H. Askr, E. Elgeldawi, H. A. Ella, Y. A. M. M. Elshaier, M. M. Gomaa, A. E. Hassanien, Deep learning in drug discovery: an integrative review and future challenges, Artif. Intell. Rev. 56 (2023) 5975–
2023
-
[4]
K. Yang, K. Swanson, W. Jin, C. W. Coley, P. Eiden, H. Gao, A. Guzman-Perez, T. Hopper, B. Kelley, M. Mathea, A. Palmer, V. Settels, T. S. Jaakkola, K. F. Jensen, R. Barzilay, Analyzing learned molecular representations for property prediction, J. Chem. Inf. Model. 59 (2019) 3370–3388. URL: https://doi.org/10.1021/acs.jcim.9b00237. doi:10.1021/ACS.JCIM.9B00237
-
[5]
Duvenaud, D
D. Duvenaud, D. Maclaurin, J. Aguilera-Iparraguirre, R. Gómez-Bombarelli, T. Hirzel, A. Aspuru- Guzik, R. P. Adams, Convolutional networks on graphs for learning molecular fingerprints, in: C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, R. Garnett (Eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Pro...
2015
-
[6]
S. Kearnes, K. McCloskey, M. Berndl, V. S. Pande, P. Riley, Molecular graph convolutions: Moving beyond fingerprints, CoRR abs/1603.00856 (2016). URL: http://arxiv.org/abs/1603.00856. arXiv:1603.00856
Pith/arXiv arXiv 2016
-
[7]
E. Tjoa, C. Guan, A survey on explainable artificial intelligence (xai): Toward medical xai, IEEE Transactions on Neural Networks and Learning Systems 32 (2021) 4793–4813. doi: 10.1109/ TNNLS.2020.3027314
arXiv 2021
-
[8]
Z. C. Lipton, The mythos of model interpretability, ACM Queue 16 (2018) 30. URL: https: //doi.org/10.1145/3236386.3241340. doi:10.1145/3236386.3241340
arXiv 2018
-
[9]
Weininger, Smiles, a chemical language and information system
D. Weininger, Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules, Journal of Chemical Information and Computer Sci- ences 28 (1988) 31–36. URL: https://doi.org/10.1021/ci00057a005. doi: 10.1021/ci00057a005. arXiv:https://doi.org/10.1021/ci00057a005
-
[10]
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, ...
2020
-
[11]
Kojima, S
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero- shot reasoners, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neu- ral Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem- ber 28 - December ...
2022
-
[12]
A. G. et al., The llama 3 herd of models, 2024. URL: https://arxiv.org/abs/2407.21783. arXiv:2407.21783
Pith/arXiv arXiv 2024
-
[13]
Qwen, :, A. Y. et al., Qwen2.5 technical report, 2025. URL: https://arxiv.org/abs/2412.15115. arXiv:2412.15115
Pith/arXiv arXiv 2025
-
[14]
DeepSeek-AI, Q. Z. et al., Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024. URL: https://arxiv.org/abs/2406.11931.arXiv:2406.11931
Pith/arXiv arXiv 2024
-
[15]
A. K. Debnath, R. L. Lopez de Compadre, G. Debnath, A. J. Shusterman, C. Hansch, Structure- activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correla- tion with molecular orbital energies and hydrophobicity, Journal of Medicinal Chem- istry 34 (1991) 786–797. URL: https://doi.org/10.1021/jm00106a046. doi:10.1021/jm00106a046. ...
-
[16]
A. M. Richard, R. Huang, S. Waidyanatha, P. Shinn, B. J. Collins, I. Thillainadarajah, C. M. Grulke, A. J. Williams, R. R. Lougee, R. S. Judson, K. A. Houck, M. Shobair, C. Yang, J. F. Rathman, A. Yasgar, S. C. Fitzpatrick, A. Simeonov, R. S. Thomas, K. M. Crofton, R. S. Paules, J. R. Bucher, C. P. Austin, R. J. Kavlock, R. R. Tice, The tox21 10k compound...
2020
-
[17]
D. Rogers, M. Hahn, Extended-connectivity fingerprints, Journal of Chemical Information and Modeling 50 (2010) 742–754. URL: https://doi.org/10.1021/ci100050t. doi:10.1021/ci100050t
-
[18]
T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net, 2017. URL: https://openreview.net/forum? id=SJU4ayYgl
2017
-
[19]
K. Xu, W. Hu, J. Leskovec, S. Jegelka, How powerful are graph neural networks?, in: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. URL: https://openreview.net/forum?id=ryGs6iA5Km
2019
-
[20]
Z. Ying, D. Bourgeois, J. You, M. Zitnik, J. Leskovec, Gnnexplainer: Generating explanations for graph neural networks, in: H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December...
2019
-
[21]
D. Luo, W. Cheng, D. Xu, W. Yu, B. Zong, H. Chen, X. Zhang, Parameterized explainer for graph neural network, 2020. URL: https://arxiv.org/abs/2011.04573.arXiv:2011.04573
Pith/arXiv arXiv 2020
-
[22]
R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, R. Stojnic, Galactica: A large language model for science, CoRR abs/2211.09085 (2022). URL: https: //doi.org/10.48550/arXiv.2211.09085. doi:10.48550/ARXIV.2211.09085. arXiv:2211.09085
-
[23]
C. Edwards, T. Lai, K. Ros, G. Honke, K. Cho, H. Ji, Translation between molecules and natural language, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022, pp. 375–413. URL: https://aclanth...
-
[24]
S. Chithrananda, G. Grand, B. Ramsundar, Chemberta: Large-scale self-supervised pretraining for molecular property prediction, 2020. URL: https://arxiv.org/abs/2010.09885. arXiv:2010.09885
Pith/arXiv arXiv 2020
-
[25]
Z. Liu, S. Li, Y. Luo, H. Fei, Y. Cao, K. Kawaguchi, X. Wang, T. Chua, Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Association for Comput...
-
[26]
X. Tang, A. Tran, J. Tan, M. B. Gerstein, Mollm: a unified language model for integrating biomedical text with 2d and 3d molecular representations, Bioinformatics 40 (2024) i357–i368. URL: https://doi.org/10.1093/bioinformatics/btae260. doi:10.1093/bioinformatics/btae260. A. Prompts Used in Configurations The following table contains the prompts used for ...
-
[6037]
doi:10.1007/S10462-022-10306-1
URL: https://doi.org/10.1007/s10462-022-10306-1. doi:10.1007/S10462-022-10306-1
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.