In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts

Christian Feiler; Christian J. Cyron; Marius Tacke; Matthias Busch; Mikhail L. Zheludkevich; Roland C. Aydin; Sviatlana V. Lamaka

REVIEW 4 major objections 5 minor 1 cited by

In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts

T0 review · 4 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs do not memorize MoleculeNet property values; they combine structure–property priors with in-context learning, and those priors often interfere with the examples.

desk verdict Solid empirical protocol showing LLMs do not verbatim-memorize MoleculeNet labels and that priors can hurt ICL; capability attributions from the six-level blinding are only as clean as the transforms allow. read the letter →

arxiv 2603.25857 v3 pith:JY7R6VOH submitted 2026-03-26 cs.LG

Matthias Busch , Marius Tacke , Sviatlana V. Lamaka , Mikhail L. Zheludkevich , Christian J. Cyron , Christian Feiler , Roland C. Aydin This is my paper

classification cs.LG

keywords molecularpropertypredictionin-contextlearninglargelanguagemodelsmemorizationdatacontaminationblindingstudyMoleculeNetknowledgeconflict

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models genuinely regress molecular properties from examples placed in the prompt, or whether strong scores on public chemistry benchmarks mostly reflect memorized labels and domain shortcuts. The authors strip information in six progressive blinding steps—removing property names, then all chemistry language, then transforming both the numeric targets and the SMILES strings—while also varying the number of in-context examples from zero to one thousand. Across nine models from three families and three standard datasets they find almost no exact value recall; instead models use learned structure–property knowledge when it is available and can still extract patterns from pure string examples when it is not. Prior knowledge frequently collides with the supplied samples, so that sixty examples often hurt relative to zero-shot and full chemical context is often not the best setting. The practical upshot is a concrete protocol for separating memorization, domain priors, and true in-context regression before trusting LLM scores on novel molecules or properties.

What carries the argument

A six-level blinding framework that successively removes property names, chemistry terminology, and applies a fixed label transform (negation and rescaling to [0, 100]) plus deterministic SMILES character substitution, paired with a four-capability taxonomy (direct memorization, learned structure–property relations, chemical in-context learning, general in-context learning) used to attribute performance changes.

What would settle it

A large fraction of exact three-significant-digit matches on untransformed zero-shot MoleculeNet labels, or a collapse of Level-6 (fully blinded) correlation to chance while unblinded accuracy remains high only because of those exact matches, would overturn the claim that models are not directly retrieving memorized values.

Watch

Extended reading notes

Core claim

Large language models show no substantial direct memorization of widely used molecular property labels. They instead flexibly combine pre-trained structure–property relationships with in-context learning. Prior knowledge is double-edged: it can raise a ceiling but often conflicts with the examples, so peak correlation frequently occurs at intermediate or fully blinded levels rather than with full chemical context, and sixty-shot prompts often underperform zero-shot until roughly a thousand examples override the priors.

Load-bearing premise

The chosen label transform and SMILES character swap fully block value and structure lookup while still leaving the same learnable structure–property map intact so that inverse-transformed correlations can be compared fairly across blinding levels.

Editorial extensions

If this is right

Unblinded MoleculeNet scores mix possible contamination, domain priors, and in-context learning and therefore cannot isolate genuine in-context regression.
Performance that survives full blinding indicates robust pattern extraction; sharp drops under blinding flag shallow priors or dataset familiarity.
When few examples are available and priors may conflict, deliberately suppressing the property name can improve accuracy.
Roughly a thousand in-context examples generally resolve the knowledge conflicts that degrade sixty-shot performance.
Zero-shot correlation itself is a useful diagnostic of how heavily a property appears in a model’s training data before any learning claims are made.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same progressive-blinding protocol could be reused on other scarce scientific ICL tasks (materials properties, reaction yields) where public benchmarks are few and contamination risk is high.
Because some models remain sensitive to the original numeric scale even after the fixed transform, multi-family transforms would strengthen the capability attributions.
Model families differ systematically in how they integrate priors with examples; an automatic prior-conflict pilot (blinded vs unblinded on a small hold-out) could become standard before deploying an LLM property predictor.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful takeaway is practical: on Delaney, Lipophilicity, and a SMILES-only QM7 variant, nine current models show almost no high-precision exact matches in 0-shot (App. A), continuous error CDFs rather than zero-error spikes, and frequent performance gains when you strip property names or transform labels/SMILES. Peak correlation is often not at full chemical context, and 60-shot frequently underperforms 0-shot until 1000 examples arrive. That is a real, non-obvious knowledge-conflict pattern, not just another leaderboard.

What is new is the progressive six-level blinding (specific → generic chemistry → domain-agnostic + SMILES alphabet rewrite) stacked with 0/60/1000-shot and a four-capability taxonomy. It extends Joe et al.’s value transforms and the authors’ own pre-analysis prompting into a contamination-aware evaluation protocol that other groups can actually reuse. Code, prompts, and pre-computed results are released; the design is systematic across three families and three size classes.

Soft spots are real but bounded. Only two runs, no uncertainty bands, proprietary APIs, and a nonstandard QM7. More importantly, the stress-test concern lands partially: the paper itself shows QM7 original scales (≈−500 to −2500) produce large oscillatory drops that the authors attribute to numerical reasoning, not capability change. So level-to-level drops cannot be read as pure deactivation of “direct memorization” vs “chemical ICL”; residual topology after character substitution and rank-order cues after rescaling remain possible. Exact-match tables kill only verbatim 3-digit recall. Causal language about “wrong knowledge” is therefore interpretive scaffolding, not airtight isolation. Still, the directional patterns (no mass memorization; priors sometimes interfere; Delaney survives full blinding better than Lipophilicity) are supported by the figures and tables.

This is for people who evaluate or deploy LLMs on molecular regression and who care about contamination and prior/ICL conflict. It is not a new SOTA predictor. I would bring it to reading group, cite the protocol and the no-memorization/knowledge-conflict observations, and send it to peer review. Ask for error bars, tighter causal wording, and an explicit discussion of transform-induced difficulty. The central empirical claims hold up well enough to deserve referee time.

Referee Report

4 major / 5 minor

Summary. The paper studies whether LLMs perform genuine in-context regression on molecular properties or mainly exploit memorized values and domain priors. It introduces a six-level blinding framework (specific/generic/agnostic property framing, with and without label inversion+rescaling and SMILES character substitution) and evaluates nine models from three families (GPT-4.1, GPT-5, Gemini 2.5) on ESOL/Delaney, Lipophilicity, and QM7 under 0-/60-/1000-shot regimes. Using inverse-transformed Pearson correlation and exact-match checks, the authors report near-absent direct label memorization, dataset-dependent 0-shot priors (strongest for Delaney), frequent 60-shot dips relative to 0-shot, and cases where peak performance occurs at intermediate or high blinding levels, which they interpret as prior–ICL knowledge conflicts. A four-capability taxonomy (direct memorization, learned structure–property relations, chemical ICL, general ICL) is used to interpret level-to-level changes.

Significance. Contamination and memorization concerns for MoleculeNet-style benchmarks are timely, and a controlled information-access protocol for molecular regression is a useful contribution. The multi-family, multi-size design, exact-match memorization tables (Appendix A), continuous error CDFs, and public code/data strengthen the empirical case against verbatim label retrieval. If the interference findings hold under cleaner isolation of capabilities, the work would give practitioners concrete guidance on when to suppress property names or domain framing and would improve how the community benchmarks LLM ICL for scientific regression. The systematic blinding idea itself is reusable beyond chemistry.

major comments (4)

[§§3.2–3.3, Figs. 4–5, §4.3] §§3.2–3.3 and Figs. 4–5: The central attribution of correlation changes to the four-capability taxonomy assumes that ×(−1)+rescale-to-[0,100] and deterministic SMILES substitution fully disable value/structure lookup while leaving an equally learnable map for inverse-transformed Pearson r. The paper’s own QM7 results contradict clean isolation: original large negative scales produce large oscillatory drops (especially GPT-5) that the authors attribute to numerical-reasoning difficulty rather than capability change. Without controls that hold numerical difficulty fixed (e.g., alternative monotone transforms, rank-only targets, or non-inverted rescales) or that verify residual SMILES recognizability, level-to-level drops (Lipophilicity collapse, peaks away from Level 1, 60-shot interference) cannot be cleanly read as memorization vs. knowledge conflict vs. transform artifacts.
[§3.5, §4, §5.5] §3.5 and §5.5: Statistical power is thin for the load-bearing claims. Only two runs per configuration, 150 test molecules for few/many-shot, and no reported confidence intervals or paired significance tests on correlation differences (0 vs 60 vs 1000; Level 1 vs Level 6). Several key patterns (60-shot dips, “peak not at Level 1,” model-family differences) are described qualitatively; without uncertainty quantification they remain suggestive rather than conclusive.
[§4.1, §5.4] §4.1 and Discussion: The paper correctly notes that even the best models do not surpass Chemprop/MPNN baselines, but the main figures report only LLM correlations. For the practical claim that blinding can improve deployment, a side-by-side comparison of blinded LLM ICL against a simple non-LLM baseline (e.g., ridge or kNN on the same 60/1000 SMILES fingerprints) under identical train–test splits is needed; otherwise it is unclear whether “general ICL” is competitive once domain priors are removed.
[Appendix A, §4.2] Appendix A and §4.2: Exact three-significant-digit matches rule out verbatim high-precision recall, which is valuable. They do not, however, rule out approximate retrieval, scale-sensitive priors, or rank-order memorization. Given that the label transform preserves rank order, a stronger memorization control (e.g., scrambled labels, random SMILES–label pairings, or nearest-neighbor retrieval baselines from the training corpus) would better support the claim of “no substantial direct memorization.”

minor comments (5)

[Abstract, Fig. 1] Figure 1 caption and abstract claim “rather than direct memorization” while the body correctly nuances this; align abstract wording with the more careful §5.2 statement.
[Table 1, Figs. 4–5] Table 1 and §3.2: “Label” / “Input” columns are clear, but the prose sometimes uses “Specific-Transformed” and “Level 2” interchangeably; a single consistent naming convention in figure legends would help.
[§3.7] §3.7: Temperature 0.7 / top_p 0.95 is stated, yet GPT-5 family parameters cannot be set; report the effective decoding settings used for those models.
[Throughout] Several OCR-style artifacts appear in the provided text (garbled words, missing symbols such as −1). Ensure the camera-ready PDF is clean.
[§2] Related Work could more explicitly position against other contamination-mitigation protocols (e.g., dynamic benchmarks, canary insertion) beyond Joe et al. 2026.

Circularity Check

1 steps flagged · score 1.0 of 10

Empirical blinding study with no derivation circularity; only minor non-load-bearing self-citation for prompting method.

self citation load bearing [§3.4 Prompting Strategy; also Related Work §2.1]
"We employ the pre-analysis prompting approach of Busch et al. 2025, which separates prediction into two phases: (1) Analysis: systematically examine training data... (2) Prediction: find similar training molecules..."

Authors cite their own prior work for the structured analysis+prediction prompt used throughout. This is ordinary method reuse and is not load-bearing for the memorization/ICL claims (those rest on the new blinding results and exact-match statistics). Flagged only as minor self-citation; does not force any reported correlation or capability attribution.

full rationale

The paper is an experimental measurement study of LLM correlations under controlled information access (0/60/1000-shot and six blinding levels). Its claims are observational (no substantial exact memorization via App. A exact-match rates; performance often peaks away from Level 1; 60-shot dips; dataset-dependent drops under blinding) rather than any first-principles derivation or fitted-parameter prediction. The four-capability taxonomy in §3.2 is explicit interpretive scaffolding that maps which blinding levels are intended to deactivate which mechanisms; it does not redefine the measured Pearson r or force the results by construction. The §3.3 label transform (×−1 then rescale to [0,100]) and SMILES character substitution are experimental controls whose validity can be debated on empirical grounds (e.g., QM7 scale sensitivity noted by the authors themselves), but they are not circular reductions of a claimed prediction to its inputs. The sole self-citation of note is to Busch et al. 2025 for the pre-analysis prompting template (§3.4 and App. B); that citation supplies a reusable method, not a uniqueness theorem or load-bearing premise for the blinding conclusions. No self-definitional equations, fitted-input-as-prediction, uniqueness import, ansatz smuggling, or renaming of known results appear. Score 1 reflects only the minor, non-central self-citation; the central empirical chain is self-contained against external benchmarks.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

Central claims rest on experimental design choices and domain assumptions about what each blinding level disables, not on free physical constants. The load-bearing pieces are the transform definitions, the interpretation that 0-shot r proxies corpus prevalence, and the capability taxonomy used to read level-to-level drops.

free parameters (4)

label transform (×−1 then rescale to [0,100])
Hand-chosen single transform applied at levels 2/4/6; metrics reported after inverse transform. Choice of range and inversion is not derived from theory.
SMILES character-substitution map
Deterministic but arbitrary atom/syntax remapping for levels 5–6; assumed to erase recognition while preserving topology.
in-context sizes {0, 60, 1000} and test sizes {1000, 150}
Chosen for cost/robustness tradeoff; 60 retained in analysis prompt for 1000-shot design is a design hyperparameter.
sampling temperature=0.7, top_p=0.95 (where allowed); 2 runs
Inference hyperparameters and replication count that affect reported correlations.

assumptions (5)

domain assumption Pearson correlation on inverse-transformed predictions fairly compares models across original and transformed label scales.
Stated in §3.6; MAE/RMSE discarded because of scale shift.
ad hoc to paper Progressive removal of property names, chemistry terms, and SMILES identity isolates memorization vs domain priors vs chemical ICL vs general ICL as in the four-capability taxonomy.
§3.2 taxonomy; capabilities are not mutually exclusive and do not combine linearly by the authors’ own admission.
domain assumption 0-shot correlation is a proxy for prevalence of the dataset/property in pretraining corpora.
§4.1 interpretation of Delaney ≫ Lipophilicity ≫ QM7 0-shot patterns.
domain assumption SMILES alone is a sufficient molecular representation for testing LLM property regression.
§3.1 and Limitations; no 3D/graph inputs.
ad hoc to paper Exact match on first three significant digits is an appropriate operational test for direct label memorization.
Appendix A definition; low-precision matches excluded.

invented entities (2)

Six-level blinding framework (Specific / Specific-Transformed / Generic / Generic-Transformed / Agnostic / Agnostic-Transformed)
purpose: Progressively strip chemical context and disrupt memorized values/structures to attribute performance to distinct LLM capabilities.
Core methodological contribution of the paper; defined in Table 1 and §3.2.
Four-level capability taxonomy (direct memorization; learned structure–property relations; chemical ICL; general ICL)
purpose: Interpret which mechanisms remain active at each blinding level.
Interpretive scaffold in §3.2; not independently measured outside this design.

how reviews work

0 comments

Cite this review

Pith. "Pith review of In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts." pith.science (2026). https://pith.science/paper/JY7R6VOH

@misc{pith2026260325857,
  author       = {Pith},
  title        = {Pith review of: In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JY7R6VOH}},
  note         = {Machine review of arXiv:2603.25857}
}

read the original abstract

The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or instead rely on verbatim retrieval of memorized target values. Furthermore, we analyze the interplay between pre-trained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet datasets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information, complemented by 0-, 60-, and 1000-shot in-context sample sizes as an additional control for information access. To validate the memorization analysis and the blinding experiments, we add a positive and a negative control for the memorization experiments and structural reference baselines for the multi-shot experiments as well as bootstrap confidence intervals for all results. We find no evidence of verbatim retrieval on the legacy benchmarks and show that blinding exposes conflicts between pre-trained knowledge and in-context information. This work provides a principled framework for evaluating molecular property prediction under controlled information access.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Comparing the Performance of Leading VQE Algorithms for Computing Ground-State Energies of Amino Acids
quant-ph 2026-07 conditional novelty 5.5 of 10

An open modular pipeline benchmarks 10+ VQE ansatzes and two truncations on QMProt amino acids across noise resilience, trainability, cost evaluations, and energy error.

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.