Skip to content
DraconiticResearch
Search Sources API
203,234 studies31 sourcesUpdated 42 min ago
Peer-reviewedCC BY-NC-NDJournal of chemical information and modeling

In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts.

  • M. Busch1
  • M. Tacke2
  • Sviatlana V. Lamaka2
  • Mikhail L. Zheludkevich2
  • Christian Johannes Cyron2,1
  • Christian Feiler3,2
  • Roland Can Aydin4,5,1
5 affiliations
  1. 1Technical University of Hamburg , , Eißendorfer Straße , ,
  2. 2Helmholtz-Zentrum Hereon , , Max-Planck-Straße , ,
  3. 3Hamburg University of Technology , , Am Irrgarten 3-9 , ,
  4. 4German Center for Artificial Intelligence (DFKI) , ,
  5. 5Saarland University , ,
Read full textPDFDOICite

Abstract

The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or instead rely on verbatim retrieval of memorized target values. Furthermore, we analyze the interplay between pretrained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet data sets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information, complemented by 0-, 60-, and 1000-shot in-context sample sizes as an additional control for information access. To validate the memorization analysis and the blinding experiments, we add a positive and a negative control for the memorization experiments and structural reference baselines for the multishot experiments, as well as bootstrap confidence intervals for all results. We find no evidence of verbatim retrieval on the legacy benchmarks and show that blinding exposes conflicts between pretrained knowledge and in-context information. This work provides a principled framework for evaluating molecular property prediction under controlled information access.

Related

  1. Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements
    Jingjie Ning, Xiaochuan Li, Ji Zeng +2
    arXiv (Cornell University)PreprintLicence not statedFull text8 citedMedicinal chemistry & drug discoveryOpenAlex
  2. Topology-Guided Multi-Level Graph Learning: A Unified Framework for Molecular Property Prediction
    Hasnain Ali, Momina Arshad, Malik Ahsin Iqbal +2
    Molecules (Basel, Switzerland)Peer-reviewedCC BYFull textGeneral chemistryEurope PMC
  3. Geometry-Aware Line Graph Transformer Pretraining for Molecular Property Prediction
    Peizhen Bai, Xianyuan Liu, Wenrui Fan +3
    IEEE Transactions on Neural Networks and Learning SystemsPeer-reviewedLicence not statedFull text8 citedMedicinal chemistry & drug discoveryOpenAlex
  4. Weighted Views Representation with Multi-Head Coulomb Attention for Molecular Property Prediction
    Muhammad Shahzeb Ali, Martin J. Mohlenkamp
    ACS OmegaPeer-reviewedCC BY-NC-ND 4.0Full text5 citedMedicinal chemistry & drug discoveryCrossrefOpenAlexPMC
  5. MolExplain: An Interactive Tool for Explainable Molecular Property Prediction
    Pirm Dhararag, Dylan Cashman
    arXivPreprintarXiv licenceFull textPhysical chemistryarXiv

Cite

BibTeXRIS
BibTeX
@article{busch2026incontext,
  title = {{In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts.}},
  author = {M. Busch and M. Tacke and Sviatlana V. Lamaka and Mikhail L. Zheludkevich and Christian Johannes Cyron and Christian Feiler and Roland Can Aydin},
  journal = {Journal of chemical information and modeling},
  year = {2026},
  month = sep,
  doi = {10.1021/acs.jcim.6c00961},
  url = {https://europepmc.org/article/MED/42912903},
  note = {CC BY-NC-ND},
}
RIS
TY  - JOUR
TI  - In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts.
AU  - M. Busch
AU  - M. Tacke
AU  - Sviatlana V. Lamaka
AU  - Mikhail L. Zheludkevich
AU  - Christian Johannes Cyron
AU  - Christian Feiler
AU  - Roland Can Aydin
PY  - 2026
DA  - 2026/09/17
JO  - Journal of chemical information and modeling
DO  - 10.1021/acs.jcim.6c00961
UR  - https://europepmc.org/article/MED/42912903
AB  - The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or instead rely on verbatim retrieval of memorized target values. Furthermore, we analyze the interplay between pretrained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet data sets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information, complemented by 0-, 60-, and 1000-shot in-context sample sizes as an additional control for information access. To validate the memorization analysis and the blinding experiments, we add a positive and a negative control for the memorization experiments and structural reference baselines for the multishot experiments, as well as bootstrap confidence intervals for all results. We find no evidence of verbatim retrieval on the legacy benchmarks and show that blinding exposes conflicts between pretrained knowledge and in-context information. This work provides a principled framework for evaluating molecular property prediction under controlled information access.
SN  - 1549-9596
SN  - 1549-960X
ER  -