Does Pretraining Teach Chemical Language Models Chemistry?
Abstract
Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.
Related
- Literature-Scale Mössbauer Data Enable Prediction, Electronic-State Interpretation, and Chemical Insight in Molecular Iron Chemistry
- Large Language Models in Chemistry: Opportunities, Limitations, and Hallucination Risks
- Chemistry, Generative AI, and English: implications for chemistry researchers from diverse language backgrounds
- ZnO Nanowires by Chemical Bath Deposition: A Review on Nucleation, Growth, and Green Chemistry
- Chemistry of adaptive polymer-based protective coatings for harsh environments: chemical aging, anticorrosion, antifouling, and self-healing design
BibTeX
@misc{sieben2026does,
title = {{Does Pretraining Teach Chemical Language Models Chemistry?}},
author = {Leif Sieben and Yoel Zimmermann},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.18651},
eprint = {2610.18651},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.18651},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Does Pretraining Teach Chemical Language Models Chemistry?
AU - Leif Sieben
AU - Yoel Zimmermann
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.18651
UR - https://arxiv.org/abs/2610.18651
AB - Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.
ER -