The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning
Abstract
In scientific machine learning, Δ-learning trains models on residual errors relative to physical baselines, assuming that more accurate baselines with smaller residual scales inherently improve downstream performance. Here, we demonstrate that residual scale alone is an insufficient heuristic for learnability. Evaluating molecular graph neural networks on total energy targets, we show that complex local descriptor baselines can yield small residual targets that are disproportionately rough within architecture-informed proxy spaces and harder to learn relative to their scale. Conversely, semi-empirical baseline reduces both scale and normalized roughness, improving in-domain and out-of-domain prediction. We introduce scale-normalized graph Dirichlet roughness (DIQR) as a pre-training diagnostic for residual learnability and establish baseline complementarity as a core target-design principle, elevating target space formulation alongside model architecture as a key axis for scientific machine learning.
Related
- Machine-learning-guided discovery of selective CK2 inhibitors with reduced CDK off-target activity
- Repurposing drugs for multi-target treatment of cytokine storm using machine learning
- ProFormer: generalizable classification of single-cell and plasma proteomes using deep learning
- Folding and Function as Two Sides of One Coin: Statistical Mechanics, Information Theory, and Deep Learning of Binding, Catalysis, and Allostery in an Exactly Enumerable Lattice Protein
- Learning consistent molecular mechanics force fields from first principles
BibTeX
@misc{gameel2026mechanics,
title = {{The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning}},
author = {Kareem M. Gameel and Ihor Neporozhnii and Sjoerd Hoogland and Oleksandr Voznyy},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.28782},
eprint = {2609.28782},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.28782},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning
AU - Kareem M. Gameel
AU - Ihor Neporozhnii
AU - Sjoerd Hoogland
AU - Oleksandr Voznyy
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.28782
UR - https://arxiv.org/abs/2609.28782
AB - In scientific machine learning, Δ-learning trains models on residual errors relative to physical baselines, assuming that more accurate baselines with smaller residual scales inherently improve downstream performance. Here, we demonstrate that residual scale alone is an insufficient heuristic for learnability. Evaluating molecular graph neural networks on total energy targets, we show that complex local descriptor baselines can yield small residual targets that are disproportionately rough within architecture-informed proxy spaces and harder to learn relative to their scale. Conversely, semi-empirical baseline reduces both scale and normalized roughness, improving in-domain and out-of-domain prediction. We introduce scale-normalized graph Dirichlet roughness (D_IQR) as a pre-training diagnostic for residual learnability and establish baseline complementarity as a core target-design principle, elevating target space formulation alongside model architecture as a key axis for scientific machine learning.
ER -