Transformer-inspired architectures make machine-learned electron densities accurate and transferable enough to bypass self-consistency
Abstract
We introduce machine learning models that accurately predict the electron density, the central variable of density functional theory. They are built on Transformer-inspired architectures, which require little computation per parameter and can therefore be enlarged at low cost. The proposed models, SwinUNETR-V2 and its attention-free counterpart NeXtUNETR, achieve state-of-the-art accuracy on QM9, with normalized mean absolute errors in the density averaging 0.075% and 0.046%, respectively. The predicted densities are accurate enough to bypass the self-consistent iteration: a single Kohn-Sham step reproduces the self-consistent electronic structure. For NeXtUNETR, the median errors are 0.036 meV in the total energy, 0.65 meV/angstrom in the forces and 1.8 meV in the HOMO-LUMO gap. The forces not only recover the self-consistent minima in geometry optimization but also successfully drive ab initio molecular dynamics. Fine-tuned on small datasets, both networks also transfer accurately to systems containing elements absent from QM9 or larger than its molecules.
Related
- Orbital-Free Surrogate Functionals Yield Transferable Interatomic Potentials and Electron Densities
- Bio-inspired decoration of Ag nanoparticles on eggshell-derived hierarchical CaO-based flower-like architectures for enhanced antibacterial, photocatalytic, and hemocompatible applications
- Correcting the Liquid Densities of Universal Machine-Learning Interatomic Potentials Using Local-Number-Density Repulsion
- Hybridizing expert descriptors and learned embeddings yields transferable Mat2Vec-* element representations
- Response-state Learning for Transferable Vibrational Spectroscopic Characterization with Electron Prior
BibTeX
@misc{woo2026transformerinspired,
title = {{Transformer-inspired architectures make machine-learned electron densities accurate and transferable enough to bypass self-consistency}},
author = {Jeheon Woo},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.39687},
eprint = {2610.39687},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.39687},
}
RIS
TY - UNPB
TI - Transformer-inspired architectures make machine-learned electron densities accurate and transferable enough to bypass self-consistency
AU - Jeheon Woo
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.39687
UR - https://arxiv.org/abs/2610.39687
AB - We introduce machine learning models that accurately predict the electron density, the central variable of density functional theory. They are built on Transformer-inspired architectures, which require little computation per parameter and can therefore be enlarged at low cost. The proposed models, SwinUNETR-V2 and its attention-free counterpart NeXtUNETR, achieve state-of-the-art accuracy on QM9, with normalized mean absolute errors in the density averaging 0.075% and 0.046%, respectively. The predicted densities are accurate enough to bypass the self-consistent iteration: a single Kohn-Sham step reproduces the self-consistent electronic structure. For NeXtUNETR, the median errors are 0.036 meV in the total energy, 0.65 meV/angstrom in the forces and 1.8 meV in the HOMO-LUMO gap. The forces not only recover the self-consistent minima in geometry optimization but also successfully drive ab initio molecular dynamics. Fine-tuned on small datasets, both networks also transfer accurately to systems containing elements absent from QM9 or larger than its molecules.
ER -