Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models
Abstract
Although ๐ผ[dropout(x)] = x, here we show that ๐ผ[LayerNorm(dropout(x))] is not equal to LayerNorm(x). Accordingly, the pattern of a Dropout layer followed by a LayerNorm, which is common to many AlphaFold2-based protein structure predictors, produces a systematic bias at evaluation time that can hamper performance. To address this, we derive a closed-form, first-order correction for this gap, which we call a Dropout-LayerNorm Correction (DLC). DLC empirically matches the performance boost of large Monte Carlo dropout ensembles. We evaluate its effect across nine protein structure models (ESMFold, OpenFold, ABB3, FlashABB, Ibex, NbForge, Genie1, Genie2, Genie3) on both paired and single-chain antibody structures as well as one protein-ligand docking model (QuickBind). The correction is computationally negligible and improves accuracy in all ten models tested (โผ0.3%โ13%), with a modest but consistent improvement in ESMFold and OpenFold and a substantial improvement in antibody-specific models. This work identifies the mathematical consequence of chaining together Dropout and LayerNorm and provides a free, principled adjustment to improve the evaluation performance of many pretrained models. Code to reproduce the experiments is available at https://github.com/oxpig/DLC.
Related
- Laser phase plate improves structure determination of small proteins by cryo-EM
- Deep learning-based genetic disease prediction with Monte Carlo (MC) dropout uncertainty estimation: a comparative study with classical machine learning baselines
- Comparison of Prime Editing and CRISPR-Cas Mediated HDR in Correcting CFTR-F508del Mutation in Patient iPSC-Derived Airway Basal Cells
- ParaEM: antigen-aware sequence-based paratope prediction with expectation-maximization.
- A method for correcting background non-uniformity while preserving specimen-level information in Fourier ptychographic microscopy images
BibTeX
@misc{ellmen2026correcting,
title = {{Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models}},
author = {Isaac Ellmen and David Errington and Matthew I. J. Raybould and Charlotte M. Deane},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.32062},
eprint = {2609.32062},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.32062},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models
AU - Isaac Ellmen
AU - David Errington
AU - Matthew I. J. Raybould
AU - Charlotte M. Deane
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.32062
UR - https://arxiv.org/abs/2609.32062
AB - Although ๐ผ[dropout(x)] = x, here we show that ๐ผ[LayerNorm(dropout(x))] is not equal to LayerNorm(x). Accordingly, the pattern of a Dropout layer followed by a LayerNorm, which is common to many AlphaFold2-based protein structure predictors, produces a systematic bias at evaluation time that can hamper performance. To address this, we derive a closed-form, first-order correction for this gap, which we call a Dropout-LayerNorm Correction (DLC). DLC empirically matches the performance boost of large Monte Carlo dropout ensembles. We evaluate its effect across nine protein structure models (ESMFold, OpenFold, ABB3, FlashABB, Ibex, NbForge, Genie1, Genie2, Genie3) on both paired and single-chain antibody structures as well as one protein-ligand docking model (QuickBind). The correction is computationally negligible and improves accuracy in all ten models tested (โผ0.3%โ13%), with a modest but consistent improvement in ESMFold and OpenFold and a substantial improvement in antibody-specific models. This work identifies the mathematical consequence of chaining together Dropout and LayerNorm and provides a free, principled adjustment to improve the evaluation performance of many pretrained models. Code to reproduce the experiments is available at https://github.com/oxpig/DLC.
ER -