Skip to content
DraconiticResearch
Search Sources API
181,716 studies31 sourcesUpdated 52 min ago
Peer-reviewedCC BYNature Biotechnology

Deep learning perturbation models can outperform baselines on calibrated metrics

  • Henry E. Miller1
  • Gabriel Mejia2
  • Francis Leblanc2
  • Brendan Swain2
  • Bo Wang3,4,5,6
  • Lucas Paulo de Lima Camillo7,2
7 affiliations
  1. 1Shift Bioscience Ltd., Cambridge, UK. [email protected]
  2. 2Shift Bioscience Ltd., Cambridge, UK
  3. 3Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
  4. 4Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Ontario, Canada
  5. 5Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada
  6. 6Vector Institute, Toronto, Ontario, Canada
  7. 7School of Clinical Medicine, University of Cambridge, Cambridge, UK
Open at sourcePDFDOICite

Abstract

Recent benchmarks report that deep-learning-based genetic perturbation models fail to outperform uninformative baselines. Here we introduce a positive control baseline and metric calibration framework, showing—across 14 datasets and 18 metrics—that common benchmarking metrics (for example, mean squared error and Pearson Δ ) are frequently miscalibrated, with reduced sensitivity to measure positive model performance. Under well-calibrated metrics, we find that deep-learning-based genetic perturbation models can outperform uninformative baselines.

Related

  1. Deep learning-based genetic disease prediction with Monte Carlo (MC) dropout uncertainty estimation: a comparative study with classical machine learning baselines
    V S Keerthana, Tarak Hussain
    Frontiers in geneticsPeer-reviewedCC BYFull textGeneral biochemistryEurope PMCPMCEurope PMC
  2. Bayesian Learning of Distance Metrics Beyond RMSD for Biomolecule Alignment, Clustering, and Domain Identification.
    Saumyak Mukherjee, Gerhard Hummer
    Journal of chemical theory and computationPeer-reviewedCC BYFull textMolecular biology & gene regulationEurope PMC
  3. BioGraphBench-PPI enables audit-first evaluation of negative-sampling contracts and structural baselines in protein interaction networks
    Alejandro Cabrera-Andrade, Jorge Robalino-Díaz, Iván Dueñas-Espín +2
    Frontiers in bioinformaticsPeer-reviewedCC BYFull textMolecular biology & gene regulationEurope PMC
  4. Calibrated simulations for dynamic focusing of ultrasound through the temporal window
    Ehsan Dadgar-Kiani, Vibha Hebbale, Malachy W. Newman +13
    Cell Reports MethodsPeer-reviewedCC BY 4.0Full textMolecular biology & gene regulationPMC
  5. Quantitative lipidomics reveals Lipid perturbation of plasma apoptotic extracellular vesicles during physiological aging
    Maojiao Li, Haisen Huang, Qi Tang
    Frontiers in cell and developmental biologyPeer-reviewedCC BYFull textGeneral biochemistryEurope PMC

Cite

BibTeXRIS
BibTeX
@article{miller2026deep,
  title = {{Deep learning perturbation models can outperform baselines on calibrated metrics}},
  author = {Henry E. Miller and Gabriel Mejia and Francis Leblanc and Brendan Swain and Bo Wang and Lucas Paulo de Lima Camillo},
  journal = {Nature Biotechnology},
  year = {2026},
  month = oct,
  doi = {10.1038/s41587-026-03307-w},
  url = {https://doi.org/10.1038/s41587-026-03307-w},
  note = {CC BY},
}
RIS
TY  - JOUR
TI  - Deep learning perturbation models can outperform baselines on calibrated metrics
AU  - Henry E. Miller
AU  - Gabriel Mejia
AU  - Francis Leblanc
AU  - Brendan Swain
AU  - Bo Wang
AU  - Lucas Paulo de Lima Camillo
PY  - 2026
DA  - 2026/10/01
JO  - Nature Biotechnology
DO  - 10.1038/s41587-026-03307-w
UR  - https://doi.org/10.1038/s41587-026-03307-w
AB  - Recent benchmarks report that deep-learning-based genetic perturbation models fail to outperform uninformative baselines. Here we introduce a positive control baseline and metric calibration framework, showing—across 14 datasets and 18 metrics—that common benchmarking metrics (for example, mean squared error and Pearson Δ ) are frequently miscalibrated, with reduced sensitivity to measure positive model performance. Under well-calibrated metrics, we find that deep-learning-based genetic perturbation models can outperform uninformative baselines.
SN  - 0733-222X
SN  - 1087-0156
SN  - 1546-1696
SN  - 2331-3684
ER  -