Addressing Power of Equivalence Tests for Statistical Fairness Assessments of Machine Learning Models in Healthcare
4 affiliations
- 1Department of Quantitative Health Sciences, Mayo Clinic, Rochester, MN 55905, USA
- 2Precision Population Science Lab, Mayo Clinic, Rochester, MN 55905, USA
- 3Department of Pediatric and Adolescent Medicine, Mayo Clinic, Rochester, MN 55905, USA
- 4Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA
Abstract
Background/Objectives: Machine learning and artificial intelligence (AI) have substantial potential to change the practice of oncology through individualized patient care; however, it is important to ensure that any benefits are realized equitably. There are several examples of AI tools exhibiting bias towards traditionally under-resourced or marginalized groups. Proper validation that these models perform well across populations requires rigorous statistical design and interpretation. One component that is currently lacking in fairness assessments is a clear, prespecified definition of the evidence required to conclude that an AI tool is fair. Methods: To address this issue, we embed the notion of fairness assessment of an AI model within the statistical equivalence testing framework. This approach provides an a priori definition of fairness and an explicit decision rule for concluding, with a high degree of confidence, that model performance falls within a prespecified margin. The framework also supplies power and sample size calculations to support the design phase of a fairness assessment, which we develop for the case of modeling a binary outcome and a binary classification. As a demonstration, we use a “four-fifths” rule of thumb to set equivalence margins but emphasize the necessity of clinical and ethical context for choosing the margins in practice. Results: Using two published oncology models, we show that a realistic sample size can be adequate for some performance metrics and inadequate for others. We also address power and sample size considerations for asymmetric association measures and impossibility results for fairness. Conclusions: Assessments of AI performance fairness should be designed with intention such that target metrics and equivalence margins are specified in advance with ample sample size to conclude fairness.
Related
- Survival modeling and assessing prognostic factors of colorectal cancer in Ethiopia: Addressing violation of statistical assumptions
- Content and design considerations for digital supportive care tools for patients with cancer: A textual statistical analysis of patients’, relatives’, and healthcare professionals’ perspectives
- Assessment of Fairness and Bias of an Image-based Surgical Site Infection Detection AI Model
- Complementary chemical and effect-based risk assessments reveal ecotoxicological risks in Caribbean coastal bays and coral reefs
- Greater valgus severity is associated with larger lateral flexion‐gap changes between intraoperative assessments during robot‐assisted total knee arthroplasty: A retrospective cohort study
BibTeX
@article{watson2026addressing,
title = {{Addressing Power of Equivalence Tests for Statistical Fairness Assessments of Machine Learning Models in Healthcare}},
author = {David Watson and Momin M. Malik and Jennifer Le-Rademacher and Mahmoud AlJuhani and Shauna M. Overgaard and Chung-Il Wi and Young J. Juhn},
journal = {Cancers},
year = {2026},
month = oct,
doi = {10.3390/cancers18213514},
url = {https://www.mdpi.com/2072-6694/18/21/3514},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Addressing Power of Equivalence Tests for Statistical Fairness Assessments of Machine Learning Models in Healthcare
AU - David Watson
AU - Momin M. Malik
AU - Jennifer Le-Rademacher
AU - Mahmoud AlJuhani
AU - Shauna M. Overgaard
AU - Chung-Il Wi
AU - Young J. Juhn
PY - 2026
DA - 2026/10/31
JO - Cancers
DO - 10.3390/cancers18213514
UR - https://www.mdpi.com/2072-6694/18/21/3514
AB - Background/Objectives: Machine learning and artificial intelligence (AI) have substantial potential to change the practice of oncology through individualized patient care; however, it is important to ensure that any benefits are realized equitably. There are several examples of AI tools exhibiting bias towards traditionally under-resourced or marginalized groups. Proper validation that these models perform well across populations requires rigorous statistical design and interpretation. One component that is currently lacking in fairness assessments is a clear, prespecified definition of the evidence required to conclude that an AI tool is fair. Methods: To address this issue, we embed the notion of fairness assessment of an AI model within the statistical equivalence testing framework. This approach provides an a priori definition of fairness and an explicit decision rule for concluding, with a high degree of confidence, that model performance falls within a prespecified margin. The framework also supplies power and sample size calculations to support the design phase of a fairness assessment, which we develop for the case of modeling a binary outcome and a binary classification. As a demonstration, we use a “four-fifths” rule of thumb to set equivalence margins but emphasize the necessity of clinical and ethical context for choosing the margins in practice. Results: Using two published oncology models, we show that a realistic sample size can be adequate for some performance metrics and inadequate for others. We also address power and sample size considerations for asymmetric association measures and impossibility results for fairness. Conclusions: Assessments of AI performance fairness should be designed with intention such that target metrics and equivalence margins are specified in advance with ample sample size to conclude fairness.
SN - 2072-6694
ER -