Do machine learning models really encode cross-cultural lifestyle-health differences? A leakage-free reanalysis of two university cohorts in India and Japan
- 1Faculty of Science & Technology, Sophia University , ,
- 2School of Engineering, St Aloysius (Deemed to be University) , ,
- 3Father Muller Medical College , ,
Abstract
Introduction: Machine-learning (ML) studies increasingly compare feature-importance rankings between small, unevenly sized cohorts and interpret the differences as evidence of culturally specific “risk architectures.” We show that such comparisons can be highly misleading without leakage-free evaluation, using a worked example from our own cross-sectional lifestyle and health data collected during June-August 2024 from two universities in India and Japan. Three problems were identified in our preliminary analysis: (1) an outcome (obesity, defined as BMI ≥ 25) that was tautologically predicted by including BMI itself as a model input; (2) model evaluation without a held-out test set; and (3) feature-importance comparisons that ignored sampling variability from an eightfold difference in cohort size.
Methods: We addressed each problem using penalized logistic regression, Random Forest, and gradient-boosted trees with proper train/test splitting, held-out permutation importance, and bootstrap confidence intervals. We also ran a sensitivity analysis that repeatedly subsampled the Indian cohort down to the size of the Japanese cohort.
Results: Removing body mass index (BMI) as a predictor collapsed obesity-model discrimination from perfect (AUC = 1.00) to chance level (AUC = 0.50-0.57), confirming that the BMI-obesity association was definitional rather than empirical. Held-out predictive performance for the remaining viable outcome, elevated self-rated stress, was modest at best (AUC 0.49-0.71). Critically, in the subsampling analysis, every feature-importance value observed in the actual Japanese data fell within the range produced by sample-size noise alone.
Discussion: We present this analysis as a transparent, reproducible case study of specific, checkable pitfalls: label leakage, missing train/test evaluation, unadjusted small-sample comparison, and mislabeled psychometric instruments. These should be routinely screened for before ML feature importance is interpreted as evidence of population-level or cultural differences in digital health research.
Related
- Exploring the Association of Fear, Mental Health, and Lifestyle with Insomnia: A Machine Learning Approach among University Students.
- Intentional Use of Voluntarily Fortified Foods in Relation to Socio-Demographic and Lifestyle Factors, Including Adherence to a DASH-Style Diet, Among University Students
- Language attitudes and ethnic or cultural identity among ethnic minority university students in China: a systematic bilingual knowledge synthesis
- Prevalence and functional impact of musculoskeletal pain among university students in seven Arab countries and its sociodemographic and lifestyle associations
- A descriptive reanalysis of prompt context differences in the Trinity score corpus
BibTeX
@article{takaoka2026machine,
title = {{Do machine learning models really encode cross-cultural lifestyle-health differences? A leakage-free reanalysis of two university cohorts in India and Japan}},
author = {Eiko Takaoka and S. Ruban and B. Mohammed Siraj and Varenya Vinay and Hilda Shanthini and Akshay Dias},
journal = {Frontiers in digital health},
year = {2026},
month = oct,
doi = {10.3389/fdgth.2026.1791066},
url = {https://europepmc.org/article/PMC/PMC13719548},
note = {CC BY},
}
RIS
TY - JOUR
TI - Do machine learning models really encode cross-cultural lifestyle-health differences? A leakage-free reanalysis of two university cohorts in India and Japan
AU - Eiko Takaoka
AU - S. Ruban
AU - B. Mohammed Siraj
AU - Varenya Vinay
AU - Hilda Shanthini
AU - Akshay Dias
PY - 2026
DA - 2026/10/16
JO - Frontiers in digital health
DO - 10.3389/fdgth.2026.1791066
UR - https://europepmc.org/article/PMC/PMC13719548
AB - Introduction: Machine-learning (ML) studies increasingly compare feature-importance rankings between small, unevenly sized cohorts and interpret the differences as evidence of culturally specific “risk architectures.” We show that such comparisons can be highly misleading without leakage-free evaluation, using a worked example from our own cross-sectional lifestyle and health data collected during June-August 2024 from two universities in India and Japan. Three problems were identified in our preliminary analysis: (1) an outcome (obesity, defined as BMI ≥ 25) that was tautologically predicted by including BMI itself as a model input; (2) model evaluation without a held-out test set; and (3) feature-importance comparisons that ignored sampling variability from an eightfold difference in cohort size. Methods: We addressed each problem using penalized logistic regression, Random Forest, and gradient-boosted trees with proper train/test splitting, held-out permutation importance, and bootstrap confidence intervals. We also ran a sensitivity analysis that repeatedly subsampled the Indian cohort down to the size of the Japanese cohort. Results: Removing body mass index (BMI) as a predictor collapsed obesity-model discrimination from perfect (AUC = 1.00) to chance level (AUC = 0.50-0.57), confirming that the BMI-obesity association was definitional rather than empirical. Held-out predictive performance for the remaining viable outcome, elevated self-rated stress, was modest at best (AUC 0.49-0.71). Critically, in the subsampling analysis, every feature-importance value observed in the actual Japanese data fell within the range produced by sample-size noise alone. Discussion: We present this analysis as a transparent, reproducible case study of specific, checkable pitfalls: label leakage, missing train/test evaluation, unadjusted small-sample comparison, and mislabeled psychometric instruments. These should be routinely screened for before ML feature importance is interpreted as evidence of population-level or cultural differences in digital health research.
SN - 2673-253X
ER -