Same Effect, Different Magnitude: How the Choice of Pre- or Post-Treatment Standard Deviation Shapes Standardized Effect Sizes in Psychological Clinical Trials
Abstract
Standardized mean differences (SMDs) are used to express treatment effects on a common metric, yet there is no consensus on whether mean differences should be standardized using the outcome standard deviation (SD) at baseline or follow-up. This study examined whether this choice systematically affects effect-size magnitude in a reproducible sample of 100 randomized psychological clinical trials. For each trial, SMDs were calculated using either the pooled baseline SD (SMDpre) or the pooled follow-up SD (SMDpost). The unweighted paired mean difference showed that |SMDpre| was, 0.11 SMD units larger than |SMDpost|, 95% CI [0.07, 0.14]. In a multilevel meta-regression |SMDpre| was 0.082 SMD units larger than |SMDpost|, equalling a 13% increase, 95% CI [0.042, 0.123]. Bland–Altman analysis indicated substantial between-study variation, 95% limits of agreement [−0.25, 0.46]. Secondary analyses including 2,228 effect sizes across all follow-ups mirrored this pattern, indicating a 7% increase of SMD magnitude associated to |SMDpre|, 95% CI [0.021, 0.044]. Moderator analyses suggested that discrepancies were larger in studies applying symptom-threshold eligibility criteria, and in secondary analyses, using the same measure both to determine eligibility and to assess the outcome and with longer follow-up. Importantly, 70% of included studies did not report how SMDs were operationalized. These findings demonstrate that the choice of standardizing SD systematically affects estimated treatment-effect magnitudes and that the discrepancy can become more pronounced under specific design conditions. Researchers should therefore carefully consider the choice of standardizer and explicitly report the time point and method used to derive the SD.
Related
- Expert Consensus on the Standardized Diagnosis, Treatment, and Management of Cancer in High-Altitude Regions of China.
- Standardized Clinical Photography for Improving Dermatological Treatment Adherence (Scoping Review): Evidence Landscape and Research Gaps.
- Unlocking the Potential of Cardiac Rehabilitation: From Psychological Barriers to Treatment Targets
- Association between the triglyceride-glucose index and functional outcome at discharge in patients receiving standardized treatment for atherosclerotic and cardioembolic stroke
- Interplay of Biological Treatment and Psychological Features in Patients with Psoriasis: A Real-Life Prospective Study.
BibTeX
@misc{pillny2026same,
title = {{Same Effect, Different Magnitude: How the Choice of Pre- or Post-Treatment Standard Deviation Shapes Standardized Effect Sizes in Psychological Clinical Trials}},
author = {Matthias Pillny and Anika Buchholz and Anna Suling and Antonia Zapf and Tania Marie Lincoln},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/43pyn_v2},
url = {https://osf.io/preprints/psyarxiv/43pyn_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Same Effect, Different Magnitude: How the Choice of Pre- or Post-Treatment Standard Deviation Shapes Standardized Effect Sizes in Psychological Clinical Trials
AU - Matthias Pillny
AU - Anika Buchholz
AU - Anna Suling
AU - Antonia Zapf
AU - Tania Marie Lincoln
PY - 2026
DA - 2026/10/03
PB - PsyArXiv
DO - 10.31234/osf.io/43pyn_v2
UR - https://osf.io/preprints/psyarxiv/43pyn_v2/
AB - Standardized mean differences (SMDs) are used to express treatment effects on a common metric, yet there is no consensus on whether mean differences should be standardized using the outcome standard deviation (SD) at baseline or follow-up. This study examined whether this choice systematically affects effect-size magnitude in a reproducible sample of 100 randomized psychological clinical trials. For each trial, SMDs were calculated using either the pooled baseline SD (SMDpre) or the pooled follow-up SD (SMDpost). The unweighted paired mean difference showed that |SMDpre| was, 0.11 SMD units larger than |SMDpost|, 95% CI [0.07, 0.14]. In a multilevel meta-regression |SMDpre| was 0.082 SMD units larger than |SMDpost|, equalling a 13% increase, 95% CI [0.042, 0.123]. Bland–Altman analysis indicated substantial between-study variation, 95% limits of agreement [−0.25, 0.46]. Secondary analyses including 2,228 effect sizes across all follow-ups mirrored this pattern, indicating a 7% increase of SMD magnitude associated to |SMDpre|, 95% CI [0.021, 0.044]. Moderator analyses suggested that discrepancies were larger in studies applying symptom-threshold eligibility criteria, and in secondary analyses, using the same measure both to determine eligibility and to assess the outcome and with longer follow-up. Importantly, 70% of included studies did not report how SMDs were operationalized. These findings demonstrate that the choice of standardizing SD systematically affects estimated treatment-effect magnitudes and that the discrepancy can become more pronounced under specific design conditions. Researchers should therefore carefully consider the choice of standardizer and explicitly report the time point and method used to derive the SD.
ER -