Beyond Rules of Thumb: Empirical Sample Size Requirements for Detecting Institutional Effects in the context of Therapist Effects
Abstract
Objective: Multilevel modelling of therapist effects (TE) and institutional effects (IE) is standard in psychotherapy research, yet empirical sample-size guidance is scarce for TE and virtually absent for IE. Existing recommendations rely on rules of thumb from two-level designs that vary widely across sources and do not extend to three-level structures. We aimed to provide empirically derived design recommendations for reliably detecting TE and IE in three-level psychotherapy studies. Methods: We conducted a fully factorial Monte Carlo simulation crossing six design factors: true TE, true IE, patient allocation (balanced vs. unbalanced), patients per therapist, therapists per institution, and number of institutions. This yielded 52,488 unique conditions, each with 1,000 replications. For every condition we fitted a three-level random-intercept model and derived parametric bootstrap 95% confidence intervals for both intraclass correlations, evaluating bias, RMSE, coverage, and statistical power at three pre-registered thresholds. Results: Detecting TE was feasible across a wide range of realistic designs, requiring as few as 10 to 40 total therapists for moderate effects (TE ≥ 0.05). Detecting IE, in contrast, was strongly constrained by the number of institutions: no design with fewer than four institutions reached 80% power at the primary threshold, and adequate detection of small to moderate IE (≤ 0.20) required at least five institutions with substantial within-institution sampling. Conclusion: Reliably detecting IE imposes qualitatively different design requirements than TE, particularly regarding the minimum number of institutions. We provide cutoff tables and a Shiny application to support empirically informed sample-size decisions.
Related
- Immune- and Stromal-Associated Tissue-Context Profiles in Pretreatment Triple-Negative Breast Cancer: An Exploratory Institutional Cohort Study.
- The Impact of the Memory Support Intervention on Therapist Memory for Treatment Contents
- One Dance, Two Partners: Session-to-Session Therapist–Client Rating Configurations and Outcomes
- Intuitive therapist robot patient physical interaction is worth a thousand words
- Comparative Evaluation of a System One Model and a General-Purpose Large Language Model on the Korean Physical Therapist Licensing Examination
BibTeX
@misc{loray2026beyond,
title = {{Beyond Rules of Thumb: Empirical Sample Size Requirements for Detecting Institutional Effects in the context of Therapist Effects}},
author = {Juan Segundo Peña Loray and Björn S. Siepe and Miriam Hehlmann and Anne-Katharina Deisenhofer and Robin Anno Wester and Julian Rubel},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/gswc9_v1},
url = {https://osf.io/preprints/psyarxiv/gswc9_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Beyond Rules of Thumb: Empirical Sample Size Requirements for Detecting Institutional Effects in the context of Therapist Effects
AU - Juan Segundo Peña Loray
AU - Björn S. Siepe
AU - Miriam Hehlmann
AU - Anne-Katharina Deisenhofer
AU - Robin Anno Wester
AU - Julian Rubel
PY - 2026
DA - 2026/09/26
PB - PsyArXiv
DO - 10.31234/osf.io/gswc9_v1
UR - https://osf.io/preprints/psyarxiv/gswc9_v1/
AB - Objective: Multilevel modelling of therapist effects (TE) and institutional effects (IE) is standard in psychotherapy research, yet empirical sample-size guidance is scarce for TE and virtually absent for IE. Existing recommendations rely on rules of thumb from two-level designs that vary widely across sources and do not extend to three-level structures. We aimed to provide empirically derived design recommendations for reliably detecting TE and IE in three-level psychotherapy studies. Methods: We conducted a fully factorial Monte Carlo simulation crossing six design factors: true TE, true IE, patient allocation (balanced vs. unbalanced), patients per therapist, therapists per institution, and number of institutions. This yielded 52,488 unique conditions, each with 1,000 replications. For every condition we fitted a three-level random-intercept model and derived parametric bootstrap 95% confidence intervals for both intraclass correlations, evaluating bias, RMSE, coverage, and statistical power at three pre-registered thresholds. Results: Detecting TE was feasible across a wide range of realistic designs, requiring as few as 10 to 40 total therapists for moderate effects (TE ≥ 0.05). Detecting IE, in contrast, was strongly constrained by the number of institutions: no design with fewer than four institutions reached 80% power at the primary threshold, and adequate detection of small to moderate IE (≤ 0.20) required at least five institutions with substantial within-institution sampling. Conclusion: Reliably detecting IE imposes qualitatively different design requirements than TE, particularly regarding the minimum number of institutions. We provide cutoff tables and a Shiny application to support empirically informed sample-size decisions.
ER -