Registered Replication Report and Extension of Maas & Hox (2005), Sufficient Sample Sizes for Multilevel Models
Abstract
The use of multilevel models in behavioral and social sciences has risen dramatically in the past several decades. Researchers using these models often consult highly-referenced simulation studies when designing their own studies, especially for sample-size planning.However, current literature suggests two issues with this practice. First, recent efforts to replicate findings from seminal simulation studies suggest the rate of replicability is far from ideal (Luijken et al., 2024). Second, researchers are now using more complex models but foundational simulation studies were conducted under much simpler conditions. To address these two issues, we propose a replication and extension of Maas & Hox (2005), a widely cited study that recommended Level-2 sample sizes of N = 50 to be sufficient under specific conditions. We replicate their core findings and propose several extensions based on a literature search of multilevel modeling practices employed in fifty recent studies.These extensions include examining conditions with imbalanced data across higher-level units, varying the number of random slopes and cross-level interactions, and varying effect sizes. These results could ensure the existing literature relies on a replicable simulation study while also informing multilevel study planning for complex models which are becoming increasingly popular in behavioral and social sciences.
Related
- Registered Report: Replication and Extension of Nozaradan et al. (2011)
- Compatibility Effects With Simple Lever Tools: A Replication and Extension Beyond Simple Button Responses
- Differential Effects of Task-Context and Stimulus-Category Changes on Recognition Memory: A Replication and Extension of Morse et al. (2023)
- Amygdala Persistence in Aging: Replication and Extension of Puccetti et al. (2021)
- A registered report on interruptions during visual search: Examining the effects of perceptual and cognitive load.
BibTeX
@misc{moreira2026registered,
title = {{Registered Replication Report and Extension of Maas & Hox (2005), Sufficient Sample Sizes for Multilevel Models}},
author = {Joao F. Guassi Moreira and Jared Block and Alondra M. Cruz and Craig K. Enders and Amanda Montoya},
howpublished = {PsyArXiv},
year = {2026},
month = nov,
doi = {10.31234/osf.io/fk8jz_v2},
url = {https://osf.io/preprints/psyarxiv/fk8jz_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Registered Replication Report and Extension of Maas & Hox (2005), Sufficient Sample Sizes for Multilevel Models
AU - Joao F. Guassi Moreira
AU - Jared Block
AU - Alondra M. Cruz
AU - Craig K. Enders
AU - Amanda Montoya
PY - 2026
DA - 2026/11/05
PB - PsyArXiv
DO - 10.31234/osf.io/fk8jz_v2
UR - https://osf.io/preprints/psyarxiv/fk8jz_v2/
AB - The use of multilevel models in behavioral and social sciences has risen dramatically in the past several decades. Researchers using these models often consult highly-referenced simulation studies when designing their own studies, especially for sample-size planning.However, current literature suggests two issues with this practice. First, recent efforts to replicate findings from seminal simulation studies suggest the rate of replicability is far from ideal (Luijken et al., 2024). Second, researchers are now using more complex models but foundational simulation studies were conducted under much simpler conditions. To address these two issues, we propose a replication and extension of Maas & Hox (2005), a widely cited study that recommended Level-2 sample sizes of N = 50 to be sufficient under specific conditions. We replicate their core findings and propose several extensions based on a literature search of multilevel modeling practices employed in fifty recent studies.These extensions include examining conditions with imbalanced data across higher-level units, varying the number of random slopes and cross-level interactions, and varying effect sizes. These results could ensure the existing literature relies on a replicable simulation study while also informing multilevel study planning for complex models which are becoming increasingly popular in behavioral and social sciences.
ER -