Blending Sources of Evidence: Meta-Analysis Combining Aggregated Data and Individual Participant Data from Multilab Replications
Abstract
Conventional meta-analyses synthesize average treatment effects from sample-level aggregated data (AD) and are limited to addressing questions of moderation only at the sample level. Although associations between study-mean covariates and effect sizes need not reflect individual-level relations, individual-level moderating effects can be investigated using individual participant data (IPD). Multilab replication projects have recently made large, preregistered IPD collections publicly available for effects also synthesized in conventional meta-analyses, so the two data sources could, in principle, be combined. Determining whether they should be combined, however, requires judgment about their compatibility. We propose four guiding questions for the preliminary evaluation of such syntheses, concerning outcome measurement, primary study sample sizes, the distribution of study features, and publication bias, and show how graphical and tabular summaries can inform each consideration. We then describe meta-analytic models for combining AD and IPD from studies with continuous outcomes. We review two-stage and one-stage approaches for estimating the overall effect distribution and study-level moderation, as well as for separating within-study from between-study moderation of individual characteristics, including a novel bivariate two-stage formulation that jointly synthesizes treatment and moderation effects. We illustrate the framework with two examples, each pairing an AD meta-analysis with a multilab replication of the same effect. In an example of moral licensing effects, the preliminary evaluation indicates that combining the sources is not warranted; in an example of social heuristics effects, synthesizing across data sources is warranted. Together, the examples show that whether two sources can be combined is an empirical question best settled before modeling begins, and that the specification that follows should be tailored to the features of the data that the preliminary evaluation brings to light.
Related
- Methods for Assessing Distributional Assumptions in Individual Participant Data Meta-Analysis of Diagnostic Tests
- Outdoor attractive targeted sugar bait Phase III trials for malaria control in Kenya, Mali, and Zambia: An individual participant data meta-analysis
- Smooth periodic extensions by extrapolation and blending for FFT-based differentiation
- Cauchy-Aggregated Ridge Tests for High-Dimensional Factor Pricing Models
- Activity-Conditioned Residual Association from Aggregated Relational Data
BibTeX
@misc{zhang2026blending,
title = {{Blending Sources of Evidence: Meta-Analysis Combining Aggregated Data and Individual Participant Data from Multilab Replications}},
author = {Jingru Zhang and James E Pustejovsky},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/gztv2_v1},
url = {https://osf.io/preprints/psyarxiv/gztv2_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Blending Sources of Evidence: Meta-Analysis Combining Aggregated Data and Individual Participant Data from Multilab Replications
AU - Jingru Zhang
AU - James E Pustejovsky
PY - 2026
DA - 2026/09/16
PB - PsyArXiv
DO - 10.31234/osf.io/gztv2_v1
UR - https://osf.io/preprints/psyarxiv/gztv2_v1/
AB - Conventional meta-analyses synthesize average treatment effects from sample-level aggregated data (AD) and are limited to addressing questions of moderation only at the sample level. Although associations between study-mean covariates and effect sizes need not reflect individual-level relations, individual-level moderating effects can be investigated using individual participant data (IPD). Multilab replication projects have recently made large, preregistered IPD collections publicly available for effects also synthesized in conventional meta-analyses, so the two data sources could, in principle, be combined. Determining whether they should be combined, however, requires judgment about their compatibility. We propose four guiding questions for the preliminary evaluation of such syntheses, concerning outcome measurement, primary study sample sizes, the distribution of study features, and publication bias, and show how graphical and tabular summaries can inform each consideration. We then describe meta-analytic models for combining AD and IPD from studies with continuous outcomes. We review two-stage and one-stage approaches for estimating the overall effect distribution and study-level moderation, as well as for separating within-study from between-study moderation of individual characteristics, including a novel bivariate two-stage formulation that jointly synthesizes treatment and moderation effects. We illustrate the framework with two examples, each pairing an AD meta-analysis with a multilab replication of the same effect. In an example of moral licensing effects, the preliminary evaluation indicates that combining the sources is not warranted; in an example of social heuristics effects, synthesizing across data sources is warranted. Together, the examples show that whether two sources can be combined is an empirical question best settled before modeling begins, and that the specification that follows should be tailored to the features of the data that the preliminary evaluation brings to light.
ER -