When Predictive Fit Misleads: Selecting Marketing Mix Models Using Experimental RoAS Benchmarks
Abstract
Marketing Mix Models (MMMs) are used to estimate channel-level Return on Ad Spend (RoAS) and guide marketing investment, yet competing specifications are usually compared on predictive-fit metrics such as out-of-sample R-squared, MAPE, or RMSE. We show that predictive fit can diverge from recovery of the causal quantities that drive marketing decisions. In simulations with known RoAS, the best-fitting specification is often not the one that recovers RoAS most accurately; replicated studies show that this disagreement is systematic and persists under media-spend collinearity, noise, and a misspecified carryover shape. Selecting specifications against experimental RoAS benchmarks, even imprecise ones, improves RoAS recovery relative to fit-based selection, and a real-world application with experimental benchmarks shows the same pattern. We propose an applied workflow: calibrate MMM specifications against estimand-aligned experimental benchmarks when they exist, and otherwise report RoAS sensitivity across plausible specifications rather than relying on a single best-fitting model.
Related
- Robust Scenario-Based Data-Enabled Predictive Control of a Battery Energy Storage System: An Experimental Study
- Model Predictive Control for Dynamic Hydraulic Balancing in Building Radiator Heating Systems: Modeling, Design, and Experimental Validation
- Selecting the number of components in PCA via random signflips
- A Synthetic Benchmark Dataset with Endogenous Marketing Spend for Validating Marketing Mix Models
- Generative Marketing Mix Modeling: A Causal Inference Framework Linking GEO and GEM to Business Impact
BibTeX
@misc{chen2026when,
title = {{When Predictive Fit Misleads: Selecting Marketing Mix Models Using Experimental RoAS Benchmarks}},
author = {Hao Chen and Alvin Lim},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.06111},
eprint = {2611.06111},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.06111},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - When Predictive Fit Misleads: Selecting Marketing Mix Models Using Experimental RoAS Benchmarks
AU - Hao Chen
AU - Alvin Lim
PY - 2026
DA - 2026/11/04
PB - arXiv
DO - 10.48550/arxiv.2611.06111
UR - https://arxiv.org/abs/2611.06111
AB - Marketing Mix Models (MMMs) are used to estimate channel-level Return on Ad Spend (RoAS) and guide marketing investment, yet competing specifications are usually compared on predictive-fit metrics such as out-of-sample R-squared, MAPE, or RMSE. We show that predictive fit can diverge from recovery of the causal quantities that drive marketing decisions. In simulations with known RoAS, the best-fitting specification is often not the one that recovers RoAS most accurately; replicated studies show that this disagreement is systematic and persists under media-spend collinearity, noise, and a misspecified carryover shape. Selecting specifications against experimental RoAS benchmarks, even imprecise ones, improves RoAS recovery relative to fit-based selection, and a real-world application with experimental benchmarks shows the same pattern. We propose an applied workflow: calibrate MMM specifications against estimand-aligned experimental benchmarks when they exist, and otherwise report RoAS sensitivity across plausible specifications rather than relying on a single best-fitting model.
ER -