Modeling Word Counts: Common Practice Risks False Positives
Abstract
Word counts or relative frequencies (e.g. the number of self-related words as a proportion of total words in a text) are routinely analyzed as dependent variables in the behavioral sciences. We demonstrate that conventional methods of analyzing word counts in this way are likely to lead to false positives, especially when text length varies between levels of the predictor variable. We performed Monte Carlo simulations to assess type I error rates for a wide range of affective and linguistic constructs across 9 datasets as well as known generative models, using 126 candidate statistical methods. We distinguish between three sources of error: distributional misspecification, insufficient control for text length, and biases in frequency estimation. Distributional misspecification occured in models that do not account for the overdispersion of word count distributions, such as Poisson regression. Insufficient control for text length was evident in all models that do not include the logarithm of text length as a covariate, or model it with a spline. Although we report evidence of systematic bias in frequency estimation for shorter texts as predicted by theoretical work, this bias is negligible relative to construct- and dataset-dependent text length effects. Among the methods investigated, we recommend Poisson-lognormal regression with log word count as a covariate, or linear regression with a spline on log word count. We also strongly caution against the use of text truncation and excerpting in analyses of word counts, as these are likely to introduce confounds that cannot be corrected with any statistical method.
Related
- Empirical Bayes in Bayesian learning: understanding a common practice
- Fundamental of Modeling for the Assessment of Organizational Risks in the Integrated Safety System at Explosion and Fire-Hazardous Enterprises
- A Spreadsheet Environment for Force, Torque and Strength of Materials Modeling: Bridging Analytical Mathematics and Engineering Practice
- Post Hoc Confidence Bounds On False Positives Using Learned Templates
- Decay of weighted cusp counts for congruence subgroups of SL2 over number fields
BibTeX
@misc{teitelbaum2026modeling,
title = {{Modeling Word Counts: Common Practice Risks False Positives}},
author = {Louis Teitelbaum and Almog Simchon},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/zy6ew_v1},
url = {https://osf.io/preprints/psyarxiv/zy6ew_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Modeling Word Counts: Common Practice Risks False Positives
AU - Louis Teitelbaum
AU - Almog Simchon
PY - 2026
DA - 2026/10/20
PB - PsyArXiv
DO - 10.31234/osf.io/zy6ew_v1
UR - https://osf.io/preprints/psyarxiv/zy6ew_v1/
AB - Word counts or relative frequencies (e.g. the number of self-related words as a proportion of total words in a text) are routinely analyzed as dependent variables in the behavioral sciences. We demonstrate that conventional methods of analyzing word counts in this way are likely to lead to false positives, especially when text length varies between levels of the predictor variable. We performed Monte Carlo simulations to assess type I error rates for a wide range of affective and linguistic constructs across 9 datasets as well as known generative models, using 126 candidate statistical methods. We distinguish between three sources of error: distributional misspecification, insufficient control for text length, and biases in frequency estimation. Distributional misspecification occured in models that do not account for the overdispersion of word count distributions, such as Poisson regression. Insufficient control for text length was evident in all models that do not include the logarithm of text length as a covariate, or model it with a spline. Although we report evidence of systematic bias in frequency estimation for shorter texts as predicted by theoretical work, this bias is negligible relative to construct- and dataset-dependent text length effects. Among the methods investigated, we recommend Poisson-lognormal regression with log word count as a covariate, or linear regression with a spline on log word count. We also strongly caution against the use of text truncation and excerpting in analyses of word counts, as these are likely to introduce confounds that cannot be corrected with any statistical method.
ER -