In AI Do We Trust? Comparing a Transformer-Based Natural Language Processing Model Against a Traditional Content Analysis Approach to Qualitative Research
Abstract
Central to debates about the use of artificial intelligence (AI) models for qualitative research are concerns about comparability and trust in AI generated findings. The present methodological study compared findings from a transformer-based Natural Language Processing (NLP) model, BERT, against a traditional content analysis approach to qualitative research using the same text data. We examined claims that AI models generate findings that are identical or similar to traditional qualitative approaches. The study assessed three aspects of comparability between the BERT model solution and human-coded findings that included topic labeling, classification overlap, and AI replicability. Results indicated broad similarity in the semantic meaning of topic categories generated by the BERT model and themes from content analysis. There was low overlap in the classification of individual participant responses between BERT and content analysis categories, even for categories similar in semantic meaning. Running the BERT model a second time on the same input data resulted in differences in topic categories and document classification between the two approaches. Study implications and research recommendations for the use of NLP models are discussed.
Related
- Variational Bayesian inference for logistic regression models for group testing data: Comparing a new hybrid variational inference approach to traditional methods.
- Context-Aware Detection of Content Evidencing Illegal or Predatory Fintech Lending Practices on Indonesian Digital Platforms Using Transformer-Based Language Models
- Comparing local-global obstructions on algebraic stacks
- Comparing word probabilities in finite groups
- Comparing the Lee-Yang gap with the mass gap in the sub-critical planar Ising model
BibTeX
@misc{baron2026trust,
title = {{In AI Do We Trust? Comparing a Transformer-Based Natural Language Processing Model Against a Traditional Content Analysis Approach to Qualitative Research}},
author = {Matthew Baron and Catherine Stein},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/53uds_v1},
url = {https://osf.io/preprints/psyarxiv/53uds_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - In AI Do We Trust? Comparing a Transformer-Based Natural Language Processing Model Against a Traditional Content Analysis Approach to Qualitative Research
AU - Matthew Baron
AU - Catherine Stein
PY - 2026
DA - 2026/10/09
PB - PsyArXiv
DO - 10.31234/osf.io/53uds_v1
UR - https://osf.io/preprints/psyarxiv/53uds_v1/
AB - Central to debates about the use of artificial intelligence (AI) models for qualitative research are concerns about comparability and trust in AI generated findings. The present methodological study compared findings from a transformer-based Natural Language Processing (NLP) model, BERT, against a traditional content analysis approach to qualitative research using the same text data. We examined claims that AI models generate findings that are identical or similar to traditional qualitative approaches. The study assessed three aspects of comparability between the BERT model solution and human-coded findings that included topic labeling, classification overlap, and AI replicability. Results indicated broad similarity in the semantic meaning of topic categories generated by the BERT model and themes from content analysis. There was low overlap in the classification of individual participant responses between BERT and content analysis categories, even for categories similar in semantic meaning. Running the BERT model a second time on the same input data resulted in differences in topic categories and document classification between the two approaches. Study implications and research recommendations for the use of NLP models are discussed.
ER -