Context Matters in LLM-Assisted Qualitative Data Analysis: Workflow Development and Multidimensional Evaluation in Health Research
- 1Leiden University Medical Center
Abstract
Large language models (LLMs) are increasingly used for qualitative analysis, but strong language performance does not guarantee strong interpretation when meaning depends on method and context. We developed a context-specific workflow that translated framework analysis into bounded, sequential LLM-supported tasks and applied it to 20 Chinese-language interviews on fear of dementia (approximately 150,000 characters). Four multilingual LLM-generated outputs and a previous researcher-only analysis were blindly evaluated by 16 relevant experts across different analytical domains and qualitative quality dimensions (496 expert-item records). All LLMs completed the workflow, but rankings varied markedly. Qwen-output showed the most consistently favourable overall profile, while Claude-output was most often ranked first in individual domain-dimension evaluations. The researcher analysis received both highly favourable and highly unfavourable rankings; no output was consistently preferred. LLM-assisted qualitative analysis should be evaluated within its research contexts, with transparent workflows, multidimensional assessment, and continued researcher oversight.
Related
- Measuring residents' perceptions of social-ecological healthy communities: development and initial psychometric evaluation of a multidimensional scale.
- Recommendations for the development, implementation, and evaluation of technology-assisted physical activity interventions to support healthy aging: insights from PhysAgeNet
- Real-world implementation of AI-assisted PERT-workflow and its association with care processes and budget impact: a retrospective cohort study
- Using qualitative workflow mapping to plan for implementation of pediatric vaccine promotion intervention and trial
- Development and psychometric validation of a multidimensional menstrual hygiene scale
BibTeX
@misc{dai2026context,
title = {{Context Matters in LLM-Assisted Qualitative Data Analysis: Workflow Development and Multidimensional Evaluation in Health Research}},
author = {H. Dai and Y. Li and M. Villalobos-Quesada and Y. M. Ng and J. Aardoom and A. Versluis and H. Shen and M. Spruit and N. Chavannes},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.07.26362410},
url = {https://www.medrxiv.org/content/10.64898/2026.09.07.26362410v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Context Matters in LLM-Assisted Qualitative Data Analysis: Workflow Development and Multidimensional Evaluation in Health Research
AU - H. Dai
AU - Y. Li
AU - M. Villalobos-Quesada
AU - Y. M. Ng
AU - J. Aardoom
AU - A. Versluis
AU - H. Shen
AU - M. Spruit
AU - N. Chavannes
PY - 2026
DA - 2026/09/07
PB - medRxiv
DO - 10.64898/2026.09.07.26362410
UR - https://www.medrxiv.org/content/10.64898/2026.09.07.26362410v1
AB - Large language models (LLMs) are increasingly used for qualitative analysis, but strong language performance does not guarantee strong interpretation when meaning depends on method and context. We developed a context-specific workflow that translated framework analysis into bounded, sequential LLM-supported tasks and applied it to 20 Chinese-language interviews on fear of dementia (approximately 150,000 characters). Four multilingual LLM-generated outputs and a previous researcher-only analysis were blindly evaluated by 16 relevant experts across different analytical domains and qualitative quality dimensions (496 expert-item records). All LLMs completed the workflow, but rankings varied markedly. Qwen-output showed the most consistently favourable overall profile, while Claude-output was most often ranked first in individual domain-dimension evaluations. The researcher analysis received both highly favourable and highly unfavourable rankings; no output was consistently preferred. LLM-assisted qualitative analysis should be evaluated within its research contexts, with transparent workflows, multidimensional assessment, and continued researcher oversight.
ER -