Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications.
- 1Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh, 3396 Fifth Avenue, Pittsburgh, PA, 15213, United States, 1 412-383-6653.
Abstract
Background
Privacy concerns represent a primary barrier to the adoption of large language model (LLM) applications, particularly in sensitive domains such as health care. Although a growing body of survey-based research has examined user privacy concerns with LLMs, no standardized instrument exists for measuring this construct.Objective
This study aimed to develop and validate a concise, standardized questionnaire for assessing user privacy concerns with LLM-based systems, suitable for use by researchers and practitioners such as AI application developers and health care professionals.Methods
A literature review was conducted to identify privacy-related domains relevant to LLM use. A draft questionnaire was developed and subjected to content validity review by 8 experts. The finalized instrument was administered via Qualtrics to adults aged 18 years or older who were aware of LLM-based applications, recruited through the University of Pittsburgh, LinkedIn, and the Pitt+Me research participant registry between April 2025 and July 2026. Internal consistency reliability and factor structure were assessed, with factor structure examined in a split-sample design, in which exploratory factor analysis (EFA) was conducted on a random one-third of the sample and confirmatory factor analysis (CFA) on the holdout two-thirds.Results
All questionnaire items achieved acceptable content validity (item-level content validity index >0.80; average scale-level content validity index=0.964). A total of 603 responses were collected. After preprocessing, the final sample comprised 361 participants. Most participants were younger than 35 years of age (198/361, 54.8%), female (230/361, 63.7%), and White (285/361, 78.9%); 88.1% (318/361) held at least a bachelor's degree. The 6-item questionnaire showed high internal consistency (Cronbach α=0.90; McDonald ω=0.90). EFA (n=121) supported a single dominant factor with all standardized loadings ≥0.70. CFA (n=240) showed excellent fit (comparative fit index=0.996; Tucker-Lewis index=0.993; root mean square error of approximation=0.046; standardized root mean square residual=0.057). A 2-factor model separating the transparency- and control-related items fit significantly better than the 1-factor model (Δχ²₁=8.9; P=.003), but the 2 factors were correlated at r=0.83, indicating closely related facets. Because the Fornell-Larcker criterion was met for the transparency and control subscale but not for data use and risk salience, the 6-item total score remains the recommended primary metric.Conclusions
The LLM privacy concern assessment is a concise, psychometrically sound 6-item instrument that measures privacy concern with LLM-based systems via 2 correlated subdimensions (transparency and control; data use and risk salience) summarized by a single total score. It enables standardized measurement across studies and practical privacy assessment in real-world LLM applications.
Related
- Development and Psychometric Validation of the Geriatric Patient Safety Culture Questionnaire for Intensive Care Nurses
- Development and psychometric validation of the sun exposure and protection questionnaire (SEPQ) in young adults: a university-based validation study
- Development and Psychometric Validation of a Multidimensional Questionnaire for Assessing Interpersonal Social Health
- Validation of the Qualisex Questionnaire for Psychometric Assessment of Sexual Health in Lichen Sclerosus
- Development and Psychometric Validation of the Mistreatment Questionnaire for Medical Trainees (MQMT) in Mexican Medical Students, Interns, and Residents.
BibTeX
@article{zhou2026development,
title = {{Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications.}},
author = {Leming Zhou and Hongtao Wang and Yi Xu and I Made Agus Setiawan and Haomin Hu and Andi Saptono and Yong K Choi and Bambang Parmanto},
journal = {JMIR human factors},
year = {2026},
month = oct,
doi = {10.2196/102910},
url = {https://europepmc.org/article/MED/42906530},
note = {CC BY},
}
RIS
TY - JOUR
TI - Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications.
AU - Leming Zhou
AU - Hongtao Wang
AU - Yi Xu
AU - I Made Agus Setiawan
AU - Haomin Hu
AU - Andi Saptono
AU - Yong K Choi
AU - Bambang Parmanto
PY - 2026
DA - 2026/10/21
JO - JMIR human factors
DO - 10.2196/102910
UR - https://europepmc.org/article/MED/42906530
AB - Background Privacy concerns represent a primary barrier to the adoption of large language model (LLM) applications, particularly in sensitive domains such as health care. Although a growing body of survey-based research has examined user privacy concerns with LLMs, no standardized instrument exists for measuring this construct.Objective This study aimed to develop and validate a concise, standardized questionnaire for assessing user privacy concerns with LLM-based systems, suitable for use by researchers and practitioners such as AI application developers and health care professionals.Methods A literature review was conducted to identify privacy-related domains relevant to LLM use. A draft questionnaire was developed and subjected to content validity review by 8 experts. The finalized instrument was administered via Qualtrics to adults aged 18 years or older who were aware of LLM-based applications, recruited through the University of Pittsburgh, LinkedIn, and the Pitt+Me research participant registry between April 2025 and July 2026. Internal consistency reliability and factor structure were assessed, with factor structure examined in a split-sample design, in which exploratory factor analysis (EFA) was conducted on a random one-third of the sample and confirmatory factor analysis (CFA) on the holdout two-thirds.Results All questionnaire items achieved acceptable content validity (item-level content validity index >0.80; average scale-level content validity index=0.964). A total of 603 responses were collected. After preprocessing, the final sample comprised 361 participants. Most participants were younger than 35 years of age (198/361, 54.8%), female (230/361, 63.7%), and White (285/361, 78.9%); 88.1% (318/361) held at least a bachelor's degree. The 6-item questionnaire showed high internal consistency (Cronbach α=0.90; McDonald ω=0.90). EFA (n=121) supported a single dominant factor with all standardized loadings ≥0.70. CFA (n=240) showed excellent fit (comparative fit index=0.996; Tucker-Lewis index=0.993; root mean square error of approximation=0.046; standardized root mean square residual=0.057). A 2-factor model separating the transparency- and control-related items fit significantly better than the 1-factor model (Δχ²₁=8.9; P=.003), but the 2 factors were correlated at r=0.83, indicating closely related facets. Because the Fornell-Larcker criterion was met for the transparency and control subscale but not for data use and risk salience, the 6-item total score remains the recommended primary metric.Conclusions The LLM privacy concern assessment is a concise, psychometrically sound 6-item instrument that measures privacy concern with LLM-based systems via 2 correlated subdimensions (transparency and control; data use and risk salience) summarized by a single total score. It enables standardized measurement across studies and practical privacy assessment in real-world LLM applications.
SN - 2292-9495
ER -