Psychometric applications of generative artificial intelligence: Lifecycle, risks, and research agenda
- 1Escuela de Posgrado, Universidad Continental, Lima, Peru
- 2Digital Health Research Center, Digital Health S.A., Lima, Peru
- 3Instituto Peruano de Orientación Psicológica, Lima, Peru
Abstract
Generative artificial intelligence (GenAI), including large language models and large multimodal models, is increasingly used to support psychological assessment and digital mental health measurement. This review proposes a lifecycle framework for evaluating these uses without weakening psychometric standards. The framework covers construct definition; item, prompt, indicator, or signal generation; evaluation of content, response processes, usability, and data quality; piloting and calibration; validation; fairness and measurement invariance; scoring and interpretation; documentation; and post-deployment monitoring. We argue that digital traces from smartphones, chatbots, ecological momentary assessment, wearables, social media, clinical notes, and multimodal systems should not be treated as psychometric measures by default. They should remain raw data, candidate indicators, or algorithmic scores until their construct interpretation and intended use are theoretically specified, technically verified, empirically calibrated, and validated with human data. GenAI may help generate candidate items, refine wording, classify open-text responses, extract structured information, support multimodal integration, and assist scoring under explicit rules. These uses may improve efficiency and scale, but they do not establish validity, objectivity, fairness, or clinical meaning. We distinguish two complementary facets: GenAI for psychometrics, in which models support measurement development and scoring, and psychometrics for GenAI, in which psychometric methods evaluate model behavior when LLMs are part of measurement workflows. Key risks include construct drift, face validity without structural validity, compressed variability in synthetic respondents, algorithmic bias, lack of invariance, prompt sensitivity, model drift, automation bias, data-security and privacy failures, and loss of subjectivity and disagreement. Responsible use requires human oversight, transparent reporting, validation with human data, fairness evaluation, secure data governance, and continuous monitoring. GenAI should augment, not replace, psychometric science.
Related
- Bias Mitigation Across the Healthcare Artificial Intelligence Lifecycle: A Structured Narrative Review
- Generative Artificial Intelligence Models in Breast Cancer Diagnosis: A Systematic Review
- Nursing students’ experiences with using generative artificial intelligence: a descriptive qualitative study
- Generative Artificial Intelligence in Periodontology: Application of OpenAI Models for the Classification of Periodontitis and Risk Assessment in Patients Undergoing Supportive Periodontal Care
- Simulating Psychotherapeutic Sessions with Generative Artificial Intelligence: A Proof-of-Concept Study of In Silico Psychotherapy Research
BibTeX
@article{villarrealzegarra2026psychometric,
title = {{Psychometric applications of generative artificial intelligence: Lifecycle, risks, and research agenda}},
author = {David Villarreal-Zegarra and Yscenia Paredes-Gonzales and Jackeline García-Serna},
journal = {PLOS Mental Health},
year = {2026},
month = sep,
doi = {10.1371/journal.pmen.0000702},
url = {https://journals.plos.org/mentalhealth/article?id=10.1371/journal.pmen.0000702},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Psychometric applications of generative artificial intelligence: Lifecycle, risks, and research agenda
AU - David Villarreal-Zegarra
AU - Yscenia Paredes-Gonzales
AU - Jackeline García-Serna
PY - 2026
DA - 2026/09/23
JO - PLOS Mental Health
DO - 10.1371/journal.pmen.0000702
UR - https://journals.plos.org/mentalhealth/article?id=10.1371/journal.pmen.0000702
AB - Generative artificial intelligence (GenAI), including large language models and large multimodal models, is increasingly used to support psychological assessment and digital mental health measurement. This review proposes a lifecycle framework for evaluating these uses without weakening psychometric standards. The framework covers construct definition; item, prompt, indicator, or signal generation; evaluation of content, response processes, usability, and data quality; piloting and calibration; validation; fairness and measurement invariance; scoring and interpretation; documentation; and post-deployment monitoring. We argue that digital traces from smartphones, chatbots, ecological momentary assessment, wearables, social media, clinical notes, and multimodal systems should not be treated as psychometric measures by default. They should remain raw data, candidate indicators, or algorithmic scores until their construct interpretation and intended use are theoretically specified, technically verified, empirically calibrated, and validated with human data. GenAI may help generate candidate items, refine wording, classify open-text responses, extract structured information, support multimodal integration, and assist scoring under explicit rules. These uses may improve efficiency and scale, but they do not establish validity, objectivity, fairness, or clinical meaning. We distinguish two complementary facets: GenAI for psychometrics, in which models support measurement development and scoring, and psychometrics for GenAI, in which psychometric methods evaluate model behavior when LLMs are part of measurement workflows. Key risks include construct drift, face validity without structural validity, compressed variability in synthetic respondents, algorithmic bias, lack of invariance, prompt sensitivity, model drift, automation bias, data-security and privacy failures, and loss of subjectivity and disagreement. Responsible use requires human oversight, transparent reporting, validation with human data, fairness evaluation, secure data governance, and continuous monitoring. GenAI should augment, not replace, psychometric science.
SN - 2837-8156
ER -