LLM-Generated Well-being Intervention Recommendations Are Sensitive to Psychological Profiles: A Proof-of-Concept Expert-Rating Study
Abstract
Personalizing digital well-being interventions to fit the needs and preferences of individual users has been a longstanding challenge. Large language models (LLMs) may be well-suited for this task, using natural language reasoning to integrate heterogeneous user information into personalized guidance. We tested whether LLMs could generate intervention recommendations that were sensitive to differences across recipients’ psychological profiles. Across 178 undergraduate student profiles spanning 14 psychological measures, four LLMs recommended 10 scripted chatbot dialogues from a library of 222 options. Next, two experts rated the recommendations’ appropriateness for a given profile (1-10). For half of the profiles, raters saw recommendations generated for that student (matched condition); for the remainder, they saw recommendations generated for a different student (unmatched condition). As hypothesized, matched recommendations (M = 6.74) were rated as more appropriate than unmatched ones (M = 5.64, SMD = 0.50). These results provide evidence that LLMs can adapt their intervention recommendations to match psychological profiles. However, inter-rater reliability was poor (ICC = 0.22), and all LLMs showed substantial sensitivity to catalog order. As such, we stress the preliminary nature of the findings. Whether such tailoring translates into improved well-being outcomes for users remains to be tested.
Related
- AI-Generated Treatment Recommendations in Newly Diagnosed Lung Cancer: A Single-Center Feasibility Study in a Resource-Limited Setting.
- miR-221/222 Profiles in Small Extracellular Vesicles Derived from Gemcitabine-Sensitive and -Resistant A549 Cancer Stem-like Cell Spheroids: An In Vitro Study
- Beyond survival: a cross-sectional analysis of guideline recommendations for cognitive, sexual, and psychological problems in adult cancer survivors
- Protocol: WorkoutCPP - a pilot series of N-of-1 trials evaluating RL-generated adaptive exercise recommendations for pelvic pain management
- Controlled deep breathing technique intervention effects on resting EEG, subjective psychological distress, and executive function performance in young adults
BibTeX
@misc{kaveladze2026llmgenerated,
title = {{LLM-Generated Well-being Intervention Recommendations Are Sensitive to Psychological Profiles: A Proof-of-Concept Expert-Rating Study}},
author = {Benjamin Kaveladze and Elizabeth W. Lampe and Michael V. Heinz and Lisa Marsch and Andrew T. Campbell and Nicholas C. Jacobson},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/y3k8a_v3},
url = {https://osf.io/preprints/psyarxiv/y3k8a_v3/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - LLM-Generated Well-being Intervention Recommendations Are Sensitive to Psychological Profiles: A Proof-of-Concept Expert-Rating Study
AU - Benjamin Kaveladze
AU - Elizabeth W. Lampe
AU - Michael V. Heinz
AU - Lisa Marsch
AU - Andrew T. Campbell
AU - Nicholas C. Jacobson
PY - 2026
DA - 2026/10/15
PB - PsyArXiv
DO - 10.31234/osf.io/y3k8a_v3
UR - https://osf.io/preprints/psyarxiv/y3k8a_v3/
AB - Personalizing digital well-being interventions to fit the needs and preferences of individual users has been a longstanding challenge. Large language models (LLMs) may be well-suited for this task, using natural language reasoning to integrate heterogeneous user information into personalized guidance. We tested whether LLMs could generate intervention recommendations that were sensitive to differences across recipients’ psychological profiles. Across 178 undergraduate student profiles spanning 14 psychological measures, four LLMs recommended 10 scripted chatbot dialogues from a library of 222 options. Next, two experts rated the recommendations’ appropriateness for a given profile (1-10). For half of the profiles, raters saw recommendations generated for that student (matched condition); for the remainder, they saw recommendations generated for a different student (unmatched condition). As hypothesized, matched recommendations (M = 6.74) were rated as more appropriate than unmatched ones (M = 5.64, SMD = 0.50). These results provide evidence that LLMs can adapt their intervention recommendations to match psychological profiles. However, inter-rater reliability was poor (ICC = 0.22), and all LLMs showed substantial sensitivity to catalog order. As such, we stress the preliminary nature of the findings. Whether such tailoring translates into improved well-being outcomes for users remains to be tested.
ER -