Systematic evaluation of semantic similarity across language models for schizophrenia spectrum disorders
Abstract
BackgroundSemantic similarity features derived from language embeddings have been proposed as markers of schizophrenia spectrum disorders (SSD), but prior studies have reported inconsistent directions of effect and limited evidence regarding their relationship to specific dimensions of thought and language disturbance. We systematically evaluated how language model architecture and preprocessing strategy influence semantic similarity signals in SSD.
MethodsSpeech samples from 228 participants (152 SSD, 76 healthy volunteers) were analyzed using six embedding models: GloVe, LSA, Word2Vec, RoBERTa, GPT-2, and Llama. Semantic similarity features were computed across three preprocessing levels (verbatim, disfluency-removed, and disfluency-removed with stopword filtering) using k-interword, moving-window, and sentence-to-sentence comparisons, summarized with various statistical descriptors. Features were compared between SSD and healthy volunteers and examined for associations with dimensions of thought and language disturbance, including inefficiency, incoherence, and impaired expressivity.
ResultsSSD was predominantly associated with increased semantic similarity across models and comparison strategies. Model performance varied by preprocessing. RoBERTa performed best on minimally processed transcripts, yielding the greatest number of diagnosis-associated features, whereas GPT-2 achieved the largest diagnostic effect size after maximal preprocessing. GPT-2 showed the strongest signal for inefficiency and incoherence, while RoBERTa yielded the greatest number of associations with impaired expressivity.
ConclusionsSSD diagnosis and its dimensions of thought and language abnormalities are characterized primarily by greater semantic proximity in open-ended speech. Optimal preprocessing depends on model architecture and the clinical dimension of interest, highlighting the importance of model-and phenotype-specific analytic choices in computational studies of thought and language disturbance in SSD.
Related
- Examining the links between vitamin C and schizophrenia spectrum disorders: a systematic review and meta-analysis on circulating levels and supplementation outcomes.
- Retrospective harmonization of neuropsychiatric questionnaires in a large multi-site research consortium using expert mappings and semantic textual similarity
- Using Diffusion Similarity Networks to Identify, Track, and Predict Movement Disorders
- Ultra-rare coding variants in schizophrenia risk genes are associated with longitudinal phenotypic differences across psychotic disorders
- Pleiotropic genetic architecture linking schizophrenia and substance use disorders
BibTeX
@misc{nikzad2026systematic,
title = {{Systematic evaluation of semantic similarity across language models for schizophrenia spectrum disorders}},
author = {Amir H. Nikzad and Yan Cong and Ryan Partlan and Jiefei Li and Sunghye Cho and Sunny X. Tang},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.23.26366346},
url = {https://www.medrxiv.org/content/10.64898/2026.10.23.26366346v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Systematic evaluation of semantic similarity across language models for schizophrenia spectrum disorders
AU - Amir H. Nikzad
AU - Yan Cong
AU - Ryan Partlan
AU - Jiefei Li
AU - Sunghye Cho
AU - Sunny X. Tang
PY - 2026
DA - 2026/10/23
PB - medRxiv
DO - 10.64898/2026.10.23.26366346
UR - https://www.medrxiv.org/content/10.64898/2026.10.23.26366346v1
AB - BackgroundSemantic similarity features derived from language embeddings have been proposed as markers of schizophrenia spectrum disorders (SSD), but prior studies have reported inconsistent directions of effect and limited evidence regarding their relationship to specific dimensions of thought and language disturbance. We systematically evaluated how language model architecture and preprocessing strategy influence semantic similarity signals in SSD. MethodsSpeech samples from 228 participants (152 SSD, 76 healthy volunteers) were analyzed using six embedding models: GloVe, LSA, Word2Vec, RoBERTa, GPT-2, and Llama. Semantic similarity features were computed across three preprocessing levels (verbatim, disfluency-removed, and disfluency-removed with stopword filtering) using k-interword, moving-window, and sentence-to-sentence comparisons, summarized with various statistical descriptors. Features were compared between SSD and healthy volunteers and examined for associations with dimensions of thought and language disturbance, including inefficiency, incoherence, and impaired expressivity. ResultsSSD was predominantly associated with increased semantic similarity across models and comparison strategies. Model performance varied by preprocessing. RoBERTa performed best on minimally processed transcripts, yielding the greatest number of diagnosis-associated features, whereas GPT-2 achieved the largest diagnostic effect size after maximal preprocessing. GPT-2 showed the strongest signal for inefficiency and incoherence, while RoBERTa yielded the greatest number of associations with impaired expressivity. ConclusionsSSD diagnosis and its dimensions of thought and language abnormalities are characterized primarily by greater semantic proximity in open-ended speech. Optimal preprocessing depends on model architecture and the clinical dimension of interest, highlighting the importance of model-and phenotype-specific analytic choices in computational studies of thought and language disturbance in SSD.
ER -