LLM-powered automatic scoring tools for memory recall
- 1Department of Psychological and Brain Sciences, Washington University in St. Louis
Abstract
Free recall of naturalistic stimuli, such as films and stories, reveals what people remember and how they organize it. Scoring such recall requires matching each recalled utterance to the encoded stimulus, and doing this manually often makes large studies impractical. Existing automated methods mostly return similarity or aggregate scores rather than explicit matches between recalled and stimulus content. Here we introduce an open-source, model-agnostic pipeline that uses a large language model (LLM) to divide the stimulus and the recall into information units and to match units between the two texts. The output is a table of unit-to-unit matches with accuracy labels, from which standard analysis scripts compute recall measures. We validated the pipeline with four LLMs (Claude Sonnet 4.5, GPT 5.4, Gemini 2.5 Pro, and Gemini 2.5 Flash) on three datasets. On typed recall of four short stories, agreement between each model and two trained raters ({phi} = .73-.79) was close to the agreement between the raters themselves ({phi} = .77). A second run reproduced each model's matches ({phi} = .91-.97) more closely than the raters agreed with each other. On spoken recall of 10 films, every model attributed recall to the correct film with high agreement. On uncorrected speech-to-text recall of a television episode, agreement varied more across models. The median cost was under $1 per participant for every model. These results suggest that the pipeline can serve as a reliable, low-cost alternative to manual scoring of free recall.
Related
- 793 PD1 expression in LAG3+ exhausted T cells restrains memory recall responses through a mechanism involving TCR repertoire diversity and non CD8+ T cell populations
- Episodic memory rescues working memory via pattern separation, pattern completion, and predictive recall
- The activation function of CA3 pyramidal neurons is optimal for the stable recall of memory patterns
- Longitudinal analysis of the effects of cladribine tablets on B and T cell memory recall responses to vaccine antigens in people with multiple sclerosis
- Performance-Metacognition Dissociation During Fear Memory Recall: Role of Task Sequence and Prefrontal TMS
BibTeX
@misc{tang2026llmpowered,
title = {{LLM-powered automatic scoring tools for memory recall}},
author = {R. S. Tang and S. J. Born and A. I. Delarazan and P. L. Demarest and A. B. Karagoz and Z. M. Reagh},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.01.755994},
url = {https://www.biorxiv.org/content/10.64898/2026.10.01.755994v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - LLM-powered automatic scoring tools for memory recall
AU - R. S. Tang
AU - S. J. Born
AU - A. I. Delarazan
AU - P. L. Demarest
AU - A. B. Karagoz
AU - Z. M. Reagh
PY - 2026
DA - 2026/10/01
PB - bioRxiv
DO - 10.64898/2026.10.01.755994
UR - https://www.biorxiv.org/content/10.64898/2026.10.01.755994v1
AB - Free recall of naturalistic stimuli, such as films and stories, reveals what people remember and how they organize it. Scoring such recall requires matching each recalled utterance to the encoded stimulus, and doing this manually often makes large studies impractical. Existing automated methods mostly return similarity or aggregate scores rather than explicit matches between recalled and stimulus content. Here we introduce an open-source, model-agnostic pipeline that uses a large language model (LLM) to divide the stimulus and the recall into information units and to match units between the two texts. The output is a table of unit-to-unit matches with accuracy labels, from which standard analysis scripts compute recall measures. We validated the pipeline with four LLMs (Claude Sonnet 4.5, GPT 5.4, Gemini 2.5 Pro, and Gemini 2.5 Flash) on three datasets. On typed recall of four short stories, agreement between each model and two trained raters ({phi} = .73-.79) was close to the agreement between the raters themselves ({phi} = .77). A second run reproduced each model's matches ({phi} = .91-.97) more closely than the raters agreed with each other. On spoken recall of 10 films, every model attributed recall to the correct film with high agreement. On uncorrected speech-to-text recall of a television episode, agreement varied more across models. The median cost was under $1 per participant for every model. These results suggest that the pipeline can serve as a reliable, low-cost alternative to manual scoring of free recall.
ER -