CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models
Abstract
Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data from its source study. Like humans, models respond more to n-back lures than to controls, yet they make far more false alarms on ordinary non-targets. Most models' reported confidence separates correct from incorrect answers less well than humans' does, and most always choose the same wager, including models that classify accurately. These patterns span many model families. Across tasks, model scores share two dimensions. The first is closely tied to scale, yet larger models are not consistently closer to humans. Predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size. These results show that matching human effect directions does not establish human-like behavior. CogBattery provides condition-level human references for testing errors and confidence reports in new models.
Related
- Neurocognition of dance: Guidelines for using dance as model behavior and research subject in cognitive neuroscience
- Optimizing Cognitive Phenotyping for Genetic Discovery in Alzheimer's Disease: A Simulation-Based Evaluation of Mini-Mental State Examination-Based Decline Measures
- The role of astrocytes in the temporoammonic pathway: masticatory behavior as a neuroprotective strategy against age-related cognitive decline.
- Teaching others reveals hidden structure in conceptual cognitive maps
- A mechanistic, mortality-referenced biological age from routine blood tests: construction and temporal validation of IQN-BIOAGE-01
BibTeX
@misc{hou2026cogbattery,
title = {{CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models}},
author = {Dengzhe Hou and Lingyu Jiang and Fangzhou Lin and Kazunori D Yamada},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2607.24999},
eprint = {2607.24999},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.24999},
}
RIS
TY - UNPB
TI - CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models
AU - Dengzhe Hou
AU - Lingyu Jiang
AU - Fangzhou Lin
AU - Kazunori D Yamada
PY - 2026
DA - 2026/10/30
PB - arXiv
DO - 10.48550/arxiv.2607.24999
UR - https://arxiv.org/abs/2607.24999
AB - Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data from its source study. Like humans, models respond more to n-back lures than to controls, yet they make far more false alarms on ordinary non-targets. Most models' reported confidence separates correct from incorrect answers less well than humans' does, and most always choose the same wager, including models that classify accurately. These patterns span many model families. Across tasks, model scores share two dimensions. The first is closely tied to scale, yet larger models are not consistently closer to humans. Predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size. These results show that matching human effect directions does not establish human-like behavior. CogBattery provides condition-level human references for testing errors and confidence reports in new models.
ER -