Large-scale factor analysis shows machine intelligence is only partially interpretable
Abstract
A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like fluid intelligence in humans. This assumption is rarely tested directly, and prior attempts have done so only at a much smaller scale. We take a latent variable approach to intelligence in language models, similar to how psychometricians study psychological constructs. Performance in every specific problem set is influenced by a domain-specific and a domain-agnostic latent factor. Using factor analysis as a dimension-reduction technique, we analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks. Due to the super-sparse nature of the dataset, we triangulate our analysis across different data densifiers and imputation methods. A robust pattern across different modes of bias is that 1. A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions, 2. Content-similar benchmarks do not necessarily cluster together, and 3. The g factor is not dominated by any common theme, and there is a lack of evidence that it is well-proxied by standard "intelligence" benchmarks. Our findings go against current endeavors of defining, identifying, and targeting general intelligence as a tangible construct in language model development. This leaves the strategy of targeting a single conceptual ability without support, since the first-order abilities it would have to reach are often partially idiosyncratic and not identifiable in practice.
Related
- Large Language Models Predict Human Social Behavior via Interpretable Mechanisms
- Interpretable Machine Learning Links Group-Specific Plasma Proteomics to Regional Structural Neuroimaging in Alzheimer's Disease
- All signals considered: Data quality partially explains inter-individual task differences in a large, open fNIRS dataset
- BRIDGE-AD reveals Alzheimer's disease effectors through interpretable large-scale omics integration
- Interpretable Symptom Vectors for Depression in a Large Language Model
BibTeX
@misc{haznitrama2026largescale,
title = {{Large-scale factor analysis shows machine intelligence is only partially interpretable}},
author = {Faiz Ghifari Haznitrama and Afrizal Hasbi Azizy and Faeyza Rishad Ardi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.36515},
eprint = {2609.36515},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.36515},
note = {CC BY-NC-SA 4.0},
}
RIS
TY - UNPB
TI - Large-scale factor analysis shows machine intelligence is only partially interpretable
AU - Faiz Ghifari Haznitrama
AU - Afrizal Hasbi Azizy
AU - Faeyza Rishad Ardi
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.36515
UR - https://arxiv.org/abs/2609.36515
AB - A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like fluid intelligence in humans. This assumption is rarely tested directly, and prior attempts have done so only at a much smaller scale. We take a latent variable approach to intelligence in language models, similar to how psychometricians study psychological constructs. Performance in every specific problem set is influenced by a domain-specific and a domain-agnostic latent factor. Using factor analysis as a dimension-reduction technique, we analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks. Due to the super-sparse nature of the dataset, we triangulate our analysis across different data densifiers and imputation methods. A robust pattern across different modes of bias is that 1. A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions, 2. Content-similar benchmarks do not necessarily cluster together, and 3. The g factor is not dominated by any common theme, and there is a lack of evidence that it is well-proxied by standard "intelligence" benchmarks. Our findings go against current endeavors of defining, identifying, and targeting general intelligence as a tangible construct in language model development. This leaves the strategy of targeting a single conceptual ability without support, since the first-order abilities it would have to reach are often partially idiosyncratic and not identifiable in practice.
ER -