GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students
Abstract
There is growing international interest in generative AI (GenAI) literacy and its assessment among high school students, but objective assessment in this population remains underdeveloped. This article reports the iterative development and validation of the GenAI Literacy Test (GenAIT), an 18-item multiple-choice test measuring high school students' conceptual knowledge about GenAI, with content spanning technical, practical, and human-impact domains. Expert review of relevance, clarity, and comprehensiveness provided evidence of content validity. In a large-scale survey of 7432 Estonian high school students, we evaluated the psychometric functioning of the Estonian-language GenAIT using confirmatory factor analysis, classical test theory, and item response theory. Results supported approximate unidimensionality, broadly adequate reliability for group-level research (marginal reliability = .72, KR-20 = .69), and good fit of a three-parameter logistic model (RMSEA = .013, TLI = .987, CFI = .990, SRMSR = .021). Measurement precision was sufficient for the majority of students but varied substantially across the latent trait, with lower precision for lower scoring students. GenAIT is therefore more suitable for group-level research than high-stakes individual classification. GenAIT scores were unrelated to perceived usefulness and perceived ease of use, and negatively associated with LLM use frequency, suggesting that frequent use and favorable perceptions of AI should not be treated as proxies for conceptual understanding.
Related
- From skills to smart care: The mediating role of digital literacy in the relationship between digital competence and medical artificial intelligence readiness among nursing students
- Vernacular english literacy practices among language students using AI: A case study in Saudi Arabia
- How Generative Artificial Intelligence (GAI) reshapes students’ creativity in higher education: A chain path modeling Study from “Exploration–Exploitation” to “Co-Creation–Reflection”
- Artificial intelligence-related perceptions and E-health literacy among nursing students in Palestine: A cross-sectional study
- Development and psychometric validation of Mpox knowledge and preventive behavior measures in the Turkish population
BibTeX
@misc{puppart2026genait,
title = {{GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students}},
author = {Brett Puppart and Kristjan-Julius Laak and Jaan Aru},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2608.25815},
eprint = {2608.25815},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.25815},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students
AU - Brett Puppart
AU - Kristjan-Julius Laak
AU - Jaan Aru
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2608.25815
UR - https://arxiv.org/abs/2608.25815
AB - There is growing international interest in generative AI (GenAI) literacy and its assessment among high school students, but objective assessment in this population remains underdeveloped. This article reports the iterative development and validation of the GenAI Literacy Test (GenAIT), an 18-item multiple-choice test measuring high school students' conceptual knowledge about GenAI, with content spanning technical, practical, and human-impact domains. Expert review of relevance, clarity, and comprehensiveness provided evidence of content validity. In a large-scale survey of 7432 Estonian high school students, we evaluated the psychometric functioning of the Estonian-language GenAIT using confirmatory factor analysis, classical test theory, and item response theory. Results supported approximate unidimensionality, broadly adequate reliability for group-level research (marginal reliability = .72, KR-20 = .69), and good fit of a three-parameter logistic model (RMSEA = .013, TLI = .987, CFI = .990, SRMSR = .021). Measurement precision was sufficient for the majority of students but varied substantially across the latent trait, with lower precision for lower scoring students. GenAIT is therefore more suitable for group-level research than high-stakes individual classification. GenAIT scores were unrelated to perceived usefulness and perceived ease of use, and negatively associated with LLM use frequency, suggesting that frequent use and favorable perceptions of AI should not be treated as proxies for conceptual understanding.
ER -