Attempts to investigate what words trivia experts focus on in quiz questions: Human-AI comparison through “LLM-as-a-judge” approach
- 1Faculty of Informatics, Shizuoka University, Hamamatsu-shi, Shizuoka, Japan
Abstract
Computational capacity and knowledge of humans are more limited than those of large language models (LLMs). However, in buzzer quizzes, human trivia experts can often identify the correct answer even from insufficient information such as only a few words. Investigating how they can make fast and accurate judgments is expected to highlight new characteristics of human intelligence, but little is known about that. In this exploratory and case-based analysis, we predicted that trivia experts and LLMs would differ in which words/phrases in a question are important for identifying the answer, and compared experts’ performance with LLMs’ performance in Japanese trivia questions through an LLM-as-a-judge approach: We regarded LLMs as evaluators and then used their outputs as comparative tools for experts’ evaluations. First, we constructed a quiz question processing system that tokenized question texts based on morphological analysis and then numerically evaluated the importance of each token using GPT-4o/GPT-4.1. Then, we conducted a behavioral experiment wherein actual trivia experts were asked to numerically evaluate the importance of each token, just as LLMs had performed. As a result, trivia experts treated a variety of words/phrases as important, even if each word/phrase did not appear to be strongly associated with the answer. Their evaluation scores tended to accumulate faster than those of the LLMs. This may indicate that trivia experts can make inductive inferences faster (e.g., finding a common concept even from few items). More advanced question-answering systems may be designed by applying trivia experts’ cognitive processes to LLMs’ information processing, and our findings may provide a scaffolding toward such goals.
Related
- Harnessing survey data across multiple demographic groups in Illinois to assess survey questions and reveal insights about ticks and tick-borne disease risk: a meta-comparison study
- Pediatric Sleep Assessment through Spectral Sleep Scoring: A Comparison with Polysomnography
- 618 A novel preclinical mouse model to investigate IL-18-targeting therapies
- A series of replication attempts of experimental awe-induction effects
- ForceFlowAb: physics-aware mixture-of-experts flow matching model for antibody CDRs design
BibTeX
@article{shirasuna2026attempts,
title = {{Attempts to investigate what words trivia experts focus on in quiz questions: Human-AI comparison through “LLM-as-a-judge” approach}},
author = {Masaru Shirasuna and Yuto Yoshida},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0358332},
url = {https://doi.org/10.1371/journal.pone.0358332},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Attempts to investigate what words trivia experts focus on in quiz questions: Human-AI comparison through “LLM-as-a-judge” approach
AU - Masaru Shirasuna
AU - Yuto Yoshida
PY - 2026
DA - 2026/09/22
JO - PLOS One
DO - 10.1371/journal.pone.0358332
UR - https://doi.org/10.1371/journal.pone.0358332
AB - Computational capacity and knowledge of humans are more limited than those of large language models (LLMs). However, in buzzer quizzes, human trivia experts can often identify the correct answer even from insufficient information such as only a few words. Investigating how they can make fast and accurate judgments is expected to highlight new characteristics of human intelligence, but little is known about that. In this exploratory and case-based analysis, we predicted that trivia experts and LLMs would differ in which words/phrases in a question are important for identifying the answer, and compared experts’ performance with LLMs’ performance in Japanese trivia questions through an LLM-as-a-judge approach: We regarded LLMs as evaluators and then used their outputs as comparative tools for experts’ evaluations. First, we constructed a quiz question processing system that tokenized question texts based on morphological analysis and then numerically evaluated the importance of each token using GPT-4o/GPT-4.1. Then, we conducted a behavioral experiment wherein actual trivia experts were asked to numerically evaluate the importance of each token, just as LLMs had performed. As a result, trivia experts treated a variety of words/phrases as important, even if each word/phrase did not appear to be strongly associated with the answer. Their evaluation scores tended to accumulate faster than those of the LLMs. This may indicate that trivia experts can make inductive inferences faster (e.g., finding a common concept even from few items). More advanced question-answering systems may be designed by applying trivia experts’ cognitive processes to LLMs’ information processing, and our findings may provide a scaffolding toward such goals.
SN - 1932-6203
ER -