Comparative Evaluation of a System One Model and a General-Purpose Large Language Model on the Korean Physical Therapist Licensing Examination
- 1Yonsei University - Mirae Campus
Abstract
BackgroundSystem One Model is designed for structured decision-making and can provide probabilistic outputs, but their performance in domain-specific physical therapy tasks has not been established.
ObjectiveTo evaluate the performance and potential utility of Jev, a System One Model, for professional knowledge-based selection tasks in physical therapy by comparing it with GPT-4o.
MethodsA total of 380 multiple-choice questions from the 2024 and 2025 Korean Physical Therapist Licensing Examinations were independently submitted to Jev and GPT-4o. The outcomes included answer accuracy, subgroup performance, output validity, API response latency, token usage, estimated API cost, and Jevs prediction uncertainty. A retrospective probability-based model-cascading simulation was also performed.
ResultsJev correctly answered 282 questions (74.2%), whereas GPT-4o answered 326 (85.8%), a difference of -11.6 percentage points. Jev produced valid structured responses for all questions, whereas GPT-4o produced three invalid responses. Jevs selected-answer probabilities discriminated correct from incorrect answers (AUROC = 0.866), and accuracy reached 99.3% when probabilities were greater than or equal to 0.90. The median API response latency was 0.94 s for Jev and 1.06 s for GPT-4o, with estimated costs of US$0.008 and US$0.161, respectively. At a retrospective cascading threshold of 0.70, accuracy was 86.6% with 175 simulated GPT-4o calls.
ConclusionJev showed lower overall accuracy than GPT-4o but provided reliable structured outputs, informative prediction probabilities, lower latency, and lower estimated cost. Its probabilistic outputs warrant further prospective evaluation for selective model escalation and decision-support workflows in physical therapy.
Related
- Benchmarking large language models on the Ukrainian Krok 2 licensing examination in physical therapy: accuracy, consistency, and agreement
- Research Preparedness Among Osteopathic Medical Students in the Post-Pass/Fail Licensing Examination Era: Bridging the Gap Between Rising Expectations and Student Confidence
- Performance of Contemporary Large Language Models on the National Prosthodontic Resident Examination: A Multi-Year Comparative Study
- The lived experiences of diploma nurses who did not pass their professional licensing examination on the first attempt: A qualitative phenomenological study
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
BibTeX
@misc{kim2026comparative,
title = {{Comparative Evaluation of a System One Model and a General-Purpose Large Language Model on the Korean Physical Therapist Licensing Examination}},
author = {J.-h. Kim},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.26.26364067},
url = {https://www.medrxiv.org/content/10.64898/2026.09.26.26364067v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Comparative Evaluation of a System One Model and a General-Purpose Large Language Model on the Korean Physical Therapist Licensing Examination
AU - J.-h. Kim
PY - 2026
DA - 2026/09/26
PB - medRxiv
DO - 10.64898/2026.09.26.26364067
UR - https://www.medrxiv.org/content/10.64898/2026.09.26.26364067v1
AB - BackgroundSystem One Model is designed for structured decision-making and can provide probabilistic outputs, but their performance in domain-specific physical therapy tasks has not been established. ObjectiveTo evaluate the performance and potential utility of Jev, a System One Model, for professional knowledge-based selection tasks in physical therapy by comparing it with GPT-4o. MethodsA total of 380 multiple-choice questions from the 2024 and 2025 Korean Physical Therapist Licensing Examinations were independently submitted to Jev and GPT-4o. The outcomes included answer accuracy, subgroup performance, output validity, API response latency, token usage, estimated API cost, and Jevs prediction uncertainty. A retrospective probability-based model-cascading simulation was also performed. ResultsJev correctly answered 282 questions (74.2%), whereas GPT-4o answered 326 (85.8%), a difference of -11.6 percentage points. Jev produced valid structured responses for all questions, whereas GPT-4o produced three invalid responses. Jevs selected-answer probabilities discriminated correct from incorrect answers (AUROC = 0.866), and accuracy reached 99.3% when probabilities were greater than or equal to 0.90. The median API response latency was 0.94 s for Jev and 1.06 s for GPT-4o, with estimated costs of US$0.008 and US$0.161, respectively. At a retrospective cascading threshold of 0.70, accuracy was 86.6% with 175 simulated GPT-4o calls. ConclusionJev showed lower overall accuracy than GPT-4o but provided reliable structured outputs, informative prediction probabilities, lower latency, and lower estimated cost. Its probabilistic outputs warrant further prospective evaluation for selective model escalation and decision-support workflows in physical therapy.
ER -