Designing an AI Agent That Does Not Tutor: Design Rules for Socratic Probing in Introductory Physics
Abstract
Instructors need to know whether a student understands a concept, not only whether the student can produce a correct answer, and the two have come apart: a student can paste a physics question into a chatbot and submit a fluent, correct answer without understanding it. We built a conversational AI agent for introductory mechanics that only probes. It is designed as an instrument that a correct answer alone should not pass. Its key measure is cave detection: identifying a correct first idea that the student abandons when a single variable of the scenario is changed. A single right-or-wrong item cannot see this. The agent asks a student to commit to a prediction about an everyday situation, explain it without physics vocabulary, and defend that explanation against a changed case built from the student's own words. It does not teach, hint, confirm, or correct, and it reports nothing until the end. Such an instrument works only if its replies give nothing away. LLM agents tend to leak evaluation: praise, word choice, reply length, and pacing reveal whether an answer was right, and a student who knows stops reasoning. We report design rules that suppress this leakage, derived from eight versions of the agent and more than 40 scripted simulation runs. The obvious fixes often made leakage worse: listing forbidden praise phrases increased praise, and a single worked example in the prompt spread into unrelated conversations. Each changed case had to alter exactly one variable; otherwise a changed answer could not be traced to a single idea. The evidence is simulation evidence only: no real students have used the agent, and the author wrote both the personas and the scoring rules. This paper reports design findings, not learning outcomes.
Related
- Calculating a journey to Mars with introductory physics
- Data-driven modeling in the introductory physics laboratory: Scaling analysis and data collapse in the specific heat of water experiment
- Mechanics Cognitive Diagnostic: Testing Fine-Grained Learning Objectives in Introductory Physics
- Active Learning with Generative AI in Introductory Physics: A Multi-Institutional Study of the P-A.I.R. Framework
- Adapting Public Exams for Introductory Physics: A Mixed-Methods Investigation of Student Outcomes
BibTeX
@misc{cho2026designing,
title = {{Designing an AI Agent That Does Not Tutor: Design Rules for Socratic Probing in Introductory Physics}},
author = {Byungmoon Cho},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.22949},
eprint = {2610.22949},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.22949},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Designing an AI Agent That Does Not Tutor: Design Rules for Socratic Probing in Introductory Physics
AU - Byungmoon Cho
PY - 2026
DA - 2026/10/16
PB - arXiv
DO - 10.48550/arxiv.2610.22949
UR - https://arxiv.org/abs/2610.22949
AB - Instructors need to know whether a student understands a concept, not only whether the student can produce a correct answer, and the two have come apart: a student can paste a physics question into a chatbot and submit a fluent, correct answer without understanding it. We built a conversational AI agent for introductory mechanics that only probes. It is designed as an instrument that a correct answer alone should not pass. Its key measure is cave detection: identifying a correct first idea that the student abandons when a single variable of the scenario is changed. A single right-or-wrong item cannot see this. The agent asks a student to commit to a prediction about an everyday situation, explain it without physics vocabulary, and defend that explanation against a changed case built from the student's own words. It does not teach, hint, confirm, or correct, and it reports nothing until the end. Such an instrument works only if its replies give nothing away. LLM agents tend to leak evaluation: praise, word choice, reply length, and pacing reveal whether an answer was right, and a student who knows stops reasoning. We report design rules that suppress this leakage, derived from eight versions of the agent and more than 40 scripted simulation runs. The obvious fixes often made leakage worse: listing forbidden praise phrases increased praise, and a single worked example in the prompt spread into unrelated conversations. Each changed case had to alter exactly one variable; otherwise a changed answer could not be traced to a single idea. The evidence is simulation evidence only: no real students have used the agent, and the author wrote both the personas and the scoring rules. This paper reports design findings, not learning outcomes.
ER -