Interpreting Self-Reported Outcomes After AI-Assisted Writing Workshops: An Aggregate Sensitivity Analysis of Educational Claims
- 1Concordia University
Abstract
Artificial intelligence writing workshops are increasingly used in health-professions education, but immediate self-reports can be interpreted as evidence of learning or ethical competence beyond what they measure. This study applied an aggregate-data interpretation framework to a published single-institution evaluation of a hands-on AI-assisted academic-writing workshop. Public counts and item summaries were classified by construct, summarized with binomial working-model intervals for observed respondents, and reanalyzed with complete registered-list identification intervals that assigned every questionnaire nonrespondent first to the non-positive and then to the positive category. Among 64 respondents, 54 reported improved understanding, or 84.4 percent, with a working-model interval of 73.6 percent to 91.3 percent. Confidence and perceived ethical improvement were each reported by 45 respondents, or 70.3 percent, with intervals of 58.2 percent to 80.1 percent. Across 80 registrants, unrestricted nonresponse yielded identification intervals of 67.5 percent to 87.5 percent for perceived improvement and 56.2 percent to 76.2 percent for both confidence and ethical improvement. Positive delivery ratings were common. However, these measures did not assess objective knowledge, writing quality, source verification, ethical decision-making, or sustained practice. The evidence supports bounded descriptions of respondent experience only; effectiveness claims require linked objective assessment and delayed follow-up.
Related
- Interpreting imported malaria genomes: population-genomic sensitivity analyses of West Africa-associated Plasmodium falciparum isolates detected in China
- Self-Reported AI Competency and Educational Needs for Integrated AI and Digital Rehabilitation Education Among Occupational Therapy Students: A Cross-Sectional Survey.
- Interpreting changes in reported case fatality in endemic zoonoses: an analytical framework and ecological analysis of Lassa fever surveillance in Nigeria, 2017-26
- Evaluating the impact of a gambling harm focused educational intervention on medical students' self-reported confidence: a non-randomised, single-group pre-post study
- Feasibility and Preliminary Educational Outcomes of Self-Reported Clinical Confidence After a Global Health Experience: A Pilot Study.
BibTeX
@misc{mercer2026interpreting,
title = {{Interpreting Self-Reported Outcomes After AI-Assisted Writing Workshops: An Aggregate Sensitivity Analysis of Educational Claims}},
author = {L. Mercer and M. K. Reed and H. J. Tran and A. J. Garnier},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.29.26366847},
url = {https://www.medrxiv.org/content/10.64898/2026.10.29.26366847v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Interpreting Self-Reported Outcomes After AI-Assisted Writing Workshops: An Aggregate Sensitivity Analysis of Educational Claims
AU - L. Mercer
AU - M. K. Reed
AU - H. J. Tran
AU - A. J. Garnier
PY - 2026
DA - 2026/10/29
PB - medRxiv
DO - 10.64898/2026.10.29.26366847
UR - https://www.medrxiv.org/content/10.64898/2026.10.29.26366847v1
AB - Artificial intelligence writing workshops are increasingly used in health-professions education, but immediate self-reports can be interpreted as evidence of learning or ethical competence beyond what they measure. This study applied an aggregate-data interpretation framework to a published single-institution evaluation of a hands-on AI-assisted academic-writing workshop. Public counts and item summaries were classified by construct, summarized with binomial working-model intervals for observed respondents, and reanalyzed with complete registered-list identification intervals that assigned every questionnaire nonrespondent first to the non-positive and then to the positive category. Among 64 respondents, 54 reported improved understanding, or 84.4 percent, with a working-model interval of 73.6 percent to 91.3 percent. Confidence and perceived ethical improvement were each reported by 45 respondents, or 70.3 percent, with intervals of 58.2 percent to 80.1 percent. Across 80 registrants, unrestricted nonresponse yielded identification intervals of 67.5 percent to 87.5 percent for perceived improvement and 56.2 percent to 76.2 percent for both confidence and ethical improvement. Positive delivery ratings were common. However, these measures did not assess objective knowledge, writing quality, source verification, ethical decision-making, or sustained practice. The evidence supports bounded descriptions of respondent experience only; effectiveness claims require linked objective assessment and delayed follow-up.
ER -