Skip to content
DraconiticResearch
Search Sources API
203,203 studies31 sourcesUpdated 55 min ago
Peer-reviewedCC BY 4.0JMIR Medical Informatics

Evaluating Internet-Based AI Chatbots as Unregulated Digital Health Resources in Dental Emergency Management: Comparative Multilingual Study

  • Ahmed Bashah1
  • Quan Huixin1
  • Hamza Zaawari2
  • Ali Al-Waqeerah3
  • Eslam Ghaleb4
  • Abdulkhaleq Salem5
  • Natheer Wahan5
  • Gang Chen1
5 affiliations
  1. 1Department of Stomatology, The First Affiliated Hospital of Dalian Medical University, No.222, Zhongshan Road, Dalian, Liaoning, 116011, China, +86 18098873280
  2. 2College of Dentistry, Hodeidah University, Al Hudaydah, Yemen
  3. 3Department of Respiratory, The First Affiliated Hospital of Dalian Medical University, Dalian, Liaoning, China
  4. 4Department of Biochemistry and Molecular Biology, Dalian Medical University, Dalian, Liaoning, China
  5. 5Department of Pharmacology, Dalian Medical University, Dalian, Liaoning, China
Read full textDOICite

Abstract

Background: Publicly accessible, internet-based AI chatbots have emerged as unregulated digital health resources that clinicians and patients consult in real time, including during acute medical emergencies. In dental practice—where life-threatening events such as anaphylaxis, cardiac arrest, and severe hypoglycemia can unfold within seconds—the unsupervised use of these tools as point-of-care decision aids poses uncharacterized patient safety risks. Despite widespread deployment, no study has systematically evaluated the accuracy, temporal consistency, or multilingual reliability of contemporary large language model (LLM)–based chatbots specifically for dental medical emergency management.

Objective: This study aimed to evaluate the accuracy, temporal stability, and multilingual performance of five contemporary LLM-based chatbots when queried on the management of medical emergencies in dental practice.

Methods: A comparative study was conducted to assess five AI chatbots: ChatGPT-5.2 (OpenAI), Gemini 3.1 (Google), DeepSeek-R1 (DeepSeek AI), Grok 4.20 (xAI), and Kimi K2.5 (Moonshot AI). A total of 130 questions covering dental medical emergency management were developed and categorized into four formats: multiple-choice questions (MCQs), yes-or-no questions, open-ended WH questions, and clinical scenario questions. Each question was presented using two prompting strategies (basic and advanced) across three languages (English, Arabic, and Chinese). Data collection was performed over seven consecutive days, generating 27,300 chatbot responses. Responses were evaluated by language-specific expert reviewers and classified as correct or incorrect based on established clinical references. Statistical analysis was performed using chi-square tests to assess associations between accuracy and study variables. Pairwise post hoc comparisons of column proportions were conducted using z-tests with Bonferroni correction.

Results: A total of 27,300 responses were analyzed; 25,377 (92.96%) were correct. Question type significantly influenced chatbot performance (P<.001), with MCQs demonstrating the highest accuracy (5942/6300, 94.3%), and WH questions demonstrating the lowest accuracy (7686/8400, 91.5%). Language also showed a significant association with accuracy (P=.01), with English prompts achieving the highest accuracy (8512/9100, 93.5%). Chatbot model performance differed significantly (P<.001), with Gemini 3.1 demonstrating the highest accuracy (5176/5460, 94.8%; 95% CI 94.21%‐95.39%), followed by ChatGPT-5.2 (5105/5460, 93.5%; 95% CI 92.85%‐94.15%) and Kimi K2.5 (5071/5460, 92.9%). Prompt complexity and testing day did not significantly affect response accuracy (P=.69 and P=.96, respectively).

Conclusions: Current LLM-based chatbots demonstrate high accuracy in answering structured questions related to dental medical emergency management. However, performance varies according to question format and language, and the presence of incorrect responses highlights potential safety concerns. AI chatbots may serve as educational tools, but their use as real-time decision support during dental medical emergencies should be approached with caution.

Related

  1. Evaluating disparities in counseling and resources for chemotherapy-induced hair loss: qualitative analysis of a survey-based approach.
    Lily Kaufman, Aliza Khuhro, Lilia Valentic +3
    Supportive care in cancer : official journal of the Multinational Association of Supportive Care in CancerPeer-reviewedCC BYFull textGeneral medicineEurope PMC
  2. National assessment of trauma care system capacity in Ecuador: Evaluating institutional resources and practitioner KAPS domains
    Doris Sarmiento-Altamirano, Edison Angamarca-Angamarca, Juan Carlos Salamea +2
    PLOS Global Public HealthPeer-reviewedCC BYFull textNeuroscienceMedical imaging & radiologyPLOSEurope PMC
  3. Evaluating a Clinical Decision Support System for Chronic Pain Management: A Per-Protocol Analysis
    E. McCord, J. Blackburn, A. M. Holmes +4
    medRxivPreprintAll rights reservedFull textGeneral medicinemedRxiv
  4. Evaluating a Modified American Society of Anesthesiologists Physical Status‐Based Sedation Management System for Pancreatobiliary Endoscopy: A Prospective Feasibility Study
    Taro Hanaoka, Kosuke Okuwaki, Masafumi Watanabe +6
    DEN openPeer-reviewedCC BYFull textGeneral medicineEurope PMC
  5. Evaluating large language models in patient education: a comparative analysis addressing frequently asked questions in peri-acetabular osteotomy
    TP Davis, B Guevel, K Logishetty +2
    Annals of The Royal College of Surgeons of EnglandPeer-reviewedCC BY 4.0Full text1 citedEpidemiology & public healthPMC

Cite

BibTeXRIS
BibTeX
@article{bashah2026evaluating,
  title = {{Evaluating Internet-Based AI Chatbots as Unregulated Digital Health Resources in Dental Emergency Management: Comparative Multilingual Study}},
  author = {Ahmed Bashah and Quan Huixin and Hamza Zaawari and Ali Al-Waqeerah and Eslam Ghaleb and Abdulkhaleq Salem and Natheer Wahan and Gang Chen},
  journal = {JMIR Medical Informatics},
  year = {2026},
  month = oct,
  doi = {10.2196/96731},
  url = {https://doi.org/10.2196/96731},
  note = {CC BY 4.0},
}
RIS
TY  - JOUR
TI  - Evaluating Internet-Based AI Chatbots as Unregulated Digital Health Resources in Dental Emergency Management: Comparative Multilingual Study
AU  - Ahmed Bashah
AU  - Quan Huixin
AU  - Hamza Zaawari
AU  - Ali Al-Waqeerah
AU  - Eslam Ghaleb
AU  - Abdulkhaleq Salem
AU  - Natheer Wahan
AU  - Gang Chen
PY  - 2026
DA  - 2026/10/27
JO  - JMIR Medical Informatics
DO  - 10.2196/96731
UR  - https://doi.org/10.2196/96731
AB  - Background: Publicly accessible, internet-based AI chatbots have emerged as unregulated digital health resources that clinicians and patients consult in real time, including during acute medical emergencies. In dental practice—where life-threatening events such as anaphylaxis, cardiac arrest, and severe hypoglycemia can unfold within seconds—the unsupervised use of these tools as point-of-care decision aids poses uncharacterized patient safety risks. Despite widespread deployment, no study has systematically evaluated the accuracy, temporal consistency, or multilingual reliability of contemporary large language model (LLM)–based chatbots specifically for dental medical emergency management.  Objective: This study aimed to evaluate the accuracy, temporal stability, and multilingual performance of five contemporary LLM-based chatbots when queried on the management of medical emergencies in dental practice.  Methods: A comparative study was conducted to assess five AI chatbots: ChatGPT-5.2 (OpenAI), Gemini 3.1 (Google), DeepSeek-R1 (DeepSeek AI), Grok 4.20 (xAI), and Kimi K2.5 (Moonshot AI). A total of 130 questions covering dental medical emergency management were developed and categorized into four formats: multiple-choice questions (MCQs), yes-or-no questions, open-ended WH questions, and clinical scenario questions. Each question was presented using two prompting strategies (basic and advanced) across three languages (English, Arabic, and Chinese). Data collection was performed over seven consecutive days, generating 27,300 chatbot responses. Responses were evaluated by language-specific expert reviewers and classified as correct or incorrect based on established clinical references. Statistical analysis was performed using chi-square tests to assess associations between accuracy and study variables. Pairwise post hoc comparisons of column proportions were conducted using z-tests with Bonferroni correction.  Results: A total of 27,300 responses were analyzed; 25,377 (92.96%) were correct. Question type significantly influenced chatbot performance (P<.001), with MCQs demonstrating the highest accuracy (5942/6300, 94.3%), and WH questions demonstrating the lowest accuracy (7686/8400, 91.5%). Language also showed a significant association with accuracy (P=.01), with English prompts achieving the highest accuracy (8512/9100, 93.5%). Chatbot model performance differed significantly (P<.001), with Gemini 3.1 demonstrating the highest accuracy (5176/5460, 94.8%; 95% CI 94.21%‐95.39%), followed by ChatGPT-5.2 (5105/5460, 93.5%; 95% CI 92.85%‐94.15%) and Kimi K2.5 (5071/5460, 92.9%). Prompt complexity and testing day did not significantly affect response accuracy (P=.69 and P=.96, respectively).  Conclusions: Current LLM-based chatbots demonstrate high accuracy in answering structured questions related to dental medical emergency management. However, performance varies according to question format and language, and the presence of incorrect responses highlights potential safety concerns. AI chatbots may serve as educational tools, but their use as real-time decision support during dental medical emergencies should be approached with caution.
SN  - 2291-9694
ER  -