Can dental EHR data support a learning health system? Completeness, consistency, and accuracy of structured versus narrative diagnoses
- 1The University of Texas Health Science Center at Houston
Abstract
Introduction Electronic health record (EHR) data increasingly drive quality monitoring and research, yet their reliability in dentistry is uncertain. We compared the completeness, consistency, and accuracy of periodontal diagnoses captured in two different formats (structured fields versus narrative notes) across two academic dental centers sharing the same EHR platform and diagnostic terminology. Methods In this retrospective study of qualifying periodontal encounters (2021), structured diagnoses were extracted by SQL and narrative diagnoses by a fine-tuned RoBERTa NLP model. A random sample (~20%) was independently reviewed by two board-certified periodontists, with disagreements resolved by consensus, to form the gold standard. Outcomes were completeness (presence of a diagnosis), consistency (agreement between formats), and accuracy (agreement with the gold standard), with ordinal periodontal stage compared using quadratic weighted {kappa}. Results Across 4,542 encounters, completeness varied widely by site and format (34.7% - 84.5%), and neither format was consistently more complete; at one site, over one-third of encounters (35.8%) had no documented diagnosis in either format. Among dual-documented encounters, agreement between formats was substantial but imperfect ({kappa} = 0.86 and 0.80). Accuracy against the gold standard was modest for both sources (all near 50%), with narrative notes numerically higher at both sites and favored by the severity-weighted {kappa}, though raw differences were not significant. Pooled across sites, the formats diverged in 14.8% of dual-documented encounters, 8.4% by two or more stages, with the structured field carrying the milder stage in 67.9% of disagreements. Conclusions Structured and unstructured periodontal diagnoses each matched expert adjudication only about half the time, and neither was consistently more complete. Narrative notes held a small severity-weighted accuracy advantage, and coded fields more often carried the milder stage when the two disagreed. These convergent, cross-site patterns appear systematic rather than site-specific and expose latent risks for analytics and decision support. Improving dental EHR data quality will require workflows that combine the computability of coded entries with the fidelity of free text.
Related
- Reporting Completeness of Biliary Atresia Organoid Studies: A Scoping Review and Structured Audit
- Extracorporeal Membrane Oxygenation (ECMO) Prediction of Mortality and Integrated Complication with Infection Risk Assessment by Machine Learning: Structured Narrative Review.
- Metformin-associated diarrhea in FAERS: a structured pharmacovigilance study with long-term temporal consistency assessment
- Integrating emergency medical services and hospital surge capacity for mass-gathering preparedness: a structured narrative review
- Bias Mitigation Across the Healthcare Artificial Intelligence Lifecycle: A Structured Narrative Review
BibTeX
@misc{chuang2026dental,
title = {{Can dental EHR data support a learning health system? Completeness, consistency, and accuracy of structured versus narrative diagnoses}},
author = {Y.-S. Chuang and G.-H. Lin and C.-T. Lee and R. Brandon and M. F. Walji and X. Jiang and B. Tokede},
howpublished = {medRxiv},
year = {2026},
month = nov,
doi = {10.64898/2026.11.04.26367297},
url = {https://www.medrxiv.org/content/10.64898/2026.11.04.26367297v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Can dental EHR data support a learning health system? Completeness, consistency, and accuracy of structured versus narrative diagnoses
AU - Y.-S. Chuang
AU - G.-H. Lin
AU - C.-T. Lee
AU - R. Brandon
AU - M. F. Walji
AU - X. Jiang
AU - B. Tokede
PY - 2026
DA - 2026/11/04
PB - medRxiv
DO - 10.64898/2026.11.04.26367297
UR - https://www.medrxiv.org/content/10.64898/2026.11.04.26367297v1
AB - Introduction Electronic health record (EHR) data increasingly drive quality monitoring and research, yet their reliability in dentistry is uncertain. We compared the completeness, consistency, and accuracy of periodontal diagnoses captured in two different formats (structured fields versus narrative notes) across two academic dental centers sharing the same EHR platform and diagnostic terminology. Methods In this retrospective study of qualifying periodontal encounters (2021), structured diagnoses were extracted by SQL and narrative diagnoses by a fine-tuned RoBERTa NLP model. A random sample (~20%) was independently reviewed by two board-certified periodontists, with disagreements resolved by consensus, to form the gold standard. Outcomes were completeness (presence of a diagnosis), consistency (agreement between formats), and accuracy (agreement with the gold standard), with ordinal periodontal stage compared using quadratic weighted {kappa}. Results Across 4,542 encounters, completeness varied widely by site and format (34.7% - 84.5%), and neither format was consistently more complete; at one site, over one-third of encounters (35.8%) had no documented diagnosis in either format. Among dual-documented encounters, agreement between formats was substantial but imperfect ({kappa} = 0.86 and 0.80). Accuracy against the gold standard was modest for both sources (all near 50%), with narrative notes numerically higher at both sites and favored by the severity-weighted {kappa}, though raw differences were not significant. Pooled across sites, the formats diverged in 14.8% of dual-documented encounters, 8.4% by two or more stages, with the structured field carrying the milder stage in 67.9% of disagreements. Conclusions Structured and unstructured periodontal diagnoses each matched expert adjudication only about half the time, and neither was consistently more complete. Narrative notes held a small severity-weighted accuracy advantage, and coded fields more often carried the milder stage when the two disagreed. These convergent, cross-site patterns appear systematic rather than site-specific and expose latent risks for analytics and decision support. Improving dental EHR data quality will require workflows that combine the computability of coded entries with the fidelity of free text.
ER -