Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study
27 affiliations
- 1College of Medicine and Public Health, Flinders Health and Medical Research Institute, Flinders University, Flinders University Health and Medical Research Building, Sports Rd, Bedford Park, Adelaide, South Australia, 5042, Australia, +61 8 8201 5647
- 2Science to People, Inc, Lexington, MA, United States
- 3Memorial Sloan Kettering Cancer Center, New York, NY, United States
- 4Schulich School of Medicine and Dentistry, Western University, London, ON, Canada
- 5Cancer Council South Australia, Adelaide, South Australia, Australia
- 6Caring Futures Institute, College of Health and Enablement, Flinders University, Adelaide, South Australia, Australia
- 7Department of Pharmacy Practice and Pharmacotherapeutics, College of Pharmacy, University of Sharjah, Sharjah, United Arab Emirates
- 8School of Pharmacy and Biomedical Science, College of Health, Adelaide University, Adelaide, South Australia, Australia
- 9Flinders Medical Centre, Southern Adelaide Local Health Network, Adelaide, South Australia, Australia
- 10Chandlers Hill Surgery, Adelaide, South Australia, Australia
- 11Lyell McEwin Hospital, Northern Adelaide Local Health Network, Adelaide, South Australia, Australia
- 12Department of Oncology, Sir Peter MacCallum Cancer Centre, Melbourne, Victoria, Australia
- 13Department of Medical Oncology, Peter MacCallum Cancer Centre, Melbourne, Victoria, Australia
- 14South Western Sydney Local Health District, Sydney, New South Wales, Australia
- 15SA Pharmacy, SA Health, Adelaide, South Australia, Australia
- 16Discipline of Pharmacology, Division of Health and Applied Sciences, Faculty of Science, Prince of Songkla University, Hat Yai, Songkhla, Thailand
- 17Research Center in Mathematics and Statistics with Applications, Faculty of Science, Prince of Songkla University, Hat Yai, Songkhla, Thailand
- 18Australian Institute for Machine Learning, Adelaide University, Adelaide, South Australia, Australia
- 19SickKids Research Institute, Toronto, ON, Canada
- 20Women's and Children's Health Network, Adelaide, South Australia, Australia
- 21School of Public Health and Social Work, Queensland University of Technology, Brisbane, Queensland, Australia
- 22School of Medicine, Adelaide University, Adelaide, South Australia, Australia
- 23Department of Clinical Pharmacology, Flinders Medical Centre, Southern Adelaide Local Health Network, Adelaide, South Australia, Australia
- 24Department of Medical Oncology, Flinders Centre for Innovation in Cancer, Flinders University, Adelaide, South Australia, Australia
- 25Division of Public Health Sciences, Fred Hutch Cancer Center, Seattle, WA, United States
- 26Division of Medical Oncology, School of Medicine, Duke University, Durham, NC, United States
- 27Advanced Cancer Research Group, Kirkland, WA, United States
Abstract
Background: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation.
Objective: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts.
Methods: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and refined through iterative feedback from a multidisciplinary expert panel (n=23). The pilot rubric was applied to 4 health care scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to health professional continuing education questions, and automated physician-to-physician clinical documentation generation. Via these scenarios, 15 clinicians (medical practitioners, pharmacists, and medical oncologists) evaluated 250 input-output pairs using domain-level ratings and qualitative comments. The evaluated outputs included 200 AI-generated responses across the 4 health care scenarios and 50 physician-tagged replies in the health-related Reddit questions scenario.
Results: The final rubric, OMNI, comprised 10 evaluation domains (Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness) assessed on 4-point Likert scales, supported by detailed instructional and example materials. Via the 4 pilot scenarios, assessors generated 9600 domain-level ratings and 4010 qualitative comments. The mean pairwise agreement for the 4-point Likert groups ranged from 30.8% to 53.4%, increasing to 59.3% to 79.6% when collapsed into broad agreement categories; Gwet’s AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses between models within scenarios, while recurrent limitations in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy were identified across scenarios. Multimodal image and video outputs had the highest occurrence of negative rating domains. Assessors reported OMNI was usable and indicated they would reuse it.
Conclusions: OMNI is a standardized, accessible rubric for expert-led evaluation of generative AI outputs in health contexts. Pilot application demonstrated that OMNI could differentiate AI performance across models and domains while generating structured feedback. Further work is needed to assess its applicability to purpose-built health AI tools, agentic systems, and organizational evaluation settings.
Related
- AI-Based Grading of Facial Aging Signs and Prediction of Visible Age From Standardized Facial Images: Algorithm Development and Internal Evaluation Study.
- Assessing Person-Specific Symptom Networks From Brief Narratives: Development and Initial Evaluation of the Narrative-to-Network (N2N) Approach
- Knowledge distillation-based lightweight generative large language models for standardized quality control of Chinese radiology reports
- Comparative quality, readability, and factual reliability of four publicly accessible chatbot systems for heart failure patient education: a cross-sectional evaluation
- A Methodology to Evaluate the Efficacy and Patient-Centeredness of a Patient-Facing AI Chatbot: Development and Validation of a Standardized-Patient Method
BibTeX
@article{cornelisse2026development,
title = {{Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study}},
author = {Erik Cornelisse and Dib Kousa and Bradley D Menz and Brinleigh Murphy-Reuter and Adel Shahnam and Benjamin Chin-Yee and Ryan Calabro and Lee X Li and Imogen Ramsey and Ahmad Y Abuhelwa and Jessica M Logan and Huah Shin Ng and Mythily Aravinthan and James Goonan and Hannah Carthew and Paul Molyneux and Joanne Tang and Nadia Hitchen and Martin Hong and Jimit Gandhi and Stephanie Fontanarosa and Nicholas L Scarfo and Kezia Daniel and Sally B Marotti and Joy Mariadas and Bianca van Diermen and Divya Joyce Susai Manickam and Warit Ruanglertboon and David Postma and Melissa D McCradden and Sandeep Reddy and Stephen Bacchi and Arduino A Mangoni and Ross A McKinnon and Ganessan Kichenadasse and Gary H Lyman and Nicole M Kuderer and Michael J Sorich and Andrew Rowland and Ashley M Hopkins},
journal = {Journal of Medical Internet Research},
year = {2026},
month = oct,
doi = {10.2196/100531},
url = {https://doi.org/10.2196/100531},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study
AU - Erik Cornelisse
AU - Dib Kousa
AU - Bradley D Menz
AU - Brinleigh Murphy-Reuter
AU - Adel Shahnam
AU - Benjamin Chin-Yee
AU - Ryan Calabro
AU - Lee X Li
AU - Imogen Ramsey
AU - Ahmad Y Abuhelwa
AU - Jessica M Logan
AU - Huah Shin Ng
AU - Mythily Aravinthan
AU - James Goonan
AU - Hannah Carthew
AU - Paul Molyneux
AU - Joanne Tang
AU - Nadia Hitchen
AU - Martin Hong
AU - Jimit Gandhi
AU - Stephanie Fontanarosa
AU - Nicholas L Scarfo
AU - Kezia Daniel
AU - Sally B Marotti
AU - Joy Mariadas
AU - Bianca van Diermen
AU - Divya Joyce Susai Manickam
AU - Warit Ruanglertboon
AU - David Postma
AU - Melissa D McCradden
AU - Sandeep Reddy
AU - Stephen Bacchi
AU - Arduino A Mangoni
AU - Ross A McKinnon
AU - Ganessan Kichenadasse
AU - Gary H Lyman
AU - Nicole M Kuderer
AU - Michael J Sorich
AU - Andrew Rowland
AU - Ashley M Hopkins
PY - 2026
DA - 2026/10/27
JO - Journal of Medical Internet Research
DO - 10.2196/100531
UR - https://doi.org/10.2196/100531
AB - Background: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation. Objective: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts. Methods: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and refined through iterative feedback from a multidisciplinary expert panel (n=23). The pilot rubric was applied to 4 health care scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to health professional continuing education questions, and automated physician-to-physician clinical documentation generation. Via these scenarios, 15 clinicians (medical practitioners, pharmacists, and medical oncologists) evaluated 250 input-output pairs using domain-level ratings and qualitative comments. The evaluated outputs included 200 AI-generated responses across the 4 health care scenarios and 50 physician-tagged replies in the health-related Reddit questions scenario. Results: The final rubric, OMNI, comprised 10 evaluation domains (Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness) assessed on 4-point Likert scales, supported by detailed instructional and example materials. Via the 4 pilot scenarios, assessors generated 9600 domain-level ratings and 4010 qualitative comments. The mean pairwise agreement for the 4-point Likert groups ranged from 30.8% to 53.4%, increasing to 59.3% to 79.6% when collapsed into broad agreement categories; Gwet’s AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses between models within scenarios, while recurrent limitations in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy were identified across scenarios. Multimodal image and video outputs had the highest occurrence of negative rating domains. Assessors reported OMNI was usable and indicated they would reuse it. Conclusions: OMNI is a standardized, accessible rubric for expert-led evaluation of generative AI outputs in health contexts. Pilot application demonstrated that OMNI could differentiate AI performance across models and domains while generating structured feedback. Further work is needed to assess its applicability to purpose-built health AI tools, agentic systems, and organizational evaluation settings.
SN - 1439-4456
SN - 1438-8871
ER -