{ "abstract": "Background: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation.\n\nObjective: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts.\n\nMethods: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and refined through iterative feedback from a multidisciplinary expert panel (n=23). The pilot rubric was applied to 4 health care scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to health professional continuing education questions, and automated physician-to-physician clinical documentation generation. Via these scenarios, 15 clinicians (medical practitioners, pharmacists, and medical oncologists) evaluated 250 input-output pairs using domain-level ratings and qualitative comments. The evaluated outputs included 200 AI-generated responses across the 4 health care scenarios and 50 physician-tagged replies in the health-related Reddit questions scenario.\n\nResults: The final rubric, OMNI, comprised 10 evaluation domains (Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness) assessed on 4-point Likert scales, supported by detailed instructional and example materials. Via the 4 pilot scenarios, assessors generated 9600 domain-level ratings and 4010 qualitative comments. The mean pairwise agreement for the 4-point Likert groups ranged from 30.8% to 53.4%, increasing to 59.3% to 79.6% when collapsed into broad agreement categories; Gwet’s AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses between models within scenarios, while recurrent limitations in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy were identified across scenarios. Multimodal image and video outputs had the highest occurrence of negative rating domains. Assessors reported OMNI was usable and indicated they would reuse it.\n\nConclusions: OMNI is a standardized, accessible rubric for expert-led evaluation of generative AI outputs in health contexts. Pilot application demonstrated that OMNI could differentiate AI performance across models and domains while generating structured feedback. Further work is needed to assess its applicability to purpose-built health AI tools, agentic systems, and organizational evaluation settings.", "abstract_shown": true, "affiliations": [ "College of Medicine and Public Health, Flinders Health and Medical Research Institute, Flinders University, Flinders University Health and Medical Research Building, Sports Rd, Bedford Park, Adelaide, South Australia, 5042, Australia, +61 8 8201 5647", "Science to People, Inc, Lexington, MA, United States", "Memorial Sloan Kettering Cancer Center, New York, NY, United States", "Schulich School of Medicine and Dentistry, Western University, London, ON, Canada", "Cancer Council South Australia, Adelaide, South Australia, Australia", "Caring Futures Institute, College of Health and Enablement, Flinders University, Adelaide, South Australia, Australia", "Department of Pharmacy Practice and Pharmacotherapeutics, College of Pharmacy, University of Sharjah, Sharjah, United Arab Emirates", "School of Pharmacy and Biomedical Science, College of Health, Adelaide University, Adelaide, South Australia, Australia", "Flinders Medical Centre, Southern Adelaide Local Health Network, Adelaide, South Australia, Australia", "Chandlers Hill Surgery, Adelaide, South Australia, Australia", "Lyell McEwin Hospital, Northern Adelaide Local Health Network, Adelaide, South Australia, Australia", "Department of Oncology, Sir Peter MacCallum Cancer Centre, Melbourne, Victoria, Australia", "Department of Medical Oncology, Peter MacCallum Cancer Centre, Melbourne, Victoria, Australia", "South Western Sydney Local Health District, Sydney, New South Wales, Australia", "SA Pharmacy, SA Health, Adelaide, South Australia, Australia", "Discipline of Pharmacology, Division of Health and Applied Sciences, Faculty of Science, Prince of Songkla University, Hat Yai, Songkhla, Thailand", "Research Center in Mathematics and Statistics with Applications, Faculty of Science, Prince of Songkla University, Hat Yai, Songkhla, Thailand", "Australian Institute for Machine Learning, Adelaide University, Adelaide, South Australia, Australia", "SickKids Research Institute, Toronto, ON, Canada", "Women's and Children's Health Network, Adelaide, South Australia, Australia", "School of Public Health and Social Work, Queensland University of Technology, Brisbane, Queensland, Australia", "School of Medicine, Adelaide University, Adelaide, South Australia, Australia", "Department of Clinical Pharmacology, Flinders Medical Centre, Southern Adelaide Local Health Network, Adelaide, South Australia, Australia", "Department of Medical Oncology, Flinders Centre for Innovation in Cancer, Flinders University, Adelaide, South Australia, Australia", "Division of Public Health Sciences, Fred Hutch Cancer Center, Seattle, WA, United States", "Division of Medical Oncology, School of Medicine, Duke University, Durham, NC, United States", "Advanced Cancer Research Group, Kirkland, WA, United States" ], "article_type": "research article", "arxiv": null, "author_total": 40, "authors": [ { "affs": [ 0 ], "name": "Erik Cornelisse", "orcid": "0009-0009-4588-2688" }, { "affs": [ 0 ], "name": "Dib Kousa", "orcid": "0009-0003-3941-4446" }, { "affs": [ 0 ], "name": "Bradley D Menz", "orcid": "0000-0002-0855-5081" }, { "affs": [ 1 ], "name": "Brinleigh Murphy-Reuter", "orcid": "0009-0003-2491-0105" }, { "affs": [ 2 ], "name": "Adel Shahnam", "orcid": "0000-0002-0043-8374" }, { "affs": [ 3 ], "name": "Benjamin Chin-Yee", "orcid": "0000-0003-0737-3603" }, { "affs": [ 4 ], "name": "Ryan Calabro", "orcid": "0000-0001-9082-0472" }, { "affs": [ 0 ], "name": "Lee X Li", "orcid": "0000-0001-5259-5198" }, { "affs": [ 5 ], "name": "Imogen Ramsey", "orcid": "0000-0002-8757-1572" }, { "affs": [ 6 ], "name": "Ahmad Y Abuhelwa", "orcid": "0000-0002-4182-065X" }, { "affs": [ 7 ], "name": "Jessica M Logan", "orcid": "0000-0003-4046-6908" }, { "affs": [ 0 ], "name": "Huah Shin Ng", "orcid": "0000-0001-8381-5253" }, { "affs": [ 8 ], "name": "Mythily Aravinthan", "orcid": "0000-0002-2335-8553" }, { "affs": [ 8 ], "name": "James Goonan", "orcid": "0009-0007-4010-0937" }, { "affs": [ 8 ], "name": "Hannah Carthew", "orcid": "0009-0009-4878-117X" }, { "affs": [ 9 ], "name": "Paul Molyneux", "orcid": "0009-0003-5079-1728" }, { "affs": [ 10 ], "name": "Joanne Tang", "orcid": "0009-0002-7530-9953" }, { "affs": [ 11, 12 ], "name": "Nadia Hitchen", "orcid": "0000-0001-9327-4171" }, { "affs": [ 13 ], "name": "Martin Hong", "orcid": "0000-0002-2327-4261" }, { "affs": [ 7, 14 ], "name": "Jimit Gandhi", "orcid": "0000-0001-6985-2082" }, { "affs": [ 14 ], "name": "Stephanie Fontanarosa", "orcid": "0009-0004-6618-5869" }, { "affs": [ 14 ], "name": "Nicholas L Scarfo", "orcid": "0000-0002-4344-9284" }, { "affs": [ 14 ], "name": "Kezia Daniel", "orcid": "0009-0001-0106-954X" }, { "affs": [ 0, 7, 14 ], "name": "Sally B Marotti", "orcid": "0000-0003-3098-6096" }, { "affs": [ 0 ], "name": "Joy Mariadas", "orcid": "0009-0007-8136-4920" }, { "affs": [ 9 ], "name": "Bianca van Diermen", "orcid": "0009-0000-1693-3292" }, { "affs": [ 8 ], "name": "Divya Joyce Susai Manickam", "orcid": "0009-0009-5019-7255" }, { "affs": [ 15, 16 ], "name": "Warit Ruanglertboon", "orcid": "0000-0003-0540-7427" }, { "affs": [ 0 ], "name": "David Postma", "orcid": "0009-0001-1054-762X" }, { "affs": [ 17, 18, 19 ], "name": "Melissa D McCradden", "orcid": "0000-0002-6476-2165" }, { "affs": [ 20 ], "name": "Sandeep Reddy", "orcid": "0000-0002-5824-4900" }, { "affs": [ 0, 21 ], "name": "Stephen Bacchi", "orcid": "0000-0001-5130-8628" }, { "affs": [ 0, 22 ], "name": "Arduino A Mangoni", "orcid": "0000-0001-8699-1412" }, { "affs": [ 0 ], "name": "Ross A McKinnon", "orcid": "0000-0002-3725-793X" }, { "affs": [ 0, 23 ], "name": "Ganessan Kichenadasse", "orcid": "0000-0001-9923-5149" }, { "affs": [ 24, 25 ], "name": "Gary H Lyman", "orcid": "0000-0002-0823-8086" }, { "affs": [ 26 ], "name": "Nicole M Kuderer", "orcid": "0000-0002-9263-8540" }, { "affs": [ 0 ], "name": "Michael J Sorich", "orcid": "0000-0003-1999-866X" }, { "affs": [ 0 ], "name": "Andrew Rowland", "orcid": "0000-0002-8946-3954" }, { "affs": [ 0 ], "name": "Ashley M Hopkins", "orcid": "0000-0001-7652-4378" } ], "bibtex": "https://research.draconitic.com/paper/cmv5cxs5zdp6x.bib", "cited_by": 0, "doi": "10.2196/100531", "fulltext_url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13712112/", "id": "cmv5cxs5zdp6x", "issns": [ "1439-4456", "1438-8871" ], "landing_url": "https://doi.org/10.2196/100531", "language": "en", "licence": "CC-BY-4.0", "licence_label": "CC BY 4.0", "licence_source": "article", "licence_url": "https://creativecommons.org/licenses/by/4.0/", "metadata_licence": null, "native_subjects": [ [ "heading", "Artificial Intelligence" ], [ "heading", "Digital Health, Telehealth and e-Innovation in Clinical Settings" ], [ "heading", "Generative Language Models Including ChatGPT" ], [ "heading", "New Methods and Approaches in Medical Education" ], [ "heading", "Original Paper" ], [ "openalex", "2718|Artificial Intelligence in Healthcare and Education" ] ], "openalex": "W7223823744", "pdf_url": null, "pmcid": "PMC13712112", "pmid": "42944564", "published": "2026-10-27", "publisher": "JMIR Publications Inc.", "ris": "https://research.draconitic.com/paper/cmv5cxs5zdp6x.ris", "sources": [ { "first_seen": "2026-10-31T12:22:45Z", "native_id": "PMC13712112", "source": "pmc", "url": "https://doi.org/10.2196/100531" } ], "subjects": [ { "field": "medicine", "origin": "openalex:2718|Artificial Intelligence in Healthcare and Education", "subfield": "epidemiology-public-health" } ], "title": "Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study", "title_text": "Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study", "type": "peer-reviewed", "updated_at": "2026-10-31T12:22:45Z", "url": "https://research.draconitic.com/paper/cmv5cxs5zdp6x", "venue": "Journal of Medical Internet Research", "versions": [] }