From Divergent Guidelines to Delphi-Ready Items: Tutorial for Large Language Model-Assisted Guideline Harmonization Using Penile Cancer as a Worked Example.
6 affiliations
- 1Department for Urology, University Hospital Regensburg, Caritas Krankenhaus St. Josef, Landshuter Str. 65, Regensburg, Bavaria, 93053, Germany, 49 941782 ext 3506.
- 2Department of Urology, University Hospital Augsburg, Augsburg, Bavaria, Germany.
- 3Department of Urology, Barmherzige Brüder Klinikum St. Elisabeth in Straubing, Straubing, Bavaria, Germany.
- 4Department of Epidemiology and Preventive Medicine, Medical Sociology, University of Regensburg, Regensburg, Bavaria, Germany.
- 5Department of Urology, Ludwig-Maximilians-Universität München, Munich, Bavaria, Germany.
- 6Faculty of Information Science, University of Regensburg, Regensburg, Germany.
Abstract
Background
Clinical practice guidelines are central instruments of evidence-based care; yet, in many domains, particularly rare or complex diseases, multiple guidelines coexist and frequently offer conflicting advice. Existing strategies for resolving such divergences are often labor-intensive, opaque, and insufficiently reproducible.Objective
This tutorial aimed to develop and demonstrate a reproducible workflow for transforming divergent clinical guideline recommendations and literature-derived white areas into human-reviewed Delphi-ready candidate items.Methods
Following a step-by-step tutorial structure, we demonstrate the workflow using penile cancer as a worked example. The process comprises manual selection of clinical guidelines, a systematic PubMed search for literature-derived candidate issues potentially unaddressed by the selected guidelines, selection of 3 complementary large language models (LLMs), and execution of predefined prompts, multimodal recommendation extraction, semantic harmonization, and item-level guideline stance classification. The evaluative process combines a 3-LLM ensemble consistency check with an independent pre-Delphi appraisal by 4 penile cancer experts using a predefined 5-point suitability scale and acceptance threshold.Results
In the worked example in the field of penile cancer as a high-stakes use case, the workflow extracted 157 clinical recommendations from 5 guidelines and consolidated them into 68 harmonized guideline-derived items. Stance classification identified 15 uniformly supported items and 53 divergent, ambiguous, or incompletely addressed items. In parallel, literature processing generated 95 harmonized white-area items, of which 9 were classified as true white-area items not addressed by the included guidelines. The final methodological output was a set of 62 human-reviewed Delphi-ready candidate items for subsequent clinical consensus voting.Conclusions
This methodology provides a scalable, transparent, and transferable approach for identifying and structuring interguideline inconsistencies and evidence gaps. By combining LLM-assisted analysis with independent external expert appraisal, it supports the reproducible transformation of heterogeneous guideline content into Delphi-ready candidate items.
Related
- Syndrome-oriented CNS infectious complications after hematopoietic stem-cell transplantation and CAR T-cell therapy, part 1—viral encephalitis, meningitis, and myelitis: recommendations from the Practice Harmonization & Guidelines Committee of the EBMT
- Towards global clinical practice guidelines for the management of non-specific low back pain in primary care: a review of current guideline recommendations and how they have changed over the last 30 years
- Effect of guideline-assisted prompting on large language model performance in medication-related osteonecrosis of the jaw–related endodontic decision-making
- Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
- Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome
BibTeX
@article{haas2026from,
title = {{From Divergent Guidelines to Delphi-Ready Items: Tutorial for Large Language Model-Assisted Guideline Harmonization Using Penile Cancer as a Worked Example.}},
author = {Maximilian Haas and Emily Rinderknecht and Matthias May and Maximilian Burger and Christian Gilfrich and Anne Herrmann and Sabine Brookman-May and Christoph Alexander Eckl and Udo Kruschwitz and Samy Ateia and Gregor Donabauer},
journal = {Journal of medical Internet research},
year = {2026},
month = oct,
doi = {10.2196/96886},
url = {https://europepmc.org/article/MED/42902842},
note = {CC BY},
}
RIS
TY - JOUR
TI - From Divergent Guidelines to Delphi-Ready Items: Tutorial for Large Language Model-Assisted Guideline Harmonization Using Penile Cancer as a Worked Example.
AU - Maximilian Haas
AU - Emily Rinderknecht
AU - Matthias May
AU - Maximilian Burger
AU - Christian Gilfrich
AU - Anne Herrmann
AU - Sabine Brookman-May
AU - Christoph Alexander Eckl
AU - Udo Kruschwitz
AU - Samy Ateia
AU - Gregor Donabauer
PY - 2026
DA - 2026/10/20
JO - Journal of medical Internet research
DO - 10.2196/96886
UR - https://europepmc.org/article/MED/42902842
AB - Background Clinical practice guidelines are central instruments of evidence-based care; yet, in many domains, particularly rare or complex diseases, multiple guidelines coexist and frequently offer conflicting advice. Existing strategies for resolving such divergences are often labor-intensive, opaque, and insufficiently reproducible.Objective This tutorial aimed to develop and demonstrate a reproducible workflow for transforming divergent clinical guideline recommendations and literature-derived white areas into human-reviewed Delphi-ready candidate items.Methods Following a step-by-step tutorial structure, we demonstrate the workflow using penile cancer as a worked example. The process comprises manual selection of clinical guidelines, a systematic PubMed search for literature-derived candidate issues potentially unaddressed by the selected guidelines, selection of 3 complementary large language models (LLMs), and execution of predefined prompts, multimodal recommendation extraction, semantic harmonization, and item-level guideline stance classification. The evaluative process combines a 3-LLM ensemble consistency check with an independent pre-Delphi appraisal by 4 penile cancer experts using a predefined 5-point suitability scale and acceptance threshold.Results In the worked example in the field of penile cancer as a high-stakes use case, the workflow extracted 157 clinical recommendations from 5 guidelines and consolidated them into 68 harmonized guideline-derived items. Stance classification identified 15 uniformly supported items and 53 divergent, ambiguous, or incompletely addressed items. In parallel, literature processing generated 95 harmonized white-area items, of which 9 were classified as true white-area items not addressed by the included guidelines. The final methodological output was a set of 62 human-reviewed Delphi-ready candidate items for subsequent clinical consensus voting.Conclusions This methodology provides a scalable, transparent, and transferable approach for identifying and structuring interguideline inconsistencies and evidence gaps. By combining LLM-assisted analysis with independent external expert appraisal, it supports the reproducible transformation of heterogeneous guideline content into Delphi-ready candidate items.
SN - 1439-4456
SN - 1438-8871
ER -