{
  "abstract": "Large language models are increasingly being used to evaluate clinical texts, a concept known as LLM-as-a-judge. However, most of the existing evidence focuses on English data and uses pooled agreement metrics that obscure clinically relevant failure modes. This study audited six open-weight models against expert judges on four German clinical datasets covering report and nursing documentation summarization, discharge summary generation, and patient letter generation. The models were evaluated across three paradigms and four prompt configurations. Alignment with experts varied by criterion and corpus, and no configuration proved reliable across paradigms. All judges exhibited position and identity bias. Substituting LLM-as-a-judge for the expert panel shifted 57.2% of pointwise categories, swapped 41.7% of pairwise winners, missed 10.2% of clinical errors, and inverted clinical priorities by overrating the criteria succinct and accurate. Therefore, LLM-as-a-judge is not a safe substitute for experts in German clinical text evaluations, and deployment thresholds should follow decision-impact analysis for LLM-as-a-judge in clinical contexts.",
  "abstract_shown": true,
  "affiliations": [
    "Institute for Transfusion Medicine, University Hospital Essen, Essen, Germany",
    "Department of Computer Science, University of Applied Sciences and Arts Dortmund (FHDO), Dortmund, Germany",
    "Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen, Essen, Germany",
    "Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany",
    "Central IT Department, Data Integration Center, University Hospital Essen, Essen, Germany",
    "West German Cancer Center (WTZ), University Hospital Essen, Essen, Germany",
    "Department of Medical Oncology, University Hospital Essen, Essen, Germany",
    "Department Nursing Development and Nursing Research, University Hospital Essen, Essen, Germany",
    "Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen, Essen, Germany",
    "National Center for Tumor Diseases (NCT), NCT West, Essen, Germany"
  ],
  "article_type": "research article",
  "arxiv": null,
  "author_total": 12,
  "authors": [
    {
      "affs": [
        0,
        1
      ],
      "name": "Tabea M. G. Pakull"
    },
    {
      "affs": [
        2,
        3
      ],
      "name": "Sina Warmer",
      "orcid": "0009-0002-2262-2655"
    },
    {
      "affs": [
        2,
        4
      ],
      "name": "Kamyar Arzideh"
    },
    {
      "affs": [
        5,
        6
      ],
      "name": "Thomas Hilser"
    },
    {
      "affs": [
        7
      ],
      "name": "Bernadette Hosters",
      "orcid": "0000-0002-5142-0385"
    },
    {
      "affs": [
        1,
        8
      ],
      "name": "Hendrik Damm"
    },
    {
      "affs": [
        2,
        3
      ],
      "name": "Felix Nensa"
    },
    {
      "affs": [
        5,
        6,
        9
      ],
      "name": "Martin Schuler"
    },
    {
      "affs": [
        2,
        3
      ],
      "name": "Katarzyna Borys",
      "orcid": "0000-0001-6987-6041"
    },
    {
      "affs": [
        0
      ],
      "name": "Peter A. Horn"
    },
    {
      "affs": [
        2,
        3
      ],
      "name": "René Hosch",
      "orcid": "0000-0003-1760-2342"
    },
    {
      "affs": [
        1,
        8
      ],
      "name": "Christoph M. Friedrich"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/210ekyh7q6j34.bib",
  "cited_by": 0,
  "doi": "10.1038/s41746-026-03443-2",
  "fulltext_url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13737126/",
  "id": "210ekyh7q6j34",
  "issns": [
    "2398-6352"
  ],
  "landing_url": "https://doi.org/10.1038/s41746-026-03443-2",
  "language": "en",
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "heading",
      "Article"
    ],
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ]
  ],
  "openalex": "W7225246785",
  "pdf_url": "https://www.nature.com/articles/s41746-026-03443-2.pdf",
  "pmcid": "PMC13737126",
  "pmid": "42982942",
  "published": "2026-11-04",
  "publisher": "Nature Publishing Group",
  "ris": "https://research.draconitic.com/paper/210ekyh7q6j34.ris",
  "sources": [
    {
      "first_seen": "2026-11-09T12:22:25Z",
      "native_id": "PMC13737126",
      "source": "pmc",
      "url": "https://doi.org/10.1038/s41746-026-03443-2"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "openalex:2718|Artificial Intelligence in Healthcare and Education",
      "subfield": "epidemiology-public-health"
    }
  ],
  "title": "Cross-paradigm audit of large language model judges for clinical text evaluation",
  "title_text": "Cross-paradigm audit of large language model judges for clinical text evaluation",
  "type": "peer-reviewed",
  "updated_at": "2026-11-09T12:22:25Z",
  "url": "https://research.draconitic.com/paper/210ekyh7q6j34",
  "venue": "NPJ Digital Medicine",
  "versions": []
}