{
  "abstract": "Large language models (LLMs) are increasingly evaluated for clinical tasks, but safe use requires reliable uncertainty estimates as well as high accuracy. We tested eight general-purpose LLMs on 100 validated neurology board-style multiple-choice questions and compared three confidence signals: self-reported confidence, critique-implied confidence from an independent model instance, and consistency-based confidence from repeated sampling. Accuracy varied from 45% (Llama 3.1 8B) to 92% (Gemini 3 Flash/Pro), but confidence was generally overestimated, particularly in lower-accuracy models. Consistency-based confidence provided the best discrimination between correct and incorrect answers (pooled AUROC 0.74), outperforming self-report, although calibration differences between methods were not significant. Calibration should therefore be reported alongside accuracy when clinical LLMs are evaluated.\n\nShort DescriptionWe evaluated the accuracy and confidence of eight large language models on 100 neurology board-style questions. We found substantial overconfidence, and differences between confidence methods in their ability to distinguish correct from incorrect answers.",
  "abstract_shown": true,
  "affiliations": [
    "Imperial College London"
  ],
  "article_type": "PUBLISHAHEADOFPRINT",
  "arxiv": null,
  "author_total": 5,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "N. S. Graham"
    },
    {
      "name": "S. D. Auger"
    },
    {
      "name": "B. Attaallah"
    },
    {
      "name": "A. Batla"
    },
    {
      "name": "G. Scott"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/532qf8mq9jre0.bib",
  "cited_by": 0,
  "doi": "10.64898/2026.10.17.26365807",
  "fulltext_url": "https://www.medrxiv.org/content/10.64898/2026.10.17.26365807v1.full",
  "id": "532qf8mq9jre0",
  "issns": [],
  "landing_url": "https://www.medrxiv.org/content/10.64898/2026.10.17.26365807v1",
  "language": null,
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "medrxiv",
      "neurology"
    ],
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ]
  ],
  "openalex": "W7222953311",
  "pdf_url": "https://www.medrxiv.org/content/10.64898/2026.10.17.26365807v1.full.pdf",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-17",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/532qf8mq9jre0.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:21:12Z",
      "native_id": "10.64898/2026.10.17.26365807",
      "source": "medrxiv",
      "url": "https://www.medrxiv.org/content/10.64898/2026.10.17.26365807v1"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "medrxiv:neurology",
      "subfield": "neurology"
    }
  ],
  "title": "Accuracy and confidence calibration of large language models on neurology board-style questions",
  "title_text": "Accuracy and confidence calibration of large language models on neurology board-style questions",
  "type": "preprint",
  "updated_at": "2026-10-30T12:21:12Z",
  "url": "https://research.draconitic.com/paper/532qf8mq9jre0",
  "venue": "medRxiv",
  "versions": []
}