{
  "abstract": "Background: Jev is a non-generative System One model that assigns probabilities to predefined answer options and cannot answer outside them. Its calibration and its performance on complex diagnostic cases remain insufficiently characterised. Methods: We evaluated Jev 1.13 on four medical benchmarks: MetaMedQA (1,373 examination questions), PubMedQA (500 research questions), DiagnosisArena-MCQ (915 cases) and the NEJM Case Challenges (34 cases). GPT-6 Sol, with and without reasoning, was the reference. Outcomes were top-1 accuracy (primary), calibration, selective prediction and recognition of unanswerable questions Results: Jev's accuracy was similar to that of GPT-6 Sol with medium reasoning on PubMedQA (78.4% vs 78.2%), lower on MetaMedQA (74.8% vs 82.7%) and much lower on DiagnosisArena-MCQ (59.8% vs 82.4%) and the NEJM cases (61.8% vs 82.4%, adjusted P = 0.078). On MetaMedQA, Jev's probabilities were the best calibrated (expected calibration error 0.063 vs 0.146), and its answers with a probability ≥0.9 (52.9% of questions) were 93.4% accurate, but GPT-6 Sol was as accurate when it accepted a similar proportion of questions. On DiagnosisArena-MCQ, Jev's probabilities discriminated poorly (AUROC 0.645 vs 0.768). Of the 162 questions whose correct answer was \"I don't know or cannot answer\", Jev chose that option for 10.5% (GPT-6 Sol, 8.6%). Median latency was 0.27–0.31 s; all 2,823 items cost US$0.08. Conclusions: Jev was fast and inexpensive, and its accuracy was similar to that of a frontier LLM on research abstracts but lower on examination questions and much lower on complex diagnostic cases. Its better calibration on examination questions gave no selective-prediction advantage, and it seldom chose \"I don't know\" when that was the correct answer. Task-specific validation is required before clinical use.",
  "abstract_shown": true,
  "affiliations": [
    "Independent researcher Madrid , Spain"
  ],
  "article_type": null,
  "arxiv": null,
  "author_total": 2,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Alfredo Madrid-García",
      "orcid": "0000-0002-1591-0467"
    },
    {
      "affs": [
        0
      ],
      "name": "Beatriz Merino Barbancho",
      "orcid": "0000-0001-5070-4178"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/0rxtsbr6h7g64.bib",
  "cited_by": 15,
  "doi": "10.48550/arxiv.2609.34024",
  "fulltext_url": null,
  "id": "0rxtsbr6h7g64",
  "issns": [
    "2331-8422"
  ],
  "landing_url": "https://arxiv.org/abs/2609.34024",
  "language": "en",
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "journal",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ]
  ],
  "openalex": "W7214655821",
  "pdf_url": "https://arxiv.org/pdf/2609.34024",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-27",
  "publisher": "Cornell University",
  "ris": "https://research.draconitic.com/paper/0rxtsbr6h7g64.ris",
  "sources": [
    {
      "first_seen": "2026-11-07T13:20:59Z",
      "native_id": "W7214655821",
      "source": "openalex",
      "url": "https://arxiv.org/abs/2609.34024"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "openalex:2718|Artificial Intelligence in Healthcare and Education",
      "subfield": "epidemiology-public-health"
    }
  ],
  "title": "Jev in Medicine: A Benchmark Evaluation",
  "title_text": "Jev in Medicine: A Benchmark Evaluation",
  "type": "preprint",
  "updated_at": "2026-11-13T13:21:12Z",
  "url": "https://research.draconitic.com/paper/0rxtsbr6h7g64",
  "venue": "arXiv (Cornell University)",
  "versions": []
}