AI on Neurology Boards

Neurology board certification turns on multiple-choice questions that probe specialty reasoning across the field. This study asks a direct question: how well do general-purpose generative models answer them? The team ran eight models, Bard, Claude, Gemini v1, Gemini 2.5, ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, and ChatGPT-5, through 107 board-style multiple-choice questions spanning seven neurology subspecialties, then compared their accuracy against one another and against resident performance on the same items.

The newest general-purpose models cleared the bar. ChatGPT-5 led the field, and the strongest systems outscored the resident average of 64.9 percent on the same questions.

• ChatGPT-5: 84.1%, the highest of the eight models.
• ChatGPT-4o: 81.3%, close behind.
• Gemini 2.5: 77.6%.
• ChatGPT-4: 68.2%.
• Residents: 64.9% on the same questions, the human reference point.

Full per-model and per-subspecialty results, including the older models in the lineup, live at the DOI.

Two things stand out. First, the pace: running the same questions across model generations, from Bard to ChatGPT-5, shows how quickly general-purpose systems have closed the gap on specialty-level content. Second, the caveat: board-style multiple-choice accuracy is a narrow proxy for clinical reasoning, and beating a resident average on a written test is not the same as safe practice.

That distance, between benchmark performance and trustworthy clinical use, is exactly why careful, independent evaluation has to come first. It is the same evaluation-first stance behind my work on privacy-preserving clinical AI.

• Paper: Frontiers in Digital Health. 2026;7:1737882, published 05 January 2026. The official abstract and full text are open access at the DOI.
• My role: co-author (third of nine); the study was first-authored by Mohammad Almomani, with collaborators at UTMB and beyond.

Peer-reviewed and citable. Copy a ready-made citation, or read the official abstract at the DOI.

@article{almomani2026neurology,
  author  = {Almomani, M. and Valaparla, V. and Weatherhead, James and Fang, X.
             and Dabi, A. and Li, C.-Y. and McCaffrey, P. and Hier, D.
             and Rodr{\'i}guez-Fern{\'a}ndez, J. M.},
  title   = {Evaluation of multiple generative large language models on
             neurology board-style questions},
  journal = {Frontiers in Digital Health},
  year    = {2026},
  volume  = {7},
  pages   = {1737882},
  doi     = {10.3389/fdgth.2025.1737882}
}

TY  - JOUR
AU  - Almomani, M.
AU  - Valaparla, V.
AU  - Weatherhead, James
AU  - Fang, X.
AU  - Dabi, A.
AU  - Li, C.-Y.
AU  - McCaffrey, P.
AU  - Hier, D.
AU  - Rodríguez-Fernández, J. M.
TI  - Evaluation of multiple generative large language models on neurology board-style questions
JO  - Frontiers in Digital Health
VL  - 7
SP  - 1737882
PY  - 2026
DO  - 10.3389/fdgth.2025.1737882
ER  - 

{
  "type": "article-journal",
  "title": "Evaluation of multiple generative large language models on neurology board-style questions",
  "author": [
    { "family": "Almomani", "given": "M." },
    { "family": "Valaparla", "given": "V." },
    { "family": "Weatherhead", "given": "James" },
    { "family": "Fang", "given": "X." },
    { "family": "Dabi", "given": "A." },
    { "family": "Li", "given": "C.-Y." },
    { "family": "McCaffrey", "given": "P." },
    { "family": "Hier", "given": "D." },
    { "family": "Rodríguez-Fernández", "given": "J. M." }
  ],
  "container-title": "Frontiers in Digital Health",
  "volume": "7",
  "page": "1737882",
  "issued": { "date-parts": [[2026, 1, 5]] },
  "DOI": "10.3389/fdgth.2025.1737882"
}

• ASQ-PHI: my first-author benchmark for clinical de-identification, a different kind of rigorous clinical-AI evaluation.
• VibeRad: my hackathon-winning tool for radiology education, another thread of applying generative AI in medicine.
• All three peer-reviewed papers, with plain-language summaries and BibTeX.