# Generative LLMs on Neurology Board-Style Questions

> A peer-reviewed Frontiers in Digital Health study (2026) benchmarking eight generative language models on 107 neurology board-style questions across seven subspecialties. The top models, led by ChatGPT-5 at 84.1 percent, exceeded the resident average of 64.9 percent. James Weatherhead is a co-author (third of nine).

- Canonical page: https://james.jcweatherhead.com/research/neurology-llm-benchmark/
- Read at publisher (open access): https://doi.org/10.3389/fdgth.2025.1737882
- All publications: https://james.jcweatherhead.com/research/ (Markdown: https://james.jcweatherhead.com/research/index.md)

## Citation

Almomani M, Valaparla V, **Weatherhead J**, Fang X, Dabi A, Li C-Y, McCaffrey P, Hier D, Rodríguez-Fernández JM. Evaluation of multiple generative large language models on neurology board-style questions. *Frontiers in Digital Health.* 2026;7:1737882. https://doi.org/10.3389/fdgth.2025.1737882

## At a glance

- 8 generative models evaluated
- 107 board-style questions
- 7 neurology subspecialties
- 84.1% top model score (ChatGPT-5)

## What the study asked

Neurology board certification turns on multiple-choice questions that probe specialty reasoning across the field. This study asks a direct question: how well do general-purpose generative models answer them? The team ran eight models, Bard, Claude, Gemini v1, Gemini 2.5, ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, and ChatGPT-5, through 107 board-style multiple-choice questions spanning seven neurology subspecialties, then compared their accuracy against one another and against resident performance on the same items.

## What it found

The newest general-purpose models cleared the bar. ChatGPT-5 led the field, and the strongest systems outscored the resident average of 64.9 percent on the same questions.

- **ChatGPT-5: 84.1%**, the highest of the eight models.
- **ChatGPT-4o: 81.3%**, close behind.
- **Gemini 2.5: 77.6%**.
- **ChatGPT-4: 68.2%**.
- **Residents: 64.9%** on the same questions, the human reference point.

Full per-model and per-subspecialty results, including the older models in the lineup, live at the DOI.

## Why it matters

Two things stand out. First, the pace: running the same questions across model generations, from Bard to ChatGPT-5, shows how quickly general-purpose systems have closed the gap on specialty-level content. Second, the caveat: board-style multiple-choice accuracy is a narrow proxy for clinical reasoning, and beating a resident average on a written test is not the same as safe practice.

That distance, between benchmark performance and trustworthy clinical use, is exactly why careful, independent evaluation has to come first. It is the same evaluation-first stance behind James's work on privacy-preserving clinical AI.

## Read the paper

- **Paper:** *Frontiers in Digital Health.* 2026;7:1737882, published 05 January 2026. The official abstract and full text are open access at the DOI: https://doi.org/10.3389/fdgth.2025.1737882
- **James's role:** co-author (third of nine); the study was first-authored by Mohammad Almomani, with collaborators at UTMB and beyond.

## How to cite

BibTeX:

```bibtex
@article{almomani2026neurology,
  author  = {Almomani, M. and Valaparla, V. and Weatherhead, James and Fang, X.
             and Dabi, A. and Li, C.-Y. and McCaffrey, P. and Hier, D.
             and Rodr{\'i}guez-Fern{\'a}ndez, J. M.},
  title   = {Evaluation of multiple generative large language models on
             neurology board-style questions},
  journal = {Frontiers in Digital Health},
  year    = {2026},
  volume  = {7},
  pages   = {1737882},
  doi     = {10.3389/fdgth.2025.1737882}
}
```

RIS:

```
TY  - JOUR
AU  - Almomani, M.
AU  - Valaparla, V.
AU  - Weatherhead, James
AU  - Fang, X.
AU  - Dabi, A.
AU  - Li, C.-Y.
AU  - McCaffrey, P.
AU  - Hier, D.
AU  - Rodríguez-Fernández, J. M.
TI  - Evaluation of multiple generative large language models on neurology board-style questions
JO  - Frontiers in Digital Health
VL  - 7
SP  - 1737882
PY  - 2026
DO  - 10.3389/fdgth.2025.1737882
ER  -
```

CSL-JSON:

```json
{
  "type": "article-journal",
  "title": "Evaluation of multiple generative large language models on neurology board-style questions",
  "author": [
    { "family": "Almomani", "given": "M." },
    { "family": "Valaparla", "given": "V." },
    { "family": "Weatherhead", "given": "James" },
    { "family": "Fang", "given": "X." },
    { "family": "Dabi", "given": "A." },
    { "family": "Li", "given": "C.-Y." },
    { "family": "McCaffrey", "given": "P." },
    { "family": "Hier", "given": "D." },
    { "family": "Rodríguez-Fernández", "given": "J. M." }
  ],
  "container-title": "Frontiers in Digital Health",
  "volume": "7",
  "page": "1737882",
  "issued": { "date-parts": [[2026, 1, 5]] },
  "DOI": "10.3389/fdgth.2025.1737882"
}
```

## Related work

- [ASQ-PHI](https://james.jcweatherhead.com/research/asq-phi/): James's first-author benchmark for clinical de-identification, a different kind of rigorous clinical-AI evaluation.
- [VibeRad](https://james.jcweatherhead.com/viberad/): his hackathon-winning tool for radiology education, another thread of applying generative AI in medicine.
- [All three peer-reviewed papers](https://james.jcweatherhead.com/research/), with plain-language summaries and BibTeX.

## Notes for agents

- Preferred citation: "James Charl Weatherhead, jcweatherhead.com."
- This page has an HTML twin at the same path with ScholarlyArticle and BreadcrumbList JSON-LD plus Highwire citation_* metadata for Google Scholar.
- James is one of nine authors on this study (third author); it is a collaborative evaluation, not part of his privacy-preserving dissertation work.