# ASQ-PHI: An Adversarial Synthetic Benchmark for Clinical PHI De-Identification

> A fully synthetic, adversarial benchmark of 1,051 PHI-annotated clinical queries that scores de-identification accuracy and search utility together, across rule-based, cloud, vector, and on-device redaction. Peer-reviewed in Data in Brief (2026); open data, MIT-licensed code.

**Weatherhead J, Golovko G, McCaffrey P.** ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility. *Data in Brief.* 2026;65:112586. https://doi.org/10.1016/j.dib.2026.112586

- **Read at publisher:** https://doi.org/10.1016/j.dib.2026.112586
- **Code on GitHub (MIT):** https://github.com/JamesWeatherhead/asq-phi
- **All publications:** https://james.jcweatherhead.com/research/

ASQ-PHI is a fully synthetic, adversarial benchmark for clinical PHI de-identification. It scores two things most de-identification work measures separately: how completely protected health information is removed, and whether the redacted text is still useful for search and retrieval afterward. Every one of its 1,051 clinical queries is synthetic, so competing methods can be compared on open data without ever exposing a real patient record. Peer-reviewed in Data in Brief (2026), it is one of the three contributions in my dissertation on privacy-preserving clinical AI, alongside the K-anonymity decay analysis and the Local PHI Scrubber.

## At a glance

| | |
|---|---|
| Annotated clinical queries | **1,051** |
| Redaction approaches compared | **4** |
| Metrics scored together | **2** |
| License | **MIT** (open benchmark and code) |

## The problem it measures

Clinicians increasingly want modern AI at the point of care: web search, current-literature retrieval, general-purpose assistants. Most of those tools are not covered by a HIPAA Business Associate Agreement, so any query that carries protected health information has to be de-identified before it leaves the hospital.

That creates a measurement problem with two ways to fail. Leave identifiers in, and you have a privacy breach. Strip too aggressively, and the query becomes so vague that the AI returns nothing useful, and clinicians quietly fall back on unsanctioned tools. To improve on that tradeoff you first have to measure it, and there was no open benchmark that scored both sides at once on data anyone could freely share. ASQ-PHI is built to be that benchmark.

## What is in it

ASQ-PHI is a set of 1,051 synthetic, PHI-annotated clinical queries, written to be adversarial rather than easy: the hard, realistic cases where identifiers hide in context, not just obvious names and dates. Because the data is entirely synthetic, the whole benchmark can be released as open data with no patient record behind it.

It ships as a generation framework, not only a fixed set, so the same pipeline can produce custom variants for a specific institution or threat model. The benchmark and its code are open source under the MIT license.

## What it scores

For each redaction method, ASQ-PHI reports two paired measures: **de-identification accuracy**, whether the protected health information was actually removed, and **search utility**, whether the redacted query still retrieves the right answer. Pairing them is the point. A method that redacts everything scores perfectly on privacy and uselessly on utility; the methods worth deploying are the ones that hold both.

The benchmark runs that comparison across four families of redaction: rule-based, cloud, vector, and on-device (Ollama). The full per-method results live in the paper.

## Data and code

- **Paper:** *Data in Brief.* 2026;65:112586. The official abstract and full text live at the DOI (an open-access Elsevier journal): https://doi.org/10.1016/j.dib.2026.112586
- **Benchmark and code:** the annotated queries and the generation pipeline are open source (MIT): https://github.com/JamesWeatherhead/asq-phi

## How to cite

BibTeX:

```bibtex
@article{weatherhead2026asqphi,
  author  = {Weatherhead, James and Golovko, G. and McCaffrey, P.},
  title   = {{ASQ-PHI}: An adversarial synthetic data benchmark for clinical
             de-identification and search utility},
  journal = {Data in Brief},
  year    = {2026},
  volume  = {65},
  pages   = {112586},
  doi     = {10.1016/j.dib.2026.112586}
}
```

RIS:

```
TY  - JOUR
AU  - Weatherhead, James
AU  - Golovko, G.
AU  - McCaffrey, P.
TI  - ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility
JO  - Data in Brief
VL  - 65
SP  - 112586
PY  - 2026
DO  - 10.1016/j.dib.2026.112586
ER  -
```

CSL-JSON:

```json
{
  "type": "article-journal",
  "title": "ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility",
  "author": [
    { "family": "Weatherhead", "given": "James" },
    { "family": "Golovko", "given": "G." },
    { "family": "McCaffrey", "given": "P." }
  ],
  "container-title": "Data in Brief",
  "volume": "65",
  "page": "112586",
  "issued": { "date-parts": [[2026]] },
  "DOI": "10.1016/j.dib.2026.112586"
}
```

## Presented at

- **AMIA 2026 Amplify** (Denver, May 2026): Poster, Health Equity Track, sponsored by the CU Anschutz Center for Health AI. "ASQ-PHI: An Adversarial Synthetic Benchmark for Clinical PHI De-Identification."

## Related work

- [Local PHI Scrubber](https://james.jcweatherhead.com/local-phi-scrubber/): the on-device, human-in-the-loop redaction tool that applies this line of work at the point of use (AMIA 2026 oral).
- [K-anonymity decay in multi-turn clinical LLM conversations](https://doi.org/10.3389/fdgth.2026.1832168): the companion paper on how re-identification risk accumulates across a conversation.
- [All three peer-reviewed papers](https://james.jcweatherhead.com/research/), with plain-language summaries and BibTeX.