ASQ-PHI

Clinicians increasingly want modern AI at the point of care: web search, current-literature retrieval, general-purpose assistants. Most of those tools are not covered by a HIPAA Business Associate Agreement, so any query that carries protected health information has to be de-identified before it leaves the hospital.

That creates a measurement problem with two ways to fail. Leave identifiers in, and you have a privacy breach. Strip too aggressively, and the query becomes so vague that the AI returns nothing useful, and clinicians quietly fall back on unsanctioned tools. To improve on that tradeoff you first have to measure it, and there was no open benchmark that scored both sides at once on data anyone could freely share. ASQ-PHI is built to be that benchmark.

ASQ-PHI is a set of 1,051 synthetic, PHI-annotated clinical queries, written to be adversarial rather than easy: the hard, realistic cases where identifiers hide in context, not just obvious names and dates. Because the data is entirely synthetic, the whole benchmark can be released as open data with no patient record behind it.

It ships as a generation framework, not only a fixed set, so the same pipeline can produce custom variants for a specific institution or threat model. The benchmark and its code are open source under the MIT license.

For each redaction method, ASQ-PHI reports two paired measures: de-identification accuracy, whether the protected health information was actually removed, and search utility, whether the redacted query still retrieves the right answer. Pairing them is the point. A method that redacts everything scores perfectly on privacy and uselessly on utility; the methods worth deploying are the ones that hold both.

The benchmark runs that comparison across four families of redaction: rule-based, cloud, vector, and on-device (Ollama). The full per-method results live in the paper.

• Paper: Data in Brief. 2026;65:112586. The official abstract and full text live at the DOI (an open-access Elsevier journal).
• Benchmark and code: the annotated queries and the generation pipeline are open source (MIT) at github.com/JamesWeatherhead/asq-phi.

Peer-reviewed and citable. Copy a ready-made citation, or read the official abstract at the DOI.

@article{weatherhead2026asqphi,
  author  = {Weatherhead, James and Golovko, G. and McCaffrey, P.},
  title   = {{ASQ-PHI}: An adversarial synthetic data benchmark for clinical
             de-identification and search utility},
  journal = {Data in Brief},
  year    = {2026},
  volume  = {65},
  pages   = {112586},
  doi     = {10.1016/j.dib.2026.112586}
}

TY  - JOUR
AU  - Weatherhead, James
AU  - Golovko, G.
AU  - McCaffrey, P.
TI  - ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility
JO  - Data in Brief
VL  - 65
SP  - 112586
PY  - 2026
DO  - 10.1016/j.dib.2026.112586
ER  - 

{
  "type": "article-journal",
  "title": "ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility",
  "author": [
    { "family": "Weatherhead", "given": "James" },
    { "family": "Golovko", "given": "G." },
    { "family": "McCaffrey", "given": "P." }
  ],
  "container-title": "Data in Brief",
  "volume": "65",
  "page": "112586",
  "issued": { "date-parts": [[2026]] },
  "DOI": "10.1016/j.dib.2026.112586"
}

• Local PHI Scrubber: the on-device, human-in-the-loop redaction tool that applies this line of work at the point of use (AMIA 2026 oral).
• K-anonymity decay in multi-turn clinical LLM conversations: the companion paper on how re-identification risk accumulates across a conversation.
• All three peer-reviewed papers, with plain-language summaries and BibTeX.