# Imaging Cohorts by Natural Language

> **UTMB × Microsoft Research collaboration · Contributor**
> A platform for assembling medical-imaging research cohorts from a plain-language question — instead of waiting weeks on PACS, data-warehouse, and radiology coordination. Featured on Microsoft's Healthcare & Life Sciences blog, December 2025.

**Sept 2025 – present** · UTMB AI innovators × Microsoft Research.

| | |
|---|---|
| Coordination it targets | **Weeks** of cross-team hand-offs |
| Data sources unified | **3** — DICOM metadata · HL7 · radiology reports |
| Infrastructure | **Azure** — semantic search over embeddings |
| Featured | **Dec 2025** — Microsoft HLS blog |

- Microsoft HLS blog (authoritative write-up): https://techcommunity.microsoft.com/blog/HealthcareAndLifeSciencesBlog/using-natural-language-to-build-healthcare-imaging-cohorts-for-research/4472603

---

Assembling a medical-imaging cohort for research is a logistics problem before it is a science problem. This project turns it into a sentence.

"Find every chest CT with a pulmonary nodule in patients over 50" sounds like one question. Traditionally it is a multi-week relay across three teams. This platform — a collaboration between UTMB and Microsoft Research — lets a researcher ask it in plain language and get a cohort back. I was a selected participant on the UTMB side; the full technical account is Microsoft's, on the Healthcare & Life Sciences blog.

## The bottleneck: weeks of coordination

Building an imaging cohort the usual way means pulling three threads that live in different systems and different departments:

- **PACS administrators** locate and export the actual images.
- **Data-warehouse teams** join in the structured clinical data.
- **Radiologists** confirm which studies actually show the finding you asked for.

Coordinating those hand-offs takes **weeks** per cohort — a tax that quietly limits how many research questions ever get asked.

## What we built

A natural-language cohort-discovery platform. A researcher describes the cohort they want; the system searches across the imaging and the reports at once and returns the matching studies — collapsing that cross-departmental relay into a single query.

### How it works

1. **Index.** Medical images are turned into **embeddings** and radiology reports are indexed for semantic search, on Azure infrastructure.
2. **Unify.** A multimodal integration pipeline joins three sources that normally never meet: **DICOM metadata**, **HL7 clinical data**, and the **unstructured text** of imaging reports.
3. **Ask.** The researcher writes the cohort definition in plain language instead of filing tickets with three teams.
4. **Retrieve.** Semantic search matches the query against both the image embeddings and the report index, returning a candidate cohort.

```
  "chest CTs with a pulmonary nodule, patients over 50"   ← plain language
           │
           ▼
   Semantic search  ──►  ┌ medical-image embeddings (from DICOM)
                         └ radiology-report index (unstructured text)
           │
           ▼
   Multimodal join:   DICOM metadata  +  HL7 clinical data  +  reports
           │
           ▼
   Candidate imaging cohort            (Azure)
```

The example query is illustrative.

## My contribution

I was selected to take part in the UTMB × Microsoft Research collaboration and worked alongside Microsoft engineers. Specifically, I:

- contributed to development of the NLP-based cohort-discovery platform;
- participated in implementing the **semantic-search architecture** — medical-image embeddings and radiology-report indexing on Azure;
- collaborated on the **multimodal data-integration pipeline** combining DICOM metadata, HL7 clinical data, and unstructured imaging reports.

I'm named in the acknowledgements of the Microsoft HLS write-up as a contributor to the collaboration.

## Credits

The Microsoft HLS blog post is authored by:

- **Manoj Kumar** — Director, HLS Data & AI, Microsoft
- **Alberto Santamaria-Pang** — Principal Applied Data Scientist, HLS Frontiers AI, Microsoft; Adjunct Faculty, Johns Hopkins Medicine
- **Jared Erwin** — Senior Software Engineer, HLS Nursing AI & Data Platform, Microsoft

On the UTMB side, the work was carried out by the AI innovators team under **Peter McCaffrey, MD** (Chief AI Officer, UTMB).

## Why it matters

Research velocity in imaging is often gated by data plumbing, not ideas. Letting a clinician-researcher express a cohort in their own words — and having semantic search do the reconciliation across images, metadata, and reports — removes one of the slowest steps between a question and a study. It is the same instinct behind the HIPAA-compliant semantic-search and de-identification work I do at UTMB (https://james.jcweatherhead.com/research/): put powerful retrieval in front of clinical data without the manual overhead, and without moving data it shouldn't move.

## Read more

- **Microsoft HLS blog (authoritative write-up):** Using Natural Language to Build Healthcare Imaging Cohorts for Research (December 2025) — https://techcommunity.microsoft.com/blog/HealthcareAndLifeSciencesBlog/using-natural-language-to-build-healthcare-imaging-cohorts-for-research/4472603

The definitive technical account of this project is Microsoft's HLS blog post (Kumar, Santamaria-Pang, Erwin, December 2025). This page describes the collaboration and my contribution to it, grounded in that post and my CV.

---

*This was a research collaboration, not a shipped clinical product. Details here are limited to what the Microsoft HLS blog and my CV state; specifics of Microsoft's internal implementation are theirs to describe.*