Imaging Cohorts by Natural Language: a UTMB × Microsoft Research collaboration

Assembling a medical-imaging cohort for research is a logistics problem before it is a science problem. This project turns it into a sentence.

“Find every chest CT with a pulmonary nodule in patients over 50” sounds like one question. Traditionally it is a multi-week relay across three teams. This platform — a collaboration between UTMB and Microsoft Research — lets a researcher ask it in plain language and get a cohort back. I was a selected participant on the UTMB side; the full technical account is Microsoft’s, on the Healthcare & Life Sciences blog.

The bottleneck: weeks of coordination

Building an imaging cohort the usual way means pulling three threads that live in different systems and different departments:

PACS administrators locate and export the actual images.
Data-warehouse teams join in the structured clinical data.
Radiologists confirm which studies actually show the finding you asked for.

Coordinating those hand-offs takes weeks per cohort — a tax that quietly limits how many research questions ever get asked.

What we built

A natural-language cohort-discovery platform. A researcher describes the cohort they want; the system searches across the imaging and the reports at once and returns the matching studies — collapsing that cross-departmental relay into a single query.

How it works

Index. Medical images are turned into embeddings and radiology reports are indexed for semantic search, on Azure infrastructure.
Unify. A multimodal integration pipeline joins three sources that normally never meet: DICOM metadata, HL7 clinical data, and the unstructured text of imaging reports.
Ask. The researcher writes the cohort definition in plain language instead of filing tickets with three teams.
Retrieve. Semantic search matches the query against both the image embeddings and the report index, returning a candidate cohort.

  "chest CTs with a pulmonary nodule, patients over 50"   ← plain language
           │
           ▼
   Semantic search  ──►  ┌ medical-image embeddings (from DICOM)
                         └ radiology-report index (unstructured text)
           │
           ▼
   Multimodal join:   DICOM metadata  +  HL7 clinical data  +  reports
           │
           ▼
   Candidate imaging cohort            (Azure)

The pipeline as described in the CV and Microsoft’s HLS blog. The example query is illustrative.

My contribution

I was selected to take part in the UTMB × Microsoft Research collaboration and worked alongside Microsoft engineers. Specifically, I:

contributed to development of the NLP-based cohort-discovery platform;
participated in implementing the semantic-search architecture — medical-image embeddings and radiology-report indexing on Azure;
collaborated on the multimodal data-integration pipeline combining DICOM metadata, HL7 clinical data, and unstructured imaging reports.

I’m named in the acknowledgements of the Microsoft HLS write-up as a contributor to the collaboration.

Credits

The Microsoft HLS blog post is authored by:

Manoj Kumar — Director, HLS Data & AI, Microsoft
Alberto Santamaria-Pang — Principal Applied Data Scientist, HLS Frontiers AI, Microsoft; Adjunct Faculty, Johns Hopkins Medicine
Jared Erwin — Senior Software Engineer, HLS Nursing AI & Data Platform, Microsoft

On the UTMB side, the work was carried out by the AI innovators team under Peter McCaffrey, MD (Chief AI Officer, UTMB).

Why it matters

Research velocity in imaging is often gated by data plumbing, not ideas. Letting a clinician-researcher express a cohort in their own words — and having semantic search do the reconciliation across images, metadata, and reports — removes one of the slowest steps between a question and a study. It is the same instinct behind the HIPAA-compliant semantic-search and de-identification work I do at UTMB: put powerful retrieval in front of clinical data without the manual overhead, and without moving data it shouldn’t move.

Microsoft HLS blog (authoritative write-up): Using Natural Language to Build Healthcare Imaging Cohorts for Research (December 2025)

The definitive technical account of this project is Microsoft’s HLS blog post (Kumar, Santamaria-Pang, Erwin, December 2025). This page describes the collaboration and my contribution to it, grounded in that post and my CV.

This was a research collaboration, not a shipped clinical product. Details here are limited to what the Microsoft HLS blog and my CV state; specifics of Microsoft’s internal implementation are theirs to describe.