Why few patients per site breaks the comparison
When people are not assigned to a medicine at random, the two groups differ before anyone takes anything: in age, in other illnesses, in the other medicines they use. Studies correct for this with a propensity score, a model of how likely each person was to get one medicine rather than the other, given their health before treatment. The groups are then balanced on that score.
That model needs enough people on each medicine to learn from. In a network of health records and insurance claims, each site may hold only a handful of people on a new medicine, and the sites differ from one another. A model fitted at one site can be unreliable at the next.
Three ideas
The project works on three problems that come up whenever sites learn together but each needs a model that fits its own patients.
- Adapt to each site without starting over. A site should get a model shaped to its own patients without training a full model of its own.
- Learn when the event is rare. When few people get a medicine, or few have the outcome, a site has little to learn from on its own.
- Keep what the network learned. Adapting to one site should not erase what the sites learned together.
It grows out of my disability progression study, where letting each country adapt a shared model predicted better than one shared model trained across countries, and better than pooling every record.
Two years, two settings
First, multiple sclerosis. The methods are developed and tested on routine clinical data from an international multiple sclerosis network, on a task I know well: predicting whether a person's disability will get worse within two years.
Then, pharmaceutical research. The methods that meet the criteria set in advance are used to estimate which treatment a person is likely to receive, from their health before treatment. Those estimates balance the treatment groups, so a comparison of safety and effectiveness is more credible when random assignment is not possible. The data there will be Optum's health records and insurance claims from the United States, where payers, providers and regional health systems act as the sites.
What it should deliver: a validated way of doing this, open source software, practical documentation, papers, and a workflow tested with the industry partner.
What an Innovation Mandate is
An Innovation Mandate is a grant from VLAIO, the Flemish government's agency for innovation, for a postdoctoral researcher to carry out a project in close collaboration with a company. The researcher, a Flemish university and a Flemish company form the consortium, and the researcher writes and submits the proposal. It is meant for research that is still too early and too uncertain for a company to do alone, with results the company can use.
I wrote the proposal and defended it. Hasselt University hosts the project, Liesbet Peeters is the promoter, and Johnson & Johnson Innovative Medicine is the industry partner.
The detailStatus and sources
What it does not show yet
Whether the methods carry over. In my multiple sclerosis study each patient was a few dozen clinical measures, while in claims data a person is a very long list of billing and diagnosis codes.
How I will judge it: step by step against what is done today, on whether the two groups end up alike, and whether that holds without adding bias.
Links
- The project on the Flemish research portal: Personalized Federated Learning for Heterogeneous Health Data Networks: From Methodological Development to Industrial Application in Pharmaceutical Research.
- Innovation Mandates, on the VLAIO site.
- Where it started: Two years ahead, will this person get worse?