by Luke Shulman

Predicting Emergency Room Visits:

Why Melbourne’s proof of concept is likely to succeed.

Through research into modeling care pathways, I came across Dr. Chris Pearce & team’s proposed study to gauge a patient’s risk of an emergency visit. The proof-of-concept study will use the clinical data warehouse in East Melbourne to train a predictive machine learning algorithm to identify patients likely to present at the emergency department.

In one sense, this program is typical of various population health efforts. Assemble a data source, provide a robust model for measuring risk based on multiple factors, then use that information to prevent hospitalizations. But for many reasons, this project is different both in its robust technological approach robust, and it its governance framework. Therefore, I am optimistic about this effort especially in comparison to the lackluster production ready applications of machine learning in Healthcare launched so far.

I actually e-mailed Dr. Pearce to learn a bit more about their study and then re-read their proof of concept and found some new things that make me really excited.

The factors that will lead to success

  • Great Foundational Assets: The team around Melbourne and the Melbourne East General Practice have spent quite a bit of time assembling a quality data asset from EHR data from their general practice offices. This data warehouse is critical foundation for this new project. The data warehouse effort involved the kind of door to door recruiting and general governance that is critical to getting a care community to buy in on a new data tool. The key for this project is that the POLAR team just got more connectivity to ED data enabling this type of end-to-end patient journey study.
  • Deeper/Richer Data: To help build a model that will reflect actual care journeys, the team is bringing rich definitions to each of their features that are used in the ML algorithim. The model won’t simply know that a certain lab test was done, their providers, lead by Dr. Pearce, have actually created ranges for normal, abnormal, extremely abnormal features based on the values. The same classification has been done for dosages of common medications. So the model in development knows not just that a patient is both on a certain type of medication and that the dosage is very high.

We have then gone through the data to go through not only what the data team should use, but also weighted it…So We don’t just look at Blood pressure, but rank them as normal, medium elevated or highly elevated. I went through all the medications and ranked them as : usual dose, lower than usual dose, higher than usual dose. blood sugars, presence of care plans, a large number of factors. The computer has to sort them all out. - Dr. Christopher Pearce

To illustrate the power of this by example, imagine if Netflix based their recommendation engine just based on what you watch and the genres and descriptions of those movies. It might work but not as effective as if an expert (you) provided ratings about what you liked. Dr. Pearce, and his clinician team, are supplying that. Rating certain features by their subjective expert understanding. Dr. Pearce shared with me a screen-shot of a spreed-sheet where he is reviewing what looked like 1000 prescriptions and providing clinical intelligence about the use of them by dosage. This supervision for the model occurring prior to its development is critical. It will not only improve the predictive ability of the model but I’d imagine it would help develop trust when it is finally deployed into practices.

To illustrate the power of this by example, imagine if Netflix based their recommendation engine just based on what you watch and the genres and descriptions. It might be effective but not as a effective as if an expert (you) provided ratings about what you liked. Dr. Pearce, and his clinician team, are supplying the “ratings”. Rating certain features by their subjective expert understanding. Dr. Pearce shared with me a screen-shot of a spread-sheet where he is reviewing what looked like 1000 prescriptions and providing clinical intelligence about the use of them by dosage. This supervision for the model occurring prior to its development is critical. It will not only improve the predictive ability of the model but I’d imagine it would help develop trust when it is finally deployed into practices.

  • Multi/Factorial: The underlying model looks like it will take in a wide swath of variables across 14 different groups (see above). It is absolutely one of the most comprehensive applications of a predictive model built from an outpatient dataset. That just gives the model more to work with. I am willing to bet that the output finds some funny associations but ones that could be very valuable.
  • Provider Stakeholders: In addition to the team providing subjective criteria for the machine learning algorithm, other providers are at the table prior to model development to advise on the output and how they want a risk factor to be surfaced during their practice. They are “developing the specific alert criteria of the risk identification algorithm through a range of best practice clinical guidelines.” This buy-in from the physician-scientists who have to rely on the tool ensures that the predictive model does more that than provide a good fit. It ensures that it will be given opportunity to improve care by the stakeholders.

Wrapping up

We have been reading some high-profile flops in using machine learning and advanced tools in healthcare. Surely these are due to some organizational and not just technological problems. But for the reasons I have outlined above, I am optimistic about the Melbourne proof-of-concept. they have a proven data set, a dynamic promising approach, and they know clearly the goal they want to achieve and most importantly they can empower the GPs who will utilize it.