It’s that time of year: as I drive by my local farm stands, a special cherry stands out among the rest. The Rainier cherry. Pink and red, sweet, it’s a cultivar specific to the Pacific Northwest. This was a cherry that didn’t exist for me growing up in New Jersey. I don’t know whether they simply weren’t available in my locale or my family didn’t purchase them. I have, however, come to see that they epitomize some of the best work in healthcare machine learning (ML). My reasons are:
Locality. Rainier cherries, named for our local Mt. Rainier, are unique to this region. Solutions that rely upon healthcare ML models, as I’ll explain below, best achieve their goals when local nuances are part of the model.
Time-sensitivity. These cherries are available for only a few weeks each year. Similarly, healthcare ML models may reflect the time over which the data scientists trained the model. Models are also subject to drift, a scenario in which the input variables or predictive endpoint shift over time.
Availability. Rainier cherries aren’t just at the supermarket, or even the farmer’s market, but sold out of the backs of vans that pop-up on the side of the road. This speaks to solutions that deploy model outputs where the consumers are. Most everyone in healthcare IT know that it is near-impossible to pull a user from a primary working tool, such as the electronic medical record (EMR) or care management software.
2021 has seen an increasing amount of machine learning vendors entering in the healthcare space. I’ve been asked: “How should I vet these vendors? Who is reliable and how do I know if they’ll deliver a quality model?” I think about context. It would keep costs low, and installation straightforward, to deploy a pre-trained model. Pre-trained models can ship, and even test, with very high performance metrics. Many vendors will even tout that they have trained these well-performing models on comprehensive, longitudinal datasets. However, they are missing the Rainier cherries: the unique fingerprint that is specific to your data. The services you may offer your patients or members in collaboration with community organization that don’t fit quite right into your EMR’s structured footprint. The supplemental benefits your health plan offers - a value-add that has become increasingly common in the last few years among Medicare Advantage plans. Local details change the nature of events in your data, and your model should benefit from this information.
The rebuttal is natural: wouldn’t an off-the-shelf solution and pre-trained model be less expensive? Wouldn’t it be faster to implement and adopt? Absolutely. For some scenarios, these models are suitable. To continue my analogy, there is little downside to the pint of cherries you can find at the supermarket. However, you miss on the upside of discovering a sweet local varietal. Instead of asking your prospective vendor, “how well does your model perform in the lab?” try “how will your model suit my data, my unique needs, and my business patterns?” or “How will you ensure this model performs well on my data?” Seek a vendor who welcomes additional sources of data rather than charges you for it. This additional information is how your model will exceed expectations beyond model performance metrics. Ensure your vendor will seek out that local fit: to your data, your workflow, and, most of all, your users.