Many have noted that the emergence of artificial intelligence (AI) and machine learning (ML) in healthcare in the past few years has led to a dramatic unfolding of the Garner hype cycle, with a rapid rise in “inflated expectations,” followed by an equally quick slump into the “trough of disillusionment,” as wildly over-promoted expectations simply could not be fulfilled. That said, what is emerging now is the authentic development of AI- and ML-based algorithms, being applied to patient care and to clinical operations, if far more cautiously than anticipated, and through deliberative processes involving physicians and other clinicians, and lots and lots of testing of hypotheses, all of that activity evolving forward over time.
Our own 2022 State of the Industry Survey found that, as of the fourth quarter of 2021, 30.69 percent of respondents reported that their organizations had moved forward in some clinical area, while 22.77 had moved forward in an operational or administrative area, and 16.83 percent had moved forward in a financial area.
As for clinical applications, what patient care organization leaders are finding universally is that there are literally no shortcuts involved; put specifically, teams of clinicians, data scientists, and informaticists in hospitals, medical groups, and health systems, are developing algorithms, testing them, and beginning to deploy them, all one use case at a time, while engaging physicians and nurses from the start, and developing specific algorithms based on explicitly called-for use-case needs. As many are putting it, the idea of the equivalent of “stopping off at Target to pick up algorithms off the shelf” is simply not happening.
For example, participating in a Nov. 30 panel entitled “The Business of AI in Radiology: A Cost, a Long-term Investment, or an Immediate Business Opportunity?” that was held during the 2021 RSNA (Radiological Society of North America) conference, Nina Kotler, M.D., associate medical director at the El Segundo, Calif.-based Radiology Partners radiological group, noted that, in her group’s 3,000-radiologist nationwide practice, she and her colleagues have developed seven algorithms so far, and that, per the implementation of any algorithms, “You have to have a need,” a use case, that clinicians can agree on, in order to successfully develop any algorithm and get practicing physicians to use it.
Brian Patterson, M.D., a practicing emergency physician and the physician informatics director for predictive analytics at UW Health, the academic medical center-based integrated health system based in Madison, Wisconsin, certainly agrees with that perspective. While “I don’t think the door has been closed on generalizable models—we might be able to derive models across vast numbers of use cases,” Patterson says, nevertheless, “in many cases, it’s very fair to say that just because your EHR [electronic health record] model works in one place, it won’t necessarily in another.” And that means that AI algorithms will have to be validated in every patient care organization, by teams of clinicians, data scientists, and informaticists, because of the specificity of clinical information systems.
That’s exactly the reality that Suchi Saria, Ph.D., an associate professor and the director of the Machine Learning and Healthcare Lab at Johns Hopkins University in Baltimore, told an audience on Monday, March 14, during the Machine Learning & AI Forum for Healthcare, at the HIMSS22 conference in Orlando. Speaking of the challenges involved in analyzing AI algorithms, Saria noted that many patient care organizations have implemented AI algorithms for sepsis that have been developed elsewhere, but with suboptimal results. “I’ve seen incorrect evaluation,” she said. “People measured sepsis for mortality, then deployed the tool, then used billing code data, and evaluated. It looks as though you’ve improved mortality, but there’s a dilution effect.”
Formalizing processes—a key set of steps
In terms of the role that AI algorithm development will play within their organization, no U.S. patient care organization has done more to formalize that role than has the leadership at Cedars-Sinai Medical Center in Los Angeles. There, Sumeet S. Chugh, M.D., a practicing cardiologist who specializes in heart rhythm issues, is leading a new Division of Artificial Intelligence in Medicine, the first formal division of its type at any academic medical center. As a March 1 press release announcing the formation of the division noted, Paul Noble, M.D., chair of the Department of Medicine at Cedars-Sinai had asked Dr. Chugh to lead the division, which is already involved in using clinical trial processes to test several algorithms related either to predicting heart attacks or predicting sudden cardiac arrest. Chugh and a team of three other faculty members (two cardiologists including himself and two PhD scientists), plus a staff of about 25 individuals, including clinical researchers, software programmers, data scientists, and machine learning specialists, have all been working on testing the algorithms together.
After their initial development, all the algorithms are currently going through clinical trials-based testing. The intellectual and process rigor is essential, Chugh says: “There’s a series of steps between discovery and deployment; and that’s where the innovation comes in.” Testing clinical algorithms through clinical trials is time-consuming and requires effort, he says, but says applying that level of rigor to the process is essential in order to derive actual value from the proposed algorithms. “We approach everything with a clean slate. If we’re lucky, the timeframe could be two to five years; unlucky, it may not be a positive clinical trial. But it takes time to deploy in patient care.”
The biggest lessons learned so far? “The first lesson,” Chugh says, “is, be careful about garbage-in-garbage-out. The quality of the data, the harmonization of the data, and the curation of the data, come first. And then a good clinical question, a priori, has to be developed in parallel; if you come up with a question, you have to have really good data. And as you spit out an algorithm, it requires internal validation in your system, and ideally, external validation in a different system. And third, it cannot go into patient care unless we do actual clinical trials with these algorithms. Would you take a pill that hadn’t gone through clinical trials? I wouldn’t.”
Clinical operations are fertile ground
And it’s not only clinical decision support that is receiving the benefit; clinical operations, an enormous area in hospitals in particular, is finally now seeing the leveraging of advanced data analytics to its benefit. Steve Shirley, vice president of IT and CIO at the 318-bed Parkview Medical Center in Pueblo, Colorado has been facilitating just such an initiative there.
Partnering with the Santa Clara, Calif.-based LeanTaaS company, and leveraging LeanTaaS’s iQueue software, Shirley and his colleagues have been optimizing the crucial area of time management in the hospital’s operating rooms since February 2018. The solution uses AI to continuously analyze the time blocks that make up the core time resources map in the hospital’s 12 ORs on their main campus. So often, Shirley explains, surgeons are compelled to make last-minute changes that frequently leave whole blocks of time unused. Being able to analyze patterns and processes has helped the hospital’s flagship facility to recapture numerous blocks of time. Indeed, in the very first year that the solution was implemented, the hospital was able to decrease by 20 percent, as measured in minutes, the entirely unused, allocated blocks of surgical suite time, while releasing 15 blocks of time per month. At a time when Parkview, like virtually all U.S. hospitals, is under unprecedented financial pressures, such recapturing of time is immensely important.
Shirley cites a typical example. “Let’s say a surgeon has Wednesday mornings automatically filled for an add-on. An add-on can happen because of a trauma or an add-on,” he says. “And it certainly happens that you end up with an empty OR at times. But with the LeanTaaS, we’ve got the ability of our surgeons to add on block time on their mobile phones, and crucially, to release block time as well.” That flexibility has helped to convince surgeons not to artificially hold onto blocks of time that then end up being unused and unproductive for the hospital. There are two keys, though, to this kind of AI leveraging for clinical operations, he says. First, he insists, excellent project management must be involved in implementing any such solution. In their case, he notes, the hospital’s project management director was involved from the very beginning, helping to lead the change management aspects of the initiative. Further, he says, such gains can only be made if the CIO and senior healthcare IT colleagues are deeply involved with senior physicians and other clinician leaders in the organization.
In all these initiatives, one thing has already become clear: AI and machine learning can never exist as abstract, “off the shelf,” solutions deployed by HIT leaders in ways completely disconnected from clinician, especially physician, workflow. In every case in which success is happening, it’s happening because the CIO, CMIO, and all other informaticist leaders are interacting fully with senior physician and nursing leaders, and are implementing solutions with their involvement from the outset. In other words, “going to the AI implementation store” simply is not going to be the reality.
In the end, everyone agrees, the leveraging of AI and machine learning will continue to make progress, both in patient care delivery and in clinical operations, but it’s clear that its progress will be measured algorithm by algorithm, development team by development team, and patient care organization by patient care organization.
