Back in April, the leaders of the Reston, Va.-based American College of Radiology (ACR) announced the launch of a specialized research unit to help promote and advance the use of artificial intelligence (AI) in diagnostic practice in the specialty of radiology. As the April 5 press release announcing the new venture explained it:
“The new American College of Radiology (ACR) Data Science Institute® (DSI) ACR AI-LAB™, a groundbreaking free software platform, will empower local radiologists to participate in the creation, validation and use of health care artificial intelligence (AI). The ACR DSI is committed to unlocking the potential of AI and helping radiology advance this technology throughout health care. As part of that strategy, ACR AI-LAB™ will provide radiologists with tools to develop AI algorithms at their own facilities, using their own data, to meet their own clinical needs. All of this will be done securely behind their own institutional firewalls. By combining ACR’s vast member network, its institutional connectivity to facilities and an industry community working to advance AI in health care, radiologists will be able to engage in all phases of the radiology AI development process.
ACR AI-LAB™ is an important step in an extraordinary ACR project that will help radiologists advance the use of AI throughout health care. ACR has been collaborating with industry, government and others throughout health care to promote a thriving AI ecosystem targeted towards patient and clinician needs. Standards, clinical pathways, education, and tools are all part of harnessing the potential of AI, and ACR AI-LAB™ is an important step in that journey. ACR AI-LAB™ “democratizes” AI by allowing for the direct participation of radiology departments throughout the AI development life-cycle. Through a freely available, open, vendor-neutral framework, radiologists will be able to learn about AI, contribute AI datasets, share AI algorithms, evaluate AI models, develop AI models and even combine these models though transfer learning and model ensembles to address their local clinical needs. The ACR AI-LAB™ platform will also support any future FDA initiatives that use real world data for local adaptive learning for algorithm improvement and monitoring of continuously learning algorithms.”
And a June update on ACR’s website noted that “A June update noted that “Radiology professionals from seven renowned healthcare institutions will use the ACR AI-LAB to demonstrate the process of creating investigational artificial intelligence models from image data without the use of a programming language. Using an AI model developed at one institution, each of the seven institutions will have the ability to evaluate and optimize the model for their own investigational use. Based on the recently announced ACR AI-LAB reference architecture,” the ACR stated, “this pilot represents a major milestone in the effort to allow institutions to develop high-quality algorithms that address local clinical needs, some of which may ultimately be made commercially available. In addition to the seven institutions, there are two major technology contributors; NVIDIA is providing software and edge infrastructure, and Nuance is providing last-mile integration to the participating radiologist.”
And, in the November/December issue of Healthcare Innovation, one of the participating leaders in the initiative, Keith Dreyer, D.O, is the chief data science officer at the Boston-based Partners HealthCare health system, and is vice-chair of radiology at Massachusetts General Hospital and Brigham & Women’s Hospital, two Partners hospitals located in Boston, spoke of the collaborative. Dr. Dreyer told Healthcare Innovation Editor-in-Chief Mark Hagland that “This is the wave of the future. We’re using this network to share information and insights. The problem in taking so long to build these models—there have been missteps. And then small companies that have no access to data—they grab general data and build a model. So what we can do because we have 20 billion images at Partners, Mass General and the Brigham, we have 20 billion increasing by 1 billion a year, we can build these models to improve patient care. That’s what’s new and transformational” about this initiative, he said. “These tools are on the verge of being in the hands of radiologists and clinicians. They understand what’s needed and the data, and the environments involved; they just need the models.”
Shortly after interview Dr. Dreyer, Hagland spoke with Mike Tilkin, CIO and executive vice president at the ACR. Tilkin is the staff leader of the ACR’s Informatics Commission, while Dreyer is its physician volunteer. Below are excerpts from Hagland’s interview with Tilking.
Can you explain where you specifically fit into this initiative?
Yes. I oversee the technology initiatives writ large. And the Data Science Institute falls within my purview. So I manage the team that operates the AI Lab. ACR is an interesting organization. We have this combination of volunteer leadership and full-time staff.
How many people are working on this initiative, from the ACR side?
The Data Science Institute has ten full-time staff. And then really, in excess of 100 volunteer physicians are doing various things with the DSI, working on use-case scenarios, including the AI Lab, which is one aspect. The AI Lab is a platform that engages the radiology community, and is an entry point for participation around AI.
So this is a first-of-its-kind AI lab, then, correct?
Yes. We recognized that we needed to get radiologists more engaged in AI, recognizing that some people were total novices, some had a moderate level of understanding, some were more advanced. And as we saw AI becoming more and more important to radiology, we had this concept of democratization. We wanted to create smart consumers and help radiologists to be able to navigate the world of vendor algorithms and work on problems. So our goal really was to engage ACR members. And we didn’t want to turn our members into programmers. So how could we facilitate their engagement?
The first phase was educational: let’s provide a cloud-based environment that could help radiologists, including novices, with a combination of educational tools, in terms of tools that can help you build an algorithm—including with easy drag-and-drop tools. And the next rung up was helping members to evaluate algorithms.
So our first release, in May, was a cloud-based version of the AI Lab toolkit, that allows folks to come to our data and do some training and do some hands-on evaluation; you could participate in competitions, etc. That was all cloud-based and aimed at novice users and folks who wanted to start to engage in deeper levels, but for educational purposes.
Our next evolution now is that we’re working on pilots for an on-premises version of this, allowing people to begin to evaluate algorithms using their local [organizational] data. So for folks who want to create an algorithm, what does that look like? And if I really want to create something that really can be used for clinical care, what does the regulatory landscape look like?
There are several hospitals and health systems actively collaborating?
Yes, we did an early pilot. We wanted to be able to enable collaboration, so one of our original pilots took an algorithm developed at MGH and deploy it at Ohio State, using Ohio State data, and work with that algorithm. And what happens if I want to take that further developed algorithm back to MGH?
So now we’ve got seven in the current pilot, and are getting them ramped up to participate in further activities. Those seven are Massachusetts General Hospital/Brigham & Women’s Hospital, Ohio State University Hospital, UCSF Health in San Francisco, Leahy Health in Boston, the University of Washington in Seattle, and Emory University in Atlanta.
What is happening among the seven?
Right now, we’re focusing on one algorithm with the goal of running it at all the facilities.
It’s a proof of concept?
Yes. You’ve got to deal with local connectivity issues, with the various logistics and legal issues. It’s a little more than of a proof of concept, as it’s getting infrastructure set up as well. The learning from this algorithm… we’ve already started stacking up additional algorithms.
Part of our goal is to evaluate what works well and what doesn’t in terms of things like transfer learning. Where are you improving the algorithm for everybody? The jury’s still out in terms of the literature around what’s working. It’s a breast imaging algorithm, evaluates breast density.
This is now about group learning on a broad scale, then, correct?
Yes. I think we see this technology as so disruptive, and moving so quickly, that the feeling is that if we don’t find ways to collaborate, and really answer questions quickly and lay the groundwork quickly, not only will we miss important opportunities, but we run the risk of false starts. There’s a concept of “AI winters,” where the AI doesn’t live up to the hype for a time. I think we feel that it’s important that, to not have a lot of misstarts, and to be successful, we have to quickly learn as a community. And so this becomes about logistics and people issues. So there is a scale to it that is pretty unique.
What would you like CIOs, CMIOs, and imaging informatics managers, to know about this?
I think they need to understand that this is important, disruptive technology that will definitely impact healthcare, will definitely impact imaging. I think it’s important that we be thoughtful as a community and that we become able to learn quickly as a community and put tools in the hands of physicians so that they can leverage what we’ve learned. Frankly, we need to encourage learning at scale, not only among institutions, but among vendors. We liken this a lot to the early days of DICOM. Prior to DICOM, you couldn’t assemble solutions that crossed vendor product lines, and you were limited in terms of the ability to leverage technology. With DSI and AI Lab, it’s a similar situation: how do you create an environment that allows innovation to thrive, and provides tools and options? I think what’s important for CIOs and CMIOs to understand is that this is happening, and we’re really trying to foster this community learning and evolution, etc., and what’s happening in this space. I think they’ll be better able to be successful.
