At RSNA, Leaders Describe Significant Advances in AI Adoption in Radiology
Though progress is turning out to be very complicated in multiple ways, the adoption of artificial intelligence (AI) is definitely rolling forward across the medical specialty of radiology, as speakers at RSNA24, the annual conference of the Radiological Society of North America, documented in session after session on December 2 and 3.
A number of sessions at this year’s RSNA Conference, being held at Chicago’s vast McCormick Place Convention Center, focused on practical innovations taking place in patient care organizations nationwide right now. There seem to be nearly an unlimited number of possibilities right now, speakers emphasized; the opportunity lies in focusing on specific areas of potential and moving forward thoughtfully and strategically.
Let a thousand flowers blossom
In a session Tuesday morning entitled “Best Practices for AI Model Continuous Monitoring,” Walter F. Wiggins, M.D., Ph.D., a member of the AI Innovation Team at the Greensboro, N.C.-based national radiologist group Radiology Partners, and a practicing neuroradiologist, said that one key to success will be to take steps to validate processes at every step. “We start with a rigorous five-step clinical validation process in pe-deployment,” Wiggins told his audience. Further, he said, “we need to actively mitigate human-AI bias. So we teach the radiologists how the tools work, and how they sometimes fail. And then, post-development, we’ve been developing a system for continuous monitoring in order to combat performance drift over time.” What’s more, he cited Section 1557 of the Affordable Care Act, which next year will require patient care organizations to demonstrate that they are actively taking steps to combat racial, ethnic, gender, and age bias, And, he noted, the requirements under Section 1557 of the ACA are different from previous mandates around AI development, because these requirements are requiring action from patient care organization leaders, not just vendors.
The key to success, Wiggins noted, is constant testing and evaluation, from the pre-implementation phase through to the post-implementation phase. He cited an example involving head CT scanning. He and his colleagues have implemented an intercranial hemorrhage detection tool. Per that, they’ve used ChatGPT-4 to analyze the extent to which the AI tool has been matching radiologists’ findings in that type of detection. Early on, they recorded what he said can be considered fairly decent “case concordance” (instances in which the human radiologist and the algorithm come to the same conclusion: 63.9 percent of the time. Then, though, they’ve looked at patient age at the time of exam, and the concordance rate has varied considerably by age, with a 68.8-percent concordance rate in the 80-year-old-plus category, a 63.3-percent concordance rate in the 50-79-year-old category, and only a 55.9-percent concordance rate in the 18-49-year-old age category. Much work needs to be done in this area, he concluded.
Things got particularly tricky when Wiggins and his team looked at the data variables around race and ethnicity. Indeed, he said, “The biggest issue in that area was that we discovered that, out of 2.2 million head CT exams, 1.8 million involved a null value indicating missing racial and ethnic data. The data is sparse and messy. And that means that the key for us will be improving data collection.”
Putting patients at the center
Why AI development matters to patients is a topic that came strongly into focus during a session on Tuesday afternoon entitled “Improving Patient-Centered Care in Radiology Using LLMs: Opportunities and Challenges.” As Arun Krishnaraj, M.D., M.P.H., a professor of radiology and medical imaging at the University of Virginia, stated at the outset of the session, “Unfortunately, radiology reporting, even in the 21st century, still looks like it could be produced on a 20th-century typewriter. It’s filled with jargon and long lists.” And he cited the need for providers to rethink how they produce information connected to patient care, referencing Dr. Eric Topol’s book, The Patient Will See You Now.
“How have we responded? I’m showing my information from the University of Virginia Health”—from his Epic MyChart record—Krishnaraj said, pointing to a slide, and noting that his broken finger is referred to there as “XR Finger PA Lateral LT. We’re producing reports with open access to patients now, but we can do much better,” he insisted. And though most radiologists are aware that the 21st-Century CURES Act “focuses on releasing information to patients in a timely way,” he said, “When it comes to radiologists, are the reports you’re producing patient-friendly? Probably not.”
In fact, Krishnaraj noted, “We conducted a national survey to assess how familia radiologists are with the implications of the CURES Act. Have radiologists changed their practice? Will they make changes? 68 percent of respondents were either extremely or somewhat familia with the Cures Act. But 44 percent had a negative view of its provisions, versus the 32 percent who had a positive view, that we’re releasing reports to our patients. And 69 percent responded that the CURES Act had not changed the way they’re generating their reports, nor their length or content. Also,” he said, “68 percent were more concerned with patient anger than concerned about understanding.” Meanwhile, 40 percent of radiologists noted an increase in communication requests from patients, he said, and 31 percent noted an increase in requests from referring providers to added to modify reports,” including to correct such misstatements as using “meter” for centimeter—something that every physician would immediately understand was a mistake, but patients might not.
Perr all that, Krishnaraj told his audience, “There was a contest in 2000 in Wired Magazine, where they asked, how can you make blood work reports look more accessible and aesthetically pleasing? Based on that aesthetic improvement, we looked at the lung cancer screening report at UVA, to make it more acceptable to our patients.” He showed the audience an image of a patient-friendly report about low-dose CT scans that was easily digestible and informative, with simple text and lots of helpful graphics. Such communication “humanizes the work we do and helps patients understand,” he said. And the key point, Krishnaraj said is that “Large language models and custom GPTs will allow you to produce lay-language information for your patients, at scale.”
For example, Krishnaraj said, it will become commonplace for the radiologist to quickly and easily generate a patient-friendly version of a report on a cervical spine MRI without contrast, for example. And he referenced the following sample language: “In one part of your neck, there is a bone spur and a bulging disc that is putting pressure on a nerve.” That’s the kind of language that can be created for patients.” Importantly, he emphasized, “Patients are key stakeholders; they won’t want to wait.” Indeed, he said, the paternalism of physician-patient interactions is fading amid rising consumerism. “This will ultimately lead to shared decision-making, which will ultimately reduce unnecessary image-making. And generative AI, with LLMs, can help us achieve that goal,” he said.
Dania Daye, M.D., Ph.D., associate professor of radiology at Harvard Medical School and director of the Precision Interventional and Medical Imaging lab in the Division of Vascular and Interventional Radiology at Mass General Brigham, told the audience Tuesday that the three main areas of application for large language models in medicine will be in patient care, research, and education.
Daye cited the “limitations to the widespread adoption of large language models: “hallucinations, bias reproduction, misinformation propagation, and lack of accountability.” Indeed, she stated, “I want to start with a word of caution: LLMs have really great applications; however, we are not there yet. The biggest concern is around hallucinations. GPT can hallucinate and hallucinate competently. Our patients do not have the ability to distinguish between what’s right and wrong. ChatGPT is really trained on the Internet; and right now, there’s no way to hold GPT accountable.”
That said, Daye said, there is a very broad field for the application of LLMs to a range of clinical-operational purposes. “Usually,” she said, “the imaging-care process begins with someone in the clinic entering an order. There is a decision, then a radiology requisition, a radiologist protocol, and then the patient will be prepared, the imaging is performed, the radiologist will prepare and issue a report, and the report is then accessed. LLMs can be performed at every step of this journey.”
Indeed, Daye referenced an article in Radiology entitled “A Context-based Chatbot Surpasses Radiologists and Generic ChatGPT in Following the ACR Appropriateness Guidelines,” in which a study found that Chatbot provided substantial time and cost savings. She cited several other studies in the recent literature, including one that appeared in the October 5, 2023 edition of JAMA Network Open, entitled “Generative Artificial Intelligence for Chest Radiograph Interpretation in the Emergency Department,” in which the GPT-generated reports were found to be equivalent to radiologists in the ED and better than teleradiologists.
Indeed, that article notes that “Generative artificial intelligence (AI) methods, which generate data such as text and images following user direction, may bridge this gap [around the need to produce many reports very quickly in the ED setting] by providing near-instant interpretations of medical imaging, supporting high case volumes without fatigue or personnel limitations. An important advantage of the generative approach over classification methods is the ability to produce more informative and relevant outputs via generation of the entire radiology report, providing important context for decision-making in the ED.”
And Tessa S. Cook, M.D., Ph.D., of the University of Pennsylvania, who spoke next, on the topic of “Clinical Implementation of LLMs,” said that, “As a cardiovascular radiologist, I spend a lot of my time looking at aortas; and every time I open up a case, I spend ten minutes looking for who the ordering physician is, what they were looking for, etc. Generative AI could really help a lot” in that regard, she testified.
There are in fact a host of small tasks that could be automated, Cook said, including categorizing incidental findings, and automatically processing a study, given a particular clinical content: “There are measurements that could automatically be made for me,” she said.
Cook went on to share with the audience her “wish list” for the use of LLMs and generative AI:
Ø Patient engagement: patients can ask questions about their health and radiology care and instantly get lay-language answers.
Ø Decision support: LLMs can provide guidance to ordering clinicians so they can choose the examination most likely to answer the clinical question.
Ø Intelligent imaging: LLMs can facilitate automatic scheduling and protocoling so patients can get the right exam performed in the correct way at the appropriate site.
Ø EMR summarization: LLMs can provide intelligent search and summarization of a patient’s chart and prior workup.
Ø Custom reporting: LLMs can convert the radiologist’s report into a lay-language version for patients and customized versions for generalists and other-specialized specialists.”
And, Cook noted, “The team at Stanford are currently focused on clinical text summarization. They’re using a variety of different LLMs, some open-source, some proprietary, and applying them to four different types of documents: radiology reports, patients, progress notes, patient health questions, patient/doctor dialogue. They try to refine the output, doing a variety of quantitative evaluation and assessment, and take the best-performing models and approaches and give those to human resources to evaluate. Five radiologists and five hospitalists participated in this clinical reader study,” she said, referencing a recent study. Physicians were asked to evaluate the AI-produced summaries for correctness, completeness, conciseness. The best-performing LLMs were equivalent to human summarization. There’s potential there,” she emphasized.
