Merging AI Risk Management Into Patient Safety Reporting

Raj Ratwani, Ph.D., M.P.H., director of the MedStar Health National Center for Human Factors in Healthcare, recently described the number of errors and potential patient safety issues with new AI technologies as “staggering.” In AI digital scribe evaluations that his organization has done, they see multiple errors in each patient encounter. “When we say errors, what I mean is things like errors of omission, where critical information that's discussed during the encounter is not included in the draft note, or additions, where information that should not have been included is being included.”

Ratwani, who also is vice president of scientific affairs for the MedStar Health Research Institute, was speaking during an event co-hosted by the Duke Health AI Evaluation and Governance Program and the Duke-Margolis Institute for Health Policy that explored emerging best practices and policy approaches that support scalable responsible AI risk management and patient safety event reporting.

He mentioned that there is a lot of conversation these days around human in the loop. “When we look at simulation-based studies, where we've had physicians respond to patient portal messages with an AI-generated draft message produced for them and there's an error in that message, 75% of the physicians miss catching that error,” Ratwani said. “Traditionally, human in the loop concept thinking is that we have a physician reading the AI response, therefore we should be safe. Well, 75% of the time they miss it. And the point of that study is not to say “aha, physician, we got you!” The point is to say that we as humans generally are not very good at these vigilance-type tasks, so thinking of the human in the loop as a safeguard in all circumstances really isn't appropriate.”

Ratwani also spoke about the lack of a regulatory structure in place at the federal level that would support the vetting of safety of many of these technologies that are being pretty widely adopted. “I’m not saying that it has to be a regulatory structure. It could be a public/private partnership — any kind of uniform evaluation framework would be good to have, but it's currently not in place,” he said. “Part of the reason it's not in place is because these technologies are moving so fast that I actually don’t think some kind of federal policy would work well, because it wouldn't be able to be adaptive enough and nimble enough to keep up with the technology changes.”

But because there is not a set of guardrails in place right now, it ultimately falls to the healthcare provider organizations to vet these technologies for safety.

Taken together, he said, the prevalence of safety issues that he described with these technologies and the lack of any real safeguards in place “really pushes us to say we’ve got to think deeply about our safety processes at an organizational level.”

Moderating the discussion was Nicoleta Economou, Ph.D., the director of the Duke Health AI Evaluation & Governance Program and the founding director of the Algorithm-Based Clinical Decision Support (ABCDS) Oversight initiative. She leads Duke Health’s efforts to evaluate and govern health AI technologies and also serves on the Executive Committee of the NIH Common Fund’s Bridge to Artificial Intelligence (Bridge2AI) Program. She served as scientific advisor for the Coalition for Health AI (CHAI), driving the development of guidelines for AI assurance in healthcare, from 2024 to 2025. 


Economou said Duke Health has a portfolio of more than 100 algorithms that it is managing through its AI governance structure. These include tools used in patient care, for clinical decision support, note summarization, patient communications and those intended to streamline operations. These algorithms are either internally developed, bought off the shelf from third parties, or co-developed with a third party.

She noted that AI is moving quickly into clinical care, but the infrastructure to identify, report and learn from AI-related safety issues has not kept pace across health systems. “There's still no standard way to consistently detect when AI contributed to a safety event, a near miss, or even a lower-level issue that could become a larger problem over time,” Economou said. 

Existing patient safety systems were built for environments where humans alone were making decisions, Economou added. “Once AI enters the workflow, new kinds of errors emerge, and many of them are difficult to see using our current reporting mechanisms.”
The question is no longer whether AI will be used in healthcare because it already is, Economou stressed. "The question is whether health systems are prepared to manage its risks with the same seriousness we apply to any other patient safety challenge. Today, many AI-related safety issues remain invisible unless they're reported ad hoc by end users, and in many settings, there's no consistent way to link a safety event back to a specific AI system.”

This is important for three reasons, she said. First, AI can introduce systematic errors at scale, unlike a one-off mistake, and the error could be repeated across many patients and clinicians before it's recognized and without clear attribution to AI, patterns are easy to miss. 

Second, AI risk extends beyond obvious harm. It includes emissions, hallucinations, bias, workflow disruption, usability issues, and over-reliance — signals that often fall outside traditional reporting, but are critical early warnings.

Third, both patients and frontline users may not know when AI is influencing care, making it hard to recognize and report issues in the first place.

Integrating AI into patient safety reporting

So how are health systems thinking about merging reporting AI-involved errors or concerns into patient safety reporting? 

At MedStar, Ratwani said that in the event that there is a patient safety issue that arises from AI, either one that is a potential safety issue that somebody might raise their hand on or an actual safety event, MedStar has a mechanism built into its patient safety event reporting system for people to indicate that there's a potential safety issue. 

“Now I'll say, particularly from the human factors lens, that's a weak solution,” Ratwani stated bluntly. “That’s not going to catch a whole lot, and the challenge there is that many times, frontline users may encounter a potential patient safety issue, and they may not appropriately associate that with the underlying artificial intelligence. They may associate it with something completely different. So that poses some challenges. Nonetheless, we do need some kind of immediate safety precaution in place and some immediate reporting process. So that's what we have right now. What we are building toward is to have a recurring process for assessing these AI technologies —  very much like the Leapfrog clinical decision support evaluation tool. If you're working with Leapfrog, you can imagine something similar for the various AI tools we have in place.”

Economou described how Duke Health has established an AI oversight policy, establishing which safety reporting processes users should leverage. "For instance, if it’s safety-related, we are introducing a flag within our existing patient safety reporting system, so that end-users can flag whether an AI or an algorithm was involved,” she said, adding that they also have opened an issues inbox so non-safety-related events can also be reported centrally to the AI governance team. “On the back end, we're involving in the review of some of these safety events or issues some AI-savvy clinical reviewers. We can leverage the current patient safety-reporting processes, while also bringing the subject matter experts into the review of these events. Those reviewers will work collaboratively with those accountable for the solutions in order to do a root cause analysis, but then make their own determination.”

Finally, Ratwani mentioned the importance of aligning incentives between health systems and vendors. “If you look back to what's happened with electronic health records as a model, there's an asymmetric risk relationship there whereby the provider and the healthcare system really hold all the liability, right? EHR vendors typically have a hold-harmless clause built into the contracts, and the responsibility falls on the healthcare provider organization,” he said. “I see a similar thing happening with AI technologies, where states are passing regulations that put the burden on the provider organizations. If that continues, that's going to be a really big challenge for us, because it's going to limit our uptake of these technologies. What we want to do is have a shared responsibility model. Those that are contributing to safety issues should be held accountable, and we should all be fully incentivized to ensure safe technologies. I think some correction in terms of that risk symmetry is going to be really important to move us forward.”