How Are CMIOs Assessing ROI of AI Projects?

How are health system clinical IT leaders responding when AI solutions don’t demonstrate clear efficiency gains to show a CFO? Several chief medical information officers recently described their thinking when measuring the return on investment of projects such as generative AI scribes and tools to help draft responses to patients’ portal queries. 

During a Dec. 9 meeting in the ongoing Harvard Clinical Informatics Lecture Series, Rebecca Mishuris, M.D., M.P.H., M.S. CMIO and vice president at Mass General Brigham (MGB), discussed her health system’s experience. In evaluating tools to help with patient message in-box management such as Epic’s MyChart In-Basket Augmented Response Technology (ART), Mishuris noted that the literature has found that only between 5% and 20% of the drafts are used. “People were really excited about it when we first piloted it over a year ago, but we found that only about 5% of the drafts were used, and we saw no change in response time, and no change in length of response,” she recalled. “The literature does show some reduction in cognitive burden, which can certainly help reduce burnout. But if we are only using 5% to 10% of the responses, is that really a significant reduction in burnout that is worth the expense?” 

At least in the Epic world, Mishuris added, you pay for every draft, whether someone uses it or not. Each draft doesn’t cost much, but they add up pretty quickly in large health systems. “With that low usage, we were not confident that we were actually making a difference in our providers’ lives,” she said. 

MGB chose to pivot away from physicians and nurses to front desk staff using the tool. “Our prompts did not allow the model to give clinical advice, so it was less useful to the people who would be giving clinical advice normally, but maybe useful to the front desk users,” Mishuris explained. “I would say we saw a little bit more usage there, but even there it was not the usage that you would want to see to know that you're really having an impact. We were following: how long did it take someone to write the message? How long was the message itself? How long did it take us to get back to the patient? Those are indicators of whether this tool was helping us with our challenge of in-basket management and patient experience, and we weren't seeing it yet.”

There's no right answer for how much it should be used, Mishuris added. “I think we can all agree it should be used more than 5% to be worthwhile, right? But is 20% enough? Should it be 50% or 90%?  I don't know.”

Nancy Cibotti, M.D., chief innovation officer at Beth Israel Lahey Health Primary Care and associate CIO at Beth Israel Lahey Health, responded that when her organization first rolled this same tool out, they found that advanced practice providers used it three times more frequently than the physicians. “We also found that most of our our requests go through our clinical support pool, so it's really our nurses, our medical assistants and front desks who are seeing these messages and have found it to be very useful. I think this is going to get better.”

Moving beyond the AI hype

During a Dec. 10 AMGA-sponsored webinar, Flora Zarcu-Power, M.D., director of primary care for Yale/Northeast Medical Group, was asked how she thinks about the medical group’s AI investments.

“What I am looking for in 2026 is to move beyond the AI hype and to ensure that our investments in AI adoption translate into measurable improvements for patient care, for clinician experience, and, of course, for organizational sustainability,” Power responded. “In 2025 we've seen tremendous experimentation in ambient documentation technologies, predictive analytics, and workflow automation, but the real challenge was to actually prove ROI in ways that matter to both front-line physicians and leadership, CFOs particularly.”

Power described three lessons from 2025 that are shaping her thoughts for 2026: First, ROI should be multi-dimensional. “Financial savings are certainly very important, but equally critical are actually reductions in clinician burnout, improved patient satisfaction and stronger workforce retention,” she said. “For example, AI scribes in 2025 may not have proven cost savings quickly, but if they prevent a clinician from leaving practice, that's a major ROI win. Second, evidence matters. We've seen pilot projects in 2025 that did not have clear evaluation frameworks and they struggled to justify scaling. So in 2026 I’d like to see organizations define success up front, whether it's fewer no-shows or faster documentation or improved care coordination, and then measure against those baselines.” Third, she added that equity and access cannot be ignored. “I think 2025 highlighted the digital divide, and organizations that were well resourced improve quickly, while safety net providers eventually risk to fall behind. So for 2026 I believe ROI must account for inclusivity and making sure that we use AI tools that serve diverse population sets and address disparities.”

Modest time savings from AI scribes

The Harvard Clinical Informatics panel also turned to assessing the ROI of ambient scribes, and Jonathan Hron, M.D., associate CMIO at Boston Children's Hospital, noted that although clinicians are eager to use ambient documentation, “the thing that fascinates me is that it's not really saving that much time. It looks like the time savings are really quite modest, if not negligible. But consistently across studies people are reporting being less burned out, with less cognitive load in the process. Is that enough to justify these increasing costs of ambient dictation software?"

Beth Israel Lahey’s Cibotti said a big part of the ROI comes from the quality of the visit and the documentation. “I open up my computer, I get consent and put my computer aside, and then you're having a conversation face to face with the patient, and it is just a very different experience. I think that's tremendously valuable. It's hard to capture, though, what that value is,” she added. “There are people who find this to be a true game-changer. They're not doing after-hours work. And there's some data to suggest that the more you use it, the more it will save you, and the better you get at it. I think that this is something that will become standard, and sometime in the not-so-distant future, all outpatient visits will include ambient scribes because it is a better experience, both for the clinician and the patient.”

Mishuris said she thinks it depends on what problem you set out to solve.  “We set out to solve one of our biggest challenges in healthcare, which is around clinician burnout. And our own internal data and data from many other organizations across the country now show pretty significant reductions in clinician burnout,” she said.
 “For many organizations, seeing a reduction in burnout was sufficient, but for the organizations that wanted a hard ROI from ambient documentation, I think that the data we're seeing around the modest reduction in time does not bear it out. The people who were hoping that they could add patient visits to the provider's schedule to pay for ambient documentation — if you ask me, that's the antithesis of trying to solve provider burnout. It's like, we're going to give you this tool that helps reduce your burnout and then give you more patients. I'm not sure those two things actually go together, but there are health systems out there that are asking for that.”

During a Dec. 9 Permanente Medicine webinar, Kristine Lee, M.D., associate executive director of The Permanente Medical Group, was asked how Kaiser Permanente leaders determined which digital health tools to adopt.

She said it comes down to being very clear about the problem that you're trying to solve. “When I think about our journey with AI and digital technology adoption, a couple of things really rise to the top. One of them is physician burnout,” she said. “We also think about workforce capacity and our workforce shortage. How might a new technology or a new tool be able to address that problem? We have really tackled physician burnout as our No. 1 pressing problem to solve, and I will also say that there's no such thing as one tool that's going to solve that problem by itself. So we've been taking it piece by piece and day by day and tool by tool, and really doing that evaluation and measuring the ROI in terms of what impact is this making? Having a very clear measurement strategy is very important.” 

Appearing on the same Permanente Medicine webinar was Caroline Pearson, executive director of the Peterson Health Technology Institute, where she leads efforts to evaluate the clinical and economic impacts of digital health solutions. She said the first question to ask is ROI for whom? In terms of thinking about the health system as the most common purchaser, there are two theories on where we could see revenue gains as a result of the ambient scribe solution, she said. 

“One theory was that if note-taking gets a lot faster, can you basically get more patients into every clinic? I would say we've seen very little adoption of that as a solution,” Pearson said. “Generally, you're not seeing time savings enough that it would cause folks to do that. Given the burnout, most systems have not been focused on trying to make visits shorter and add patients to every clinic. But we do see some emerging evidence that some of the coding around those notes may be more complete, so providers may be incrementally increasing their billing for every patient as you're getting better capture of the encounters in the room.” 

Pearson added that additional coding is obviously something that as a philanthropy focused on healthcare costs Peterson Health Technology Institute wants to watch carefully. “We want every visit to be coded accurately and completely and ensure that patients are getting the care that they need,” she said, “but we also don't want that to create an escalation of billing that just increases total spending without materially changing the care folks are getting.”

Looking ahead to 2026

These sessions wrapped up with execs being asked what they are thinking about for AI in 2026. 

Yale/Northeast Medical Group’s Power said she is focused on the concept of sustainability. “AI requires monitoring, recalibration, and continuous governance,” she said. “Too many organizations think that AI implementation is a one-time project, and that is not the case. It's actually a living system that requires stewardship, and I think that organizations that are successful in implementing it are aligning workflows with care teams and with the culture of the organization. If those things are not aligned, I think success is less likely and adoption could stall.”

On the Harvard Clinical Informatics panel, Chase Parsons, D.O., CMIO at Boston Children’s Hospital, said he would like to see more agentic AI implemented in healthcare, and Mishuris said MGB could really use tools to help with clinical care orchestration — “everything from acquisition of external charts to summarizing those charts to making sure that the patient is seeing the right specialist based on whatever their need is.”

Asked which AI-related project she was most excited about, Permanente Medical Group’s Lee described a chronic healthcare pilot under way that is exploring digital twin technology, using remote wearables including a continuous glucose monitor, a wrist device that monitors activity and sleep, a blood pressure cuff and a scale with the patients logging their food. “We are collecting over 3,000 data points per day on an individual and studying and learning their metabolism. There is an AI algorithm in the background that's learning how you respond to that cheese enchilada at 3 p.m. in a way that a primary care physician would never be able to deeply understand,” she said. “We are creating a digital twin of that person, and then surfacing insights back to that patient that are very personalized.” 

Lee said that she is thinking about this project as an adjunct to their population health management programs. “We have chronic condition management teams that are doing a fabulous job, but maybe I only get 90% of my patients with high blood pressure under control. What about the other 10%? Could this be a way of coming up alongside what we do traditionally in our population health management systems and personalizing this to the patient. We're about 60 days in, and we’re seeing great results. We're seeing A1Cs come down and people's weight falling also, which is great and there’s more to come.”