Penn Medicine’s John Donohue on the Subject of Infrastructure Resiliency

When it comes to securing the foundations of the information technology infrastructure of patient care organizations, there are many possible approaches. One concept that is gaining greater currency these days is that of “infrastructure resiliency.”

And one senior health system executive who has been strategizing around that concept for some time now is John P. Donohue, the associate vice president of enterprise infrastructure services at Penn Medicine, the six-hospital integrated health system based in Philadelphia.

Donohue, who has spent over 30 years in IT management, is focused on IT infrastructure issues at Penn Medicine. Recently, he spoke with Healthcare Informatics Editor-in-Chief Mark Hagland regarding some of the current challenges and opportunities that he and his team are addressing at Penn Medicine. Below are excerpts from that interview.

One of the concepts that you’ve been working with has been that of infrastructure resiliency. How do you define infrastructure resiliency?

From my perspective, infrastructure resiliency is about developing a clinical-grade capability. It’s resilient to any kind of an outage. Think about an operating room suite—you’ve got emergency power, backup, tools, etc. The suite is built to be fully resilient, so that any kind of impact doesn’t create risk for the patient. To me, then, infrastructure resiliency is about creating an infrastructure impervious to power outages, and/or failures; it’s often referred to in data center speak as ‘N+1.’ In a data center setting, you have two generators, and so if one generator fails, you’ve got another one. Same type of thing with air conditioner units. So everything has resiliency, so that if you lose a key component, it doesn’t endanger your operation. For example, when we install network links, we’ll typically install a backup link, so that if something happens, you don’t go down. That’s what the concept means.

John P. Donohue

Where are U.S. patient care organizations right now, with regard to their forward evolution around infrastructure reliability, and where is Penn Medicine, along that dimension?

I would compare us at Penn against just about anybody else, in terms of our investments. We started to focus on this about four years ago. And you can’t just flip a switch or write a check—creating this kind of infrastructure requires a multi-year investment, a plan, and an architecture that’s incredibly sound. It’s hard to quantify something like IR; I can tell you that it’s dramatically dropped outages in our organization. I’d be hard-pressed to say there are others out there ahead of us.

What are the biggest challenges in achieving infrastructure resiliency?

First, quantifying its benefits or its return on investment. People just get used to it. It’s like the light switch or the water fountain. And it’s hard to quantify across the board what a network outage would mean across the board; so, quantifying the ROI is challenging. The second thing is creating a plan that allows the infrastructure to be refreshed on a regular basis. One issue is the funding model; you have to have a good model, and then the intestinal fortitude to stay with it. And it has to be carefully planned and executed; it’s like fixing a wing while you’re flying a plane. Because if you’re not careful, you could harm your organization.

What have the biggest lessons learned been so far, for you and your team, on this journey?

I think the biggest lesson is that our clinicians and staff are no longer as good at doing things manually. I want to explain this carefully, but, for example, five or six years ago, the nurses were totally prepared for system shutdowns. But the systems are so reliable now, and so free from shutdowns, that our clinicians haven’t had practice with these kinds of situations for a long time now.

What are your strategy and philosophy around disaster recovery?

I think there are some nuances that make disaster recovery different from one organization to another. Conceptually, you can say, we should be back up at a certain point in time, etc., but every organization has nuances that make the planning different. We implement disaster recovery solutions, we vet them quarterly; we do tabletop exercises, dry runs to make sure you’ve in good shape, etc. We’ve been working more closely with the business, on business continuity—which applications come up first, which data recovery is focused on first. So we’ve been helping the business organization to focus on their needs, and what tools we have available to help them recover.

One of the issues that has come up in many discussions is what seems to be a fairly big disconnect between disaster recovery on the one hand, and business continuity, on the other. How do you see all of this? Would you agree that there is a disconnect in many patient care organizations between disaster recovery and business continuity?

I do, and it’s complex, and it’s expensive, and the ones that have unified the two have probably experienced some kind of disaster or scenario that’s helped them realize there’s some pain in not having something in place. So I’d like to think that we’ve been somewhat proactive in tying together business continuity and disaster recovery here.

What are some of the key aspects of achieving success in this area? Is gaining consensus among stakeholders one element?

Two things. First, one has to determine what one’s organizational risk profile is. And that’s a process, right? Some organizations will be more risk-tolerant than others. So you have to measure your risk tolerance. And we have a very strong governance structure here, so the “what’s first, what’s second?” really came about naturally here, because of our strong governance culture. And it doesn’t feel great to be the lowest priority on the list, but it’s not personal. It’s about the business, really.

What kind of advice would you offer to your fellow healthcare IT leaders, in terms of making this work?

My boss has a filter he calls the three Cs: common systems, centrally managed, collaboratively implemented. And it’s the last one that’s important conceptually to this conversation. There aren’t ‘”IT systems,” there are Penn Medicine systems, and all the stakeholders in the organization need to have skin in the game. The business has to buy in; the organization’s business leaders can’t point the finger and say, this is IT’s project or problem. Once you have the business people sitting at the table with you as your partners, that’s where you get that buy-in; the buy-in has been pretty strong here at Penn.

Is there anything you’d like to add? And where do you see your team, and the organization, headed in the next year?

With regard to where we’re moving in the next year, I have a few different thoughts: one is continuing to drive the organization up the maturity scale from a capabilities perspective; two is being more proactive and less reactive; three is being more innovative. Gartner talks about “bimodal”—being able to run highly resilient advanced clinical applications, while also considering innovating at the same time. You have to keep things running, but also think of innovative ways to improve things. So it’s balancing innovation and good operations. Finally, as we prepare for a new $1.5 billion patient pavilion we’re building, and as we continuously pursue new projects, we need to figure out how we scale all of this, how we manage our people and processes, to make it scalable.