On Friday, February 2, day 2 of the Health IT Summit in San Diego, sponsored by Healthcare Informatics, the event focused fully on cybersecurity issues, with a several presentations and panel discussions touching on important issues.
On Friday morning, Sri Bharadwaj, director, information services, and CISO, at UC Irvine Health (Irvine, Calif.), led a panel entitled “Ransomware Risks: What We Learned From NotPETYA and WannaCry.” Bharadwaj was joined by Stan Banash, CISO of Children’s Hospital of Orange County (Orange, Calif.); Chris Convey, vice president, IT risk management CISO, Sharp Healthcare (San Diego); Jason Johnson, information security officer, Marin General Hospital (Greenbrae, Calif.); and Christian Abou Jaoude, director of enterprise architecture and Scripps Health (San Diego). As has been widely noted, the May 2017 cybersecurity attack dubbed “WannaCry” grabbed storylines internationally and across the healthcare landscape as tens of thousands of hospitals, organizations, and agencies across 153 countries had their data held hostage, while the June PETYA/NotPETYA attack unleashed further damage worldwide.
“I was actually at a Healthcare Informatics conference” when the global WannaCry attack hit last May, Bharadwaj noted, referring to the Health IT Conference in Chicago. “I was speaking on a panel that morning, in Chicago, and this thing hit us. I got a frantic call, and I was on the phone call. For the first ten minutes, I said, OK, I’ll try to figure that out. That became six hours. I almost missed my flight home that day. It was one call after the other, providing updates, communication, etc. But we did not shut down the Internet, our Outlook, or any feedback back to the end users. We got the most hit from our medical devices. It was fairly easy to patch stuff and get stuff done, but we realized that our realm of exposure encompassed all sorts of things—who the heck knew that the parking system was running on a Windows 98G? Who knew that the cafeteria system was running an old version of Windows so old that we had to figure out what it was? So how can we learn from this?” he asked his fellow panelists.
“The key questions,” Banash said, “are, are you managing your risk? Do you understand your attack surfaces? What vectors are you vulnerable to? When this started out, no one knew what was going on; it was crazy. If you had one of those maps in your security center, it was all lit up, and it looked like ‘War Games.’ Initially, we thought it was via email, and we were chasing emails, but when we found out it was SMB vulnerability, we were able to chase that down. We were hit, but there was no successful attack on us. But understanding what was in your environment—it never became more important than on that day. And those MRI machines running on Windows XP—those machines are million-dollar pieces of equipment; it’s hard to justify new purchases to the board. I would say we were lucky; I’d like to say we manage things well, but we did get lucky.”
Asked about connections with law enforcement, Abou Jaoude said, “We do have a direct contact with law enforcement; we also have a protocol that we follow that’s been well-established. We followed those procedures, but the same thing happened to us: there wasn’t much information available during the first couple of days” following the WannaCry attack. “So I went out and read as much as I could about it, read articles to see whether there was something different about this. So we enacted that process, sent out notifications, and then a few days later, everyone learned what had happened.”
“I think we got lucky,” Sharp’s Convey said, “because this started in other parts of the world. Here in the US, we got lucky. I was at Millennium Healthcare then. SMB [Server Message Block] was blocked, that was the first thing. And then, how are our backups protected? And then patching. And it turns out, the basic security hygiene was needed. Look at what happened at NHS. And to be honest, we hadn’t patched as well as we could have. It’s hard to do, especially in the healthcare space, because you’ve got to test, and you don’t want to bring down patient care.”
“Let’s talk about communication; that’s one thing we’re always told we’re not good at in healthcare,” Bharadwaj said. “So, with regard to the oral process of communication, how did you talk to your c-suite?”
“As soon as we knew what was going on,” Banash said, “I reported to our CIO. But the first thing we needed to do was to kick off our incident response plan; and the easiest way to do that is to notify your management team. And, as you said, communication’s tough, we’re not always the best communicators. And in situations like that, you have to find a balance between over-communicating and under-communicating. You don’t want to be that CISO who’s freaking people out. As a CISO, if the only time you’re communicating with your c-suite is when something’s gone wrong, that’s not a good thing.”
“And in some respects, we won’t know what to communicate, because we don’t know what’s going on,” Bharadwaj noted.
“We went through that same struggle with Meltdown and Specter, which is that we didn’t really know what’s going on; no one did,” Marin General Hospital’s Johnson said. “So the main thing is communication. And when something like this is blasted out on CNN, everyone freaks out. I was getting calls from all sorts of people, nurse leaders, etc. I had to say, ‘Breathe, breathe.’ And I report to the CTO. And the c-suite told my CIO and CTO, you can do everything you need to do—if you need to shut things down, just do it. And so having that trust and confidence from the c-suite, was great. Because so often, we’re the redheaded stepchildren.”
Taking care of staff members’ home computing, too
“Here’s one thing we did that was different: we emailed everyone in the organization, and asked them to go check their computers at home,” Bharadwaj reported. “Many assumed we were going to take care of their desktops at work; but many had information stored in their home computers, which could get hit. So I told them, guys, go home, get your updates in, and make sure your home computers are protected. We will protect the computers in the hospital. So it’s important to communicate with your staffs. So a lot of people took time off in the afternoon, went home, and took precautions. We talked to the CEO about this, and said, look, I need to protect the organization, yes, but I also need to protect our employees. That was something we did that was a little bit different. We felt we needed to do that.”
“I’ll add to that: the lines are blurring between what we do at work and at home,” Banash said. “I have one mobile device that I use for work and personally. The lines are blurring. Things are blurring now: whether it’s remoting in; or whether your mom sends you an email at work, or you think it’s your mom, anyway.
“And we give them practical tips where they can print them out, take them home, etc.,” Johnson added. “We sent out a tip that said something like, here’s a secure storage place for your passwords, here’s how you should secure your home devices. We communicate with them regularly on those topics.”
“And we need to elevate this to a national level as well,” Bharadwaj said. “NH-ISAC announcements share information. I get pounded with information daily; I have to filter that and provide information to people. And after WannaCry, we got Petya/NotPetya.”
“We kind of treated the situation the same,” Scripps’ Abou Jaoude said. “And we had instituted an ongoing patching cycle—we have 3,000-4,000 servers, and we have a very rigid process that we follow, to patch things at least quarterly. The second thing has to do with our process of response in the wake of these incidents: we have a scripted, blueprinted process in place; meetings need to be called, reporting has to take place, on the IS side, the reporting side, etc. Chris mentioned about blurring the lines, but also, as we push applications into the cloud, into hosted environments, to same extent, you lose the possibility of having full control.”
“You bring up a very good point,” Bharadwaj said. “I was talking to another CISO, who expressed strong opposition to migrating to the cloud. And I said, look, we cannot sit back and become dinosaurs. But we’re still behind on things. And as we are hit with more attacks—and it’s nearly a year and a half since these big attacks—people are now getting paranoid—I’m seeing that happen. And I had a c-suite person send me an email saying, I got this email, but I’ not sure whether it’s really Bank of America or not. But that’s inevitable.”
Further, Bharadwaj said, “Meanwhile, switching gears from WannaCry and Petya, there are other things we can do. We advocate on network segmentation, to make sure we’ve done things right. We hope we’ve done network segmentation on the device side; but it takes a lot of time and effort. Can we talk about network segmentation?”
“When I think about segmentation, it’s data sensitivity, it’s locality, it’s location,” Convey said. “There’s an art and a science to it. Groups of IoT elements should probably be segmented off; medical devices should probably be segmented off. We’re looking at micro-segmentation. [In the manual network segmentation world], “[Y]ou have to go out and configure every switch and router, but now you can use logical networking, and software-based networking, and that’s where the industry’s going,” he went on. “And micro-segmentation is where things need to go. Because patching medical devices is a constant headache. So the logical choice is network segmentation and micro-segmentation.”
“We definitely had some struggles around segmentation,” Abou Jaoude said. “The question is, how can we break out the application from the use, from the site? We deployed firewalls in all our major sites, to make sure any outbreak would be contained and localized. And we tried to make sure that interdependencies in the apps traversed a very specific path. The biggest change and problem is when you present applications within the data center itself. A data center person might be working within Citrix and think an application is local to them when it’s not. So it’s not been easy.”
Unifying disaster recovery and business continuity strategy
When one audience member asked about how the panelists viewed the disaster recovery and business continuity aspects of this work, Convey noted that “The process of backing up is easy, but recovering is really, really hard.”
“From a business continuity and DR perspective,” Abou Jaoude said, “the biggest problem people have is that they back up their systems on tape or otherwise, and they back up and find out the data isn’t there. So we do regular data integrity checks to make sure the data is there. And we are 90-plus-percent virtualized. And the content is sitting in an alternate data center, so that we don’t need to reconfigure from a network configuration standpoint. It’s automated, not automatic. But one of the things we realized is that we need to have a pipe large enough to move the data and systems. And there should never be a scenario where you can’t use a manual process to get things done. You have to be prepared for that.”
“On the business continuity piece, one thing I’m working on now is to get senior leadership to understand that it’s not just an IT problem, and we need a business plan,” Marin General Hospital’s Johnson offered. “If I can only bring up 20 of our systems, [the members of the c-suite will] need to tell me which 20 systems to bring up. And one of our affiliate partners got hit with ransomware last summer; and they manage our billings for our clinics, etc. And their backups and everything got encrypted. So it’s 11:30 at night, and my CIO calls me and says, this company just got hit by ransomware, you’ve got to get back to the hospital right now. The security was actually the easiest thing to do. But then it took three weeks of 12-hour days to rebuild Allscripts from the ground up. And between that and the WannaCry, it’s helping our organization to understand that this is not an IT problem.”
Another audience member, Clark Kegley, the assistant vice president of information services at Scripps Health, who had participated on panels the previous day, said, “Historically, IT was assigned the disaster recovery part, and the business had the business continuity part. How do you bridge that gap?” he asked the panel.
“At Sharp,” Convey responded, “we go around and talk with the various leaders of the various segments of our organization. And part of this job is going around and really talking to the leadership and saying, we don’t mean to scare you, but this is where we need your help. There’s no way I can think of to approach this, other than with that level of cooperation.”
“About a year and a half ago, our board listed disaster recovery as a key issue for the organization,” CHOC’s Banash noted. “At that time, when I was brought into the conversation, the COO was already involved. And I was able to shift the conversation rom DR and to business resiliency. And in the end, they put together a steering committee for business resiliency; and we’ve created a new position for business resiliency, under the COO. And so that issue has been addressed.”
“We think of disaster recovery and business continuity only when there’s an issue,” Bharadwaj noted. “We think of the value of our car when it stops running; we think of the value of our phone’s battery when it dies. And so this is not a technology discussion; it’s a business discussion that has to happen at the c-suite level.”
