How secure is your disaster recovery plan?

HMT: Despite the existence of more vulnerabilities and cyber threats than we can count, it seems a lot of disaster plans are more concerned with power outages and physical damage. But, shouldn’t a disaster recovery plan include cyber attacks?

David Finn: I think this is a great topic, and I’m a little bit surprised we don’t see more about it, because healthcare really does require a different level of cybersecurity being built into your additional disaster plan. In the old world of DRs, you’d get a hardware failure or some kind of natural disaster, but now we’re looking more at cyber events – cyber incidents or attacks – and preparing for that kind of disaster, as well as the others.

Over the years, healthcare has built a level of redundancy into their systems and networks where organizations can generally get through those traditional disasters that used to shut systems down. You still have to be prepared for them, but it’s going to be the cyber incidents that catch people off guard, and that’s why it’s so important to build them into a DR plan. Frankly, a cyber attack is way more likely to happen than a major power outage or a hurricane hitting your data center.

Is part of the issue that healthcare systems simply aren’t aware of the magnitude of the threats, in terms of the overall number that exist and their seriousness? How can they stay up to date on all of the latest concerns?

Yeah, we’ve really struggled, I think –not only as an industry, but across all critical infrastructure sectors with sharing that type of information. Sharing the latest security information is always a little bit tricky.

Some may want to subscribe to a service that provides updates and information most relevant to their industry.

If you’re in healthcare, it may be interesting to know what’s attacking banking, but it may not be as important as getting information from the National Health I-SAC or your neighboring hospitals in the area – what are they doing? What kinds of attacks and problems are they seeing, and how do they protect against it? I wish there was a really good answer today, but there isn’t. I think you’re going to see a lot of evolution in terms of information sharing over the next 18 months.

For you as a service provider, what’s your role in doing that? How do you make sure you have the latest information available, so that you can then give that information to your clients? And can Symantec see new threat trends before they emerge?

Because of the size and scale of Symantec’s operations globally, we actually get a lot of advanced intel. We kind of have “canaries in bird cages” – we have honeypots globally. Also, we actually utilize our Norton information from users who are not behind a corporate firewall or a corporate system – they just have their machine at home, their laptop, or their cellphone. Through that Norton intelligence – and that’s about 60 million users – we can see a lot of what’s happening; we can detect good and bad files. We’re seeing a lot of that intelligence very early on and correlating it to build on our threats and vulnerabilities databases, which we then use in our security operations center and share with the organizations who use our services.

So the consumer and corporate worlds are coming together to paint a more complete picture?

Absolutely. It’s kind of interesting, we only started doing that about four or five years ago! (laughs) The V8 moment – you slap yourself on the forehead. Looking at the stuff people are doing behind a corporate firewall isn’t always valuable. But when you look at 60 million consumers who are out there running naked in the wilds of the Internet, you see a lot more and are able to act earlier, before a threat starts really becoming something that’s targeted at enterprises.

Switching gears a bit, what should a hospital or health system do if they think there’s been a security breach but they have yet to confirm it? Should they report it and risk media exposure before they’re sure, or should they go to their security servicer first, before reporting it as a potential HIPAA violation?

Well, if they were one of our customers they certainly could come to us. You know, back 15 years ago everyone had a disaster recovery plan, and you knew what the procedures were. Today, before you get to the disaster plan, you actually have to have an incident response plan and, if you’re a big organization, you may have that team in-house to officially assess what happened and determine if it was actually a breach and if it needs to be reported. But even if you don’t have those teams in-house – and I would recommend you do – you need to have an incident response firm on retainer that can come in and help you plan and manage that response, and do some of the initial forensics to determine what happened. And then escalate that to the broader teams – public relations, legal, and all the people who would get involved if it truly were a breach or some other event big enough where you needed to share that information with the community or the Department of Health and Human Services (HHS).

Speaking of the HHS, I see they have issued a statement on ransomware, calling it a breach. Is that the final world?

The HHS at this point is considering ransomware to be a breach. I don’t think that’s definitive. I actually just moments ago got off of a phone call with a whole group of CISOs nationally who are developing a response to that guidance.

There certainly is no legal or court precedence for it being a breach. And I think if they actually went to court, they’d be hard pressed to prove the data had actually been breached. Just like our earlier discussion,¹ you can do a lot of things to data without ever looking at it. In fact, I would argue that if it’s encrypted and even a hospital can’t look at it, that data may be more protected than it’s ever been because no one can get to it. You don’t have to view or access files to encrypt them.

Can’t this be solved by looking at the code for the ransomware in question? It seems to me that would resolve the debate, as it wouldn’t be a matter of opinion at that point.

Yeah, that’s really the discussion. HHS has previously “clarified” the rules for what a breach is and how you determine if you’ve actually been breached. And yet, they then issue this guidance saying that if you’ve been hit by ransomware, you’ve been breached. I think if you go back to the earlier breach-assessment process, you’ll see some contradictions.

And they’re right – some ransomware versions may read the data and encrypt it, and I’d say that’s a breach. But with other ransomware – these guys don’t care about accessing data. They want to encrypt files very quickly and act stealthfully. And you’ll find the data in those cases has been breached. When you do full-disc encryption, you’re not accessing all those files to encrypt everything on the disc.

During our last discussion,¹ you said you weren’t aware of any ransomware that actually viewed or retrieved files. Is that still the case?

At that point, that was the case – with all known ransomware we had seen, that was the case. I wouldn’t say that is the case today, because I am not currently up on all the new ransomware. But I’ll put it this way – I am not aware of any.

There is a new ransomware that tries to exfiltrate data, but that’s a fairly new one. But, if you have all of your other protections set up right, you’re going to see that the exfiltration doesn’t fit with the rules for sending data outside the network, and it would be stopped. But yes – there is at least one that I know of which tries to exfiltrate data.

Worst case scenario, say you’re a hospital system that doesn’t use cloud hosting and you haven’t done your proper backups – and you find yourself attacked by ransomware. How do you get up and running again, since I’m sure there’s no way to guarantee paying the ransom will free up your network, is there?

Oh man, that’s the $64,000 question. And let me first say, there is no guarantee you will get your data back. And that’s why it’s so important to have a knowledgeable response team. For example, we’re seeing something that can only be referred to as “scamware,” where the program locks your screen and gives you instructions for paying ransom, but it actually hasn’t done anything except lock your screen. If the user simply turned their computer off and restarted it, everything would be fine.

That said, if you do get a real case of ransomware, there’s no guarantee that paying the ransom will set you free. We had a couple incidents in the last two months where ransom was paid and then the cybercriminals came back and said, “Oh yeah, that was ransom number one – now it’s going to cost you this much more.” So there is no guarantee that paying the ransom will get you your data. It’s not ever a good option, but if that’s the only thing you can do in order to take care of patients, I certainly understand why hospitals have paid the ransom in the past. But it’s not a good option.

The best thing you can do is have detached backups and files that you could restore. You may have some data loss depending on how you’re doing the journaling and the backups, but at least you could get back to caring for patients without paying a ransom.

I’m not sure I’ve heard of this in healthcare, but is there any kind of situation that has arisen where someone has attacked both the main system and a backup, locking them both down?

Yeah, I’ve seen it in healthcare, but I can’t recall the exact situation right now. The hospital got the ransomware attack, and they thought they were safe because they had backups. The IT department fired up the backups, and when they tried to access the backups to restore their data, they got a message saying their backups had been encrypted and they needed to pay a ransom. Some of the new ransomware is actually pretty clever in that it looks at the system and where the system lays down its backups, and then it goes and encrypts the backups first.

The solution is to keep the backups off your main network. So even if the ransomware knows where they are located, it has no way to connect to those backups.

HMT: When we talk about disaster recovery plans, is there a different strategy for every event? Say you’re in California – would you have a specific plan for earthquakes?

David LeClair: Generally speaking, most of the technology portion of disaster recovery planning is pretty similar regardless of the type of disaster. The things that are different have to do with the surrounding portions of your plan. How do you operationalize your team? How do you actually empower people to work remotely if they need to?

Obviously, certain natural disasters are going to be more prevalent in certain parts of the country than others. Here in the Northeast, we have to worry about “Snowpocalypse.” Snow is a natural disaster that builds up over six weeks, to the point where people have roof collapses in data centers and things like that. It’s not the urgent thing of, “Oh, there’s a wildfire that’s going to take out my building, and now I’m scrambling.” You can see Snowpocalypse coming, but the impact of downtime is the same, whether it’s a slow-moving disaster, like snow, or a sudden one like a tornado.

If a hospital system is aware of an emergency event coming their way, what is the very first thing that they should do, and how do they decide what the list of priorities should be?

A lot of what needs to happen actually needs to happen on a day like today, when it’s sunny and beautiful out. That’s the time to make sure you have your full disaster prep plan squared away. You need to know now how you’re protecting your systems. Who is responsible for doing what in the event of a disaster? What are the procedures for doing things when things go wrong or if someone isn’t available – you never know if a disaster is going to hit when somebody is on vacation. You need to think through as many scenarios that can go wrong for you as possible well in advance of a disaster.

In a hospital’s case, there’s an awful lot that goes into that preparation. They need to start thinking through all of their operations. I’ve worked in the past with healthcare institutions that are responsible for doing cancer treatment. You’d think, if they have a downtime event, it’s not that big of a deal. It’s not like they’re running an emergency room. They said, “No, it is. We schedule cancer treatment centers to run 24 hours a day, seven days a week. If we have a downtime event and we can’t process somebody through because we’ve lost the server, those are treatments that cannot be made up.” For them, those are people that are going to be missing critical treatments.

They’re booked that solid?

Yeah, because they’re booked that solid. What I’ve found is that every single business is unique, and they all have critical things that simply can’t go down. It’s different for every business. That’s the reason why it’s very important to look at your own business, look at your own processes, your own procedures, your own systems and come up with your own very, very specific disaster recovery plan for your specific enterprise.

If it’s a situation where a healthcare organization or hospital is physically damaged, would cloud be the only way to ensure that all their data is backed up?

Yeah, cloud is a good solution for them to get up and running. Whether that’s they need to get up and running for a few days or for a longer period of time, it is a viable solution for doing that. Most DR solutions generally have a time limit before they start charging you additional fees. Cloud is a very viable solution. The one thing that we caution just about everybody on – and one of the big challenges as you see in DR – is that companies don’t tend to test it sufficiently.

We actually did a survey of almost 1,000 IT executives a few months ago, and one of the questions we asked them was, “How often do you test your DR environment?” Well, over half either test it once a year or never. The problem is that, if you’re only testing it once a year or never, you have no idea if your DR environment is going to function properly.

Unitrends actually has some technology that allows you to automate that testing, and automate it so that it runs on a regular basis. We can actually run our recovery assurance tests hourly, daily, weekly – however frequently a company feels they need to run it.

We actually will boot up the DR environment in a sandbox. We’ll actually validate that all of the applications that you have there are working properly, we’ll test interfaces, we’ll test the networking, and then we will actually deliver a report to the admin that certifies that they had met their recovery-point objective and their recovery-time objectives in bringing the data center back online.

As a disaster environment provider, what does Unitrends have to do to make sure they’re following certain protocols and ensure HIPAA isn’t violated? Is there anything specific?

We run audits. As a matter of fact, one of the things that we advise to anybody – no matter what cloud solution they’re looking at – there’s a report that you can ask for called an SSAE16 SOC2 report. Essentially, it’s a report that audits your security protocols, your data protection, and it looks at not only security for your data, but it also looks at the physical security of a vendor’s building, such as if they have biometrics. Any reputable cloud vendor should be able to supply that to you. That’s a very good first step to assure that you’re dealing with somebody who knows what they’re doing. It’s one of those things that no one knows about to ask, but every cloud vendor who is doing what they’re supposed to should have one of these. If they don’t, get worried.

Editor’s Note: The above was edited for clarity and concision.

Reference:

1. “Ransomware: It’s as scary as it sounds” Health Management Technology, Vol. 37, No. 4. June 2016.