The time to prepare for a disaster is before it occurs. I know that may sound very basic, but during my more than 30 years in the technology industry, I’ve seen hospitals and other companies struggle to get back on their feet. It’s not pretty. It can be heart breaking due to the costs involved and the loss of reputation a company endures – and, in some cases, can’t recover from.
While technology certainly has ma de tracking, diagnosing, and treating patients easier, these systems are particularly vulnerable in disaster recovery (DR) scenarios. Oftentimes, patient records are trapped in electronic files that can’t be accessed, or they have to be printed out and taped to the chests of transferring patients. The cloud isn’t necessarily the answer to your DR woes, since those connections aren’t guaranteed either.
The following nine steps will help you lay a solid foundation for your disaster recovery plan.
1. Meet with your staff and the business and clinical leadership to understand how you will communicate during an emergency. That discussion should include how to get a status of the whereabouts and availability of key staff. A datacenter fire on a weekend means that your junior IT staff may be the ones on duty.
Step 1 in a disaster is to determine who’s available to help – and remember that available may not mean willing. When faced with helping Mom evacuate from floodwaters or helping secure your data, what will your staff do?
2. Prioritize the recovery plan with business and clinical leadership, based on disaster type. A fire or flood at your facility will impact operations differently than regional flooding that affects the entire area. Consider the most common disasters you can expect, as well as the potential impact on operations. With these considerations in mind, what systems are the most important to restore? What secondary systems? Tertiary?
Remember, too, the ability to recover IT systems may depend on proper staffing. Also consider how staffing will affect your priorities. Your recovery priorities likely will change if you have 75 percent of your IT staff available, compared to 50 percent or just 25 percent.
3. Determine the systems required to restore any function. Hospitals have hundreds of technology systems, and many require upstream or downstream inputs to work properly. Bringing up one system may not sufficiently recover a mission-critical application for a department. Your IT staff must understand the relationship among systems in a particular department or for a particular function in order to understand your priorities for recovery.
4. Involve your third-party IT vendors. In many cases, your internal staff may not deal with certain mission-critical software, so vendor involvement in developing your DR plan is important. What disaster plans do vendors have to support your hospital’s operations in a crisis? Your IT staff must understand what systems they are responsible for in the event of a disaster and which will alternatively be handled by vendors. Your staff also must understand the relationship among IT systems – internally maintained and third party – as they pertain to a particular function or department.
5. Agree on a location for each recovery scenario. A localized disaster may not impact your data center, but a regional event such as a hurricane or flooding could. Evacuation of the hospital may require multiple back-up sites due to cost and space considerations.
6. Determine what medical records need to go with the patient. Your EMR is likely among those systems in the top recovery tier. But what constitutes a patient record, and how many systems contribute to it? A partial evacuation of elective surgery patients requires different thinking than a hospital-wide evacuation where systems already are compromised.
7. Avoid overreliance on the cloud. More hospitals are adopting cloud and software-as-a-service offerings that feature robust uptimes and HIPAA-compliant security. And cloud adoption is widely recognized as a best practice. But how will you access that information in a disaster? We worked with a client that had three outbound communications trunks and three different switches to ensure access. But a sinkhole in the hospital parking lot swallowed the entire communications infrastructure. In this case, the cloud was well beyond reach when the hospital needed it the most.
8. Consider cloud/local hybrid software installations. If you use Microsoft Office 365, you’re probably familiar with the offline feature where you can view and work with previously sent and received messages even if you can’t send/receive new messages. For data continuity, consider doing something similar for cloud-based applications by backing up a predetermined amount of data locally that would remain available if your hospital’s internet connections were severed.
This hybrid model would include a local appliance on premise that would facilitate the movement of data to the cloud while also facilitating backup to another medium, such as tape. In the event of a disaster, you would have a number of days/weeks of data available and the possibility of using the backup. Admittedly, the transfer of information to and from the cloud will be slightly delayed by the local back-up device. But that slight delay will get you a local copy of critical information should disaster strike, while helping you keep your operating license.
9. Recognize that plans aren’t concrete. Regardless of the type of disaster, the basic planning remains the same. But C-level execs and technologists must realize that not even the most comprehensive DR plan can cover every possible scenario. While that can be a scary prospect, having a plan that covers the majority of scenarios and has been vetted by the appropriate hospital stakeholders will keep you safer than having no plan at all.
