Natural disasters have been on a lot of people’s minds lately, most recently, on the East Coast, anyway, as Hurricane hit land last week and caused devastation to cities and towns in its path. According to a report in the New York Times, total damage inflicted by the storm could reach $7 billion, based on an early estimate by Kinetic Analysis Corp., Silver Spring, Md.In researching an article on disaster recovery for the October issue of Healthcare Informatics, I found that many hospital CIOs have had their own stories to share. The disasters have many underlying causes and are usually unexpected.One of the most gripping accounts I heard is from Diana Boyer, vice president and CIO of Columbus Regional Hospital (CRH) in Columbus, Ind.In June of 2008, the Columbus area experienced 11 inches of rain in a very short period of time, causing a small nearby stream to swell so rapidly that it resulted in a flash flood. In just 45 minutes, the stream overflowed its banks and spilled into the hospital parking lot, making its way toward the hospital basement that housed the data center, electrical switches, laboratory, food services, pharmacy, and other core services.The water reached the below-grade loading dock, which acted as a funnel that flooded the 160,000-square-foot basement in less than an hour. The flood submerged the electric switches, causing the hospital to lose power, and then began to bubble up to the 140,000 square-foot main floor, putting the radiology department and ED out of commission. The hospital staff successfully evacuated the hospital’s 157 patients without injuries, Boyer says.Boyer was home and couldn’t get to the hospital. She called her staff at 4 p.m. and was told everything was fine; but by 5:30, the hospital had shut down. Estimated- damages were close to $200 million and the hospital closure was estimated at 12 to 18 months.In the January prior to the flood, Columbus Regional had acquired a data center three miles away, and was in the process of installing redundant systems off site, which until then existed in the original data center that was now under water. The original data center also housed two storage area networks, one of which was to be moved to the new data center; and plans were to install a second uninterruptable power supply and backup generator. None of the installations were completed before the flood, Boyer says.
First look at damage to the data center. Photo: Columbus Regional HospitalThe original data center was destroyed in short order, with its equipment immersed in creek water, mud, corn stalks from local fields, reagents from the submerged lab, and other contaminants that made the computer drives useless. Fortunately, the hospital kept tape backups of data at another location in Indianapolis, so there was minimal data loss, Boyer says. The flood destroyed the hospital’s phone system, forcing the staff to rely on cell phones.Remarkably, Columbus Regional was operational less than five months after the flood. The Carolinas MED-1 mobile emergency unit arrived June 23, 2008. MED-1 provided emergency care for the area until CRH opened the Emergency Department in the main building on August 1. On October 27, 2008 CRH reopened patient care services, including surgery. Actual damages totaled $171 million.Today the hospital’s primary data center is housed in the new facility three miles off site, and the secondary data center is located on the first floor of the main hospital. The electrical switches have been moved out of the basement, as have the pharmacy and laboratory, which are now on the hospital’s main floor.Boyer places a lot of value on the hospital’s expert IT staff, which was hired in 1997 to develop the IT department as part of the hospital’s strategic IS plan. “We did not contract out implementations or technical support. We kept the knowledge here at Columbus Regional,” she says. “We had a very structured methodology on how we implemented systems; designing, building, testing and training; and how we keep it. Good relationships with vendors also were important,” she says.Just prior to the flood, CRH participated in a multi-county disaster recovery drill based on a potential terrorism incident. The hospital is now expanding on the command center model that was developed then. One of the biggest lessons of the flood had to do with the role reversal: hospitals traditionally receive disaster victims, but are not usually victims themselves. “That was a huge turnaround for us,” she says.All of the CIOs I interviewed for the article recognize the huge challenge of preparing for catastrophe, an undertaking even more difficult when hospitals are faced with budgetary constraints. Ironically, the best one can hope for is that all of that preparation helps to make sure that nothing happens…or when it does, the hospital can recover as quickly as possible, and learn some valuable lessons for the next time disaster strikes.
