New Orleans’ East Jefferson General Hospital and its outsourced IT department utilize a comprehensive disaster plan, along with their courage and wits, to emerge victorious from Hurricane Katrina.
When a citizen of New Orleans, or any American, needs emergency medical care, his first call usually is to a nearby community hospital. Meeting the emergency healthcare needs of New Orleans’ Jefferson Parish and surrounding communities is a core mission of East Jefferson General Hospital (EJGH) of Metairie, La.
New Orleans’ East Jefferson General Hospital and its outsourced IT department utilize a comprehensive disaster plan, along with their courage and wits, to emerge victorious from Hurricane Katrina.
But last fall, when Hurricane Katrina’s category 4 force struck Louisiana, EJGH’s ability to respond was tested to the extreme. There were 15 hospitals in the New Orleans area before the storm; following the Katrina disaster, only three remained open to provide essential care to residents and emergency workers. One of the three was East Jefferson General Hospital.
Implementing the Disaster Plan
Tropical storms, even low-category hurricanes, are not strangers to New Orleans. But, unlike past storms when the hospital has benefited from three or four days warning, Katrina’s unpredictable route provided little more than a day’s notice and, unfortunately, reset its course for New Orleans on the weekend, when contacting designated emergency staff was most difficult. From the beginning of our Katrina experience, nothing came easy.
We made the first step quickly, gathering together a 61-person “essential staff,” including 17 on-site staff members of IT outsourcing partner Phoenix Health Systems at our called disaster time of 7 a.m. Sunday, August 28. Expecting a typical two- or three-day emergency, staffers arrived with inflatable mattresses, a change of clothes and very few necessities. They joined an estimated 3,000 people, including 950 hospital employees and volunteer physicians, 380 inpatients and more than 1,600 community members who needed refuge, including children and several pets.
As the hurricane’s 140 mph winds approached New Orleans that Sunday, the onsite IT staff prepared as best as it could within its tiny window of opportunity. Highest priorities were moving the hospital’s new Cerner equipment and other hardware to upper floors, ensuring that all the facility’s PCs were moved away from windows, and printing hard-copy materials such as vendor lists, census reports and inpatient clinical summaries. EJGH and Phoenix staff worked together to set up the hospital’s 24/7 command center, equipped with PCs, printers, telephones and fax.
When the storm roared through New Orleans and its surrounding parishes that afternoon, EJGH was, on one hand, fortunate. It did not flood, and the building and its contents remained intact. On the other hand, the area’s external infrastructure was essentially destroyed: High winds and widespread flooding from breached levees soon cut off all electric power, telephones, city water and major roads. Food supplies and essential services, including police security, were severely compromised. The hospital was left an island, stranded by water on all sides (which literally lapped at its rear entrance), alone and effectively cut off from the rest of the world for more than a week.
As might be expected, EJGH maintained an IT disaster recovery plan and, in fact, Phoenix’ IT staff had updated it substantially in late 2003. However, the extraordinary breadth and duration of the Katrina disaster rendered much of the plan irrelevant. As an example, the plan detailed a hot site arrangement with SunGard, whereby weekly systems backup tapes are held in a local vault to be available at any time for transport to a SunGard facility in New Jersey. Twenty-four hours before Katrina hit, mandated evacuations forced the local vault’s staff to leave the city, eliminating access to the backup tapes. Even if IT staff could have accessed the backups, flooded highways removed any possibility of travel.
During or shortly after the storm, the following pivotal events occurred:
- EJGH lost commercial power;
- EJGH lost two of 15 backup generators. The resulting loss of air conditioning caused wiring closets to heat up to 150 degrees, and IT had to shut down the hospital’s network;
- Bell South’s Primary CO went down;
- Bell South’s Jefferson Parish CO generator ran out of diesel fuel;
- Cox Cable, EJGH’s high-speed Internet supplier, went down.
New Critical Functions Emerge
Under these extreme conditions, it quickly became clear that basics were now EJGH’s first priority. The facility had become more of a shelter than a hospital, requiring special security measures (eventually provided by the National Guard and other military personnel), food rationing, provision and tracking of needed prescription medicines to employees stranded without them, and stringent sign-in/sign-out procedures. Community members and hospital staff who had taken refuge there had to live in the hospital for a week without knowing how long the siege would last; many remained even longer, camped out in corridors and empty patient rooms.
Using these minimal communications resources, the hospital was able to coordinate with area-wide disaster efforts (federal, state and local agencies); order supplies including food, water, and diesel fuel; and eventually arrange for needed generator and mainframe repairs. Maintaining these simple communications capabilities became of paramount importance as days passed; the list of hospital needs grew immensely, partially because of the duration of the disaster and also because of the sheer number of people on site.
Internal communications became nearly as important as outside contact. The hospital’s 3,000-plus residents were unnerved by stories of street violence, and anxious about their homes, families and friends. In response, members of the IT staff focused on maintaining continuous contact through the internal telephone system, and the command center produced twice-daily reports which were run to every floor.
Locating Employees and Ensuring Safety
Secondary to ensuring the flow of information, processing payroll and posting charges soon became critical. The IT staff focused on payroll as soon as it became likely that the crisis could be lengthy. Evacuation of most EJGH employees was generating high living expenses, and many people anticipated heavy home repair costs upon their return to the area. IT, the payroll department and the hospital’s bank devised a way to move money into each employee’s account using the Internet. They developed a “Net check” approach which did not include payroll deductions but represented the amount paid to each employee during the last pay period.
In the aftermath of the storm, Phoenix’s IT management recognized that quickly locating the 43 IT staff members who had evacuated and understanding their status was necessary, to develop a work schedule going forward and maintain staffing continuity. The Phoenix corporate office set up an emergency task force which worked with a senior New Orleans-based IT manager, who had remained off-site to serve as an external liaison. This team devised a systematic process for finding employees through e-mail, Web site notices, and calls to friends and families.
The team then established a temporary seven day on/seven day off work schedule that would provide staff needed time off to address home and family consequences of Katrina, and arranged a needed shift change for the essential disaster staff, who had worked steadily for 10 days, 12 to 18 hours per day. Phoenix was able to locate all 61 of its EJGH staff within three days, which greatly facilitated a return to normal scheduling within 30 days, and kept resignations to a minimum of only three individuals. In comparison, by November 1, many New Orleans firms still had not been able to locate all of their employees, which greatly hampered these businesses’ ability to return to full operation.
Power-up for a New Day
During the week following Katrina’s landfall, while many IT staff members concentrated on maintaining communications, coordinating aid and locating employees, others focused on bringing the hospital’s systems back up. Because EJGH did not flood, hardware and software systems were generally undamaged. Once the staff was able to locate a generator mechanic mid-week after the storm, work began to return servers and systems to the first floor IT Department. Power was rationed and air conditioning turned on only for short periods, to save fuel. The IT staff began systems power-up by Friday, after commercial power and battery backups were restored, but soon discovered that nine mainframe disks needed repair. Fortunately, an IBM service person was able to access the hospital and make the repairs by Saturday.
Through a combination of intense effort and a certain amount of good fortune, EJGH was able to get its systems back up and running on Sunday, September 4th, only one week after the storm began. Despite the massive destruction that still encircled EJGH and the continued lack of many public utilities, East Jefferson General Hospital was again in a position to provide care to injured Katrina survivors and emergency workers.
In October 2005, EJGH and its Phoenix IT staff began a process of re-evaluating its disaster recovery plan and developing recommendations for improvement, drawing upon its experiences during the Katrina disaster. Given the possibility, if not the likelihood, that the New Orleans area could experience similar future hurricanes or other disasters, the disaster recovery team is focusing, in particular, on achieving a greater balance between systems/hardware recovery and communications.
For example, in the fourth quarter of 2005, the team explored alternative cell phone approaches including satellite phones, a mobile satellite truck and wide area network satellite connections for the hospital’s PCs. These solutions may be arranged on a stand-by basis, for use in a declared disaster. The team is also planning to implement automatic switching of current backup electrical feeds between generators in support of the telephone system, should one generator go down, as occurred during Katrina.
Finding an effective approach to maintaining air conditioning in the hospital’s wiring closets (to keep the network operating) is a key issue under consideration, but its solution is likely to be very costly, perhaps prohibitively so. Finally, retaining Internet access and the ability to transmit e-mail during a power outage has become a critical concern; the team is exploring options for maintaining alternate approaches to EJGH’s established high-speed service.
For more information about outsourcing services from Phoenix Health Systems,
www.rsleads.com/602ht-222
Photos of the damage from Hurricane Katrina are courtesy of East Jefferson General Hospital and Phoenix Health Systems.
Donald Chenoweth is the chief information officer (outsourced) of East Jefferson General Hospital and is a Phoenix Health Systems employee. Contact him at [email protected]. Mark Peters, M.D., is the chief executive officer and Bruce Naremore is the chief financial officer of East Jefferson General Hospital, Metairie, La.
