When Hurricane Sandy struck the Northeast in late October 2012, it left a path of destruction. Homes and businesses along the coast fell prey to winds and flood waters and hundreds of thousands were without power. Even cell phone service in many areas was knocked out.
In New York City, subways and tunnels were flooded and water from the East River rushed into the streets of Manhattan, threatening some of the city’s largest and most prestigious hospitals.
Nader Mherabi, senior vice president, vice dean and CIO of New York University Langone Medical Center was getting reports about the approaching storm and put his IT team on high-alert. And while he acknowledges that the hospital was forced to shut down and had to evacuate over 300 patients due to flooding and a power outage that knocked out even the emergency generators, he says absolutely no data was lost.
Mherabi credits a disaster recovery plan put into place in 2009 for keeping all hospital data safe. “All data for us is considered sacred,” he says. “We’re fortunate that we had that plan.”
Working with Wayne, Penn.-based SunGard Availability Services, the hospital evaluated its IT infrastructure and began a two-year implementation. As part of this strategy, the hospital rolled out an electronic medical record (EMR), which came in handy during the hurricane. Because the EMR and scheduling systems are linked, patients who had appointments that were cancelled were notified online.
The initial strategy also set priorities, Mherabi says. “We had to decide what data should be recovered first and what were the most critical systems.”
While the hospital has its own on-site datacenter, it also rents space in Sunguard’s New Jersey facility, where it set up a command center to coordinate recovery efforts, Mherabi notes.
Similar to other academic medical centers, NYU Langone relies on high-speed broadband connectivity. But that was lost during Hurricane Sandy. However, because the hospital has redundant connectivity through its providers, service was restored in about 36 hours.
But James Song, vice president of infrastructure and engineering, says that while normal broadband connectivity was down, the connection between the hospital and the data center in New Jersey stayed up.
Ensuring the security of all patient, financial and research data is a high priority at NYU Langone. Mherabi says the hospital uses cloud technology internally, but decided not to go that route for backing up mission-critical data.
Song says that because the hospital still wanted to retain physical control of all backed-up data, it upgraded from a tape back-up to discs which are collected and off-sited each night.
Although the hospital sustained hundreds of millions of dollars in damage from the storm, Mherabi says none of the IT infrastructure was permanently damaged.
As for offering others advice on setting up a disaster recovery plan, he says, “Prioritize what systems need to be recovered first, second, third.”
Song adds: “The most important thing is to understand the business requirement—what are the critical systems. Then have a clear policy of data retention and make sure everyone understands what that policy is.”
In looking back over the recovery efforts, Mherabi also stresses the importance of team work. “In a crisis, people work together,” he says. “We learned we can do things faster than we thought we could.”
