When James Wellman, CIO at Comanche County Memorial Hospital (CCMH) in Lawton, Okla., arrived at the 283-bed facility three-and-a-half years ago, its data center was a mess.
“When I showed up, we had a central data storage component with EMC but it was out of expansion capabilities. It did not have the growth to put everything on it the way we wanted to. Our digital backup before I got here had been pulled offsite to back onsite, so we had no offsite backup. Everything from the infrastructure from our network, the physical layout in the data center, right down to the floor, needed to be repaired and replaced,” Wellman says.
For a hospital that takes on a significant amount of additional data (approximately 10 terabytes) every year, having a dilapidated data center was a problem. Wellman knew he had a long way to go if he was going to fix the many issues he found in the data center. The first thing he did was a comprehensive assessment. Using a third-party consulting firm, he was able to figure out the investments needed for a redone data center. Once that was figured out, it was time to tell the CCMH Board of Directors.
James Wellman
“With any type of assessment, you have to prepare the administration and the board and talk to them. You have to let them know this is going to cost in the millions. For an organization our size, that’s not something to be taken lightly. I can’t just walk in and tell them I need $6-7 million dollars to rebuild the infrastructure,” Wellman says. With the organization investing in a new electronic medical record (EMR) system for meaningful use and undergoing data security maintenance, a significant speculation of any kind was not going to be easy to swallow.
It was going to be especially tough to swallow considering the circumstances of CCMH and the data center itself. As Wellman notes, every square foot that is spent on infrastructure in a hospital is lost revenue opportunity. Further, because CCMH is in tornado alley and subject to possible devastation, the true ideal circumstance would be a completely, remote data center with multiple feeds in a hardened shelter. Due to cost though, that wasn’t going to be the case.
Indeed, CCMH had to work in a finite area and because of the cost and enormity of the project, they needed flexibility. The different components needed to be added in stages. The hospital also needed to ensure the hospital stayed operational during the long transition.
All of these factors eventually led CCMH to Brocade, a San Jose, Calif.-based infrastructure vendor. According to Wellman, Brocade offered a valuable resource, their human capital. After hearing about a similar data center project at Children’s Hospital Los Angeles, a much larger facility, CCMH inquired about the company. Impressively, Wellman says Brocade sent the same data center design experts from that project out to remote Oklahoma. Those experts began to suggest ways they could redo the infrastructure in a costly, time-efficient manner.
“They had the best price but it wasn’t so great that we simply went with the cheapest option. Their understanding of our situation was much more comprehensive than any other contender,” Wellman says. “The design, the offer, and the thoroughness of that offer from Brocade far exceeded everyone else. They made us feel we were Children’s Hospital Los Angeles.”
Despite the progress CCMH has made in implementing a new data center and consolidating infrastructure, the project continues and will continue for the foreseeable future, Wellman says. It’s a five-year project, he says. The hospital continues to find and replace redundancies and find areas for improvement.
In a recent incident, the data center was down for 19 hours due to a downed primary condenser motor. Luckily, the IT department’s business continuity plans and backup systems worked, and nothing was lost. The incident showed Wellman that taking away the data center for an even a short amount of time would be catastrophic to the functions of a hospital. Everything relies on IT to operate smoothly within the hospital, from the obvious (electronic systems) to the understated (getting food to the patient’s bed).
“We came through it well, but it was eye-opening in showing us just how much the rest of the hospital relies on IT,” Wellman says.
One area of improvement will be around imaging in data centers. Wellman notes that images often cause the most problems for storage because of their size. CCMH is investing with the Chicago-based Merge Healthcare for an offsite vendor-neutral archive (VNA), which will allow the hospital to exchange images and improve disaster recovery efforts in case of one of those tornados.
It doesn’t end with images though. CCMH is also looking for an emergency co-location that backs up all of its data. This will be another significant project within this larger initiative taking as long as two years, Wellman says. One possibility, he mentions, is a mobile data center.
“It never ends,” Wellman says. “The biggest difference now is that we’re doing it in a planned manner. Before, it was showing up [at the data center] and saying, ‘Oh my goodness, what’s happened. How did it get to this?’ We’ve overcome that.”
