NYC Hospitals Consolidate Disparate Data Centers
The New York City Health and Hospitals Corporation (HHC) is a mammoth enterprise, with 11 acute care hospitals, four skilled nursing facilities, six large diagnostic and treatment centers and more than 80 community-based clinics. With this large health system comes a hefty amount of patient data flowing in and out of its electronic medical record and 300 other business and clinical IT applications used by 20,000 daily online users. Previously, HHC had its data centers dispersed among 11 hospitals throughout New York City five boroughs. Weighing the cost of refurbishing all the aging data centers against consolidation, the cost was deemed too exorbitant to maintain the disparate data systems. So last year HHC went about consolidating their 21,000-square-foot data footprint to a slim 8,600 square feet. HHC’s principal production data center of 6,600 square feet, which handles up to 210 network racks, is located at the Jacobi Medical Center campus in the Bronx, while the back-up 2,000 square foot data center is in New Jersey and houses 130 racks. HCI Associate Editor Jennifer Prestigiacomo spoke with Bert Robles, HHC’s senior vice president of information technology and corporate chief information officer, about the data center transformation and the resulting cost savings.
Healthcare Informatics: Can you tell me about the two current data centers? Were the Bronx and New Jersey chosen because of cost per square foot?
Bert Robles: It was certainly the cost, and the availability of the footprint that would accommodate us. We looked across various boroughs, from an accessibility standpoint for cost, staff, suppliers, and the square footage that was needed with the least amount of renovation or new construction. And we decided that [the Bronx] was the best location for the primary data center. For the secondary or back-up [data center] for disaster recovery and business continuity purposes, we looked at a variety of options of whether we’d locate it within one of our campuses or would we go with a supplier. Ultimately, the decision was to go with [the Wayne, Penn.-based] SunGard. We moved it outside of the metropolitan area to New Jersey for safety reasons and because the cost was appropriate. It’s a scalable arrangement we have with 2,600 square feet that we lease; and if we need more space, they have a capacity for growth.
HCI: What was the strategy behind this project, and did you look at other organizations or use their best practices for your strategy?
Robles: We looked to two consulting arms that we have a relationship with that have an expertise in this area. We looked at latest trends in design of data centers. We had legacy centers that were in need of renovation. The timing not only was right to look at those best practices, the literature we looked at and the consulting advice all were consistent with saying that we needed to reduce our physical footprint and condense 1,172 physical servers down to 370 servers, primarily by better utilization of the equipment and virtualization. You can take [the standard] 1:1 ratio, which is one application to one physical processor, and through virtualization we can now host anywhere from 25 to 40 applications on a bigger box, which makes it more cost efficient and secure.
HCI: What percentage of HHC’s data is virtualized?
Robles: We have roughly 60 percent of our applications and systems virtualized, which is approximately 890 virtualized environments right now running.
HCI: How do you do content tiering? Do you have separate space for static data and separate storage for dynamic data?
Robles: We have right now 1.5 petabytes of information, which is a sizable amount of storage. On the other hand, what we do is tier the storage based on the performance specifications that we need: the speed, accessibility, ect. We not only monitor the available storage, but the rate of consumption, so we can predict how much storage we’ll need in the future. We’ve completed Phase 1, and will self-fund with these savings some of these initiatives. We are looking at medical archiving for data and images. We’re researching some of those suppliers in that industry. We do know that there is information in data and imaging that is not frequently accessed and we’re looking at some newer technologies that would allow us to archive more efficiently, yet provide the performance necessary on demand to recall those archived images and data.
We do have a business intelligence strategy where we have several enterprise data warehouses for revenue, clinical, and other information. We’re in the process of initiating a program where we’re going to expand our business intelligence capabilities and consolidate those enterprise data warehouses into a single source.
HCI: How do you balance user preferences with the cost of storage?
Robles: It’s a huge challenge just talking about storage first. Storage is growing at an exponential rate. Plus some other events in the industry, e-Discovery and other legal and investigative reasons, there’s an enormous amount of data we have to retain for an indefinite period of time. The use of video is consuming a lot of storage.
One of the things we’re looking at is we have a record retention and disposition workgroup looking at all the legal, state, city, and storage requirements we have. They try to revise our retention policies because we do need to have a more efficient way of disposing of information to actually contain the rate of growth of information, and still meet all the operational and regulatory requirements we have. The workgroup consists of legal, compliance, medical records, health information management, and IT operational leadership, as well as engaging state representatives.
HCI: How did HHC fund this project?
Robles: Most of the funding, about 85 percent, was through capital funds, since there was construction component. And the remaining 15 percent was operating dollars. The total was $67 million. We had to build out 4,000 new square feet adjacent to the existing data center at the Bronx Jacobi Medical Center location. So what we did was build out our back-up data center first, and once we had it to relocate our equipment, we just ran production out of our back-up center. We shut down our primary equipment and moved it into the new space, while we renovated the remaining 2,600 square feet. We’ve completed all that.
HCI: What were the cost savings realized from the consolidation?
Robles: First of all we have picked up about 13,000 square feet we can reprogram. We had projected with our current inventory of servers hosting applications throughout the corporation to spend $116 million by 2015. We’ve reduced that projected spend down to $42 million just on the hardware. In the last 10 months, we’ve already realized $7.9 million in savings from equipment we didn’t have to buy.
We saw environmental savings in power and air conditioning. We’ve also been able to reassign staff, as we were spending a few million dollars a year on consulting services because we didn’t have enough staff.
We decided after analysis that it also made sense to consolidate our help desk operations, which were dispersed in more than 11 locations. We decided to create a Tier 1 enterprise service desk, which is a consolidation of the help desk functions for support into one location, and we expanded our services to a 24/7 single point of contact for IT services. We still have three more sites to complete the consolidation in Q1 or Q2 2011. Afterwards, we’re projecting 35,000 calls per month for service requests. We’re going to annually save about $15 to $20 million, compared to what we would have spent if we still had disparate data centers.
