Report: HC Plagued by Data Center Outages

Earlier this month, the Traverse City, Mich.-based Ponemon Institute released the first of a two-part national report on data center downtime . In that report, the healthcare sector scored the highest frequency of data center downtime, with an average of three outages over the past two years, as opposed to the financial industry, which reported the lowest frequency of downtime, with 1.8 outages in the same time period. Larry Ponemon, Ph.D, chairman and founder of the Ponemon Institute, in an e-mail interview, recently shared with HCI Associate Editor Jennifer Prestigiacomo his views on how healthcare technology growth is outpacing what legacy hospital data centers can deliver.

Healthcare Informatics: Did you find any of the report’s conclusions surprising?

Larry Ponemon, Ph.D.: I was most surprised by the lack of preparation to respond to data center outages, considering both the frequency of downtime and the realization that availability is absolutely vital to operations. Ninety-five percent of the organizations surveyed have experienced an unplanned data center outage in the past two years, yet just 37 percent feel they have the resources to bring their data center up and running in this situation. This is surprising, as more than half consider every application in their data center to be mission-critical.

HCI: What are some of the most common causes of data center downtime, and what are the implications of this?

Ponemon: Respondents to the survey cited UPS [uninterruptible power supply] battery failure most frequently regarding the cause of data center downtime, followed by exceeding UPS capacity, accidental emergency power off (EPO)/human error and UPS equipment failure. This means that vast majority of data center downtime was related to inadequate investments in a high-availability redundant power infrastructure.

HCI: The report noted that only 16 percent of respondents were certain about the cause of the outage. Why is there so much uncertainty about the root cause of data center outages?

Ponemon: There are a couple of explanations for this uncertainty. Because they’re not as involved at the data center operations level, it’s possible that senior management respondents don’t have the day-to-day knowledge of what’s happening inside the data center—in other words, they are aware when an outage happens, but they may not have all of the background. Aside from that disconnect between senior management and rank-and-file employees, the facility as a whole may not have the appropriate monitoring processes in place to determine the cause of an outage. They may be so focused on reacting to an outage that identifying the cause is an afterthought.

HCI: What are the most effective prevention tactics that IT managers can employ to prevent outages?

Ponemon: Ensuring that UPS batteries are properly maintained, serviced, and replaced is one of the most effective ways to prevent a breach in downtime. Using an integrated battery monitoring system and spare charged batteries onsite will also help to combat an outage before it happens. It’s also important for IT managers to understand their data center’s typical load so the appropriate UPS can be deployed. The survey showed that more than half of all downtime events were the result of exceeded UPS capacity, making this an important issue to address prior to an outage.

HCI: What are the main challenges for CIOs to combat data center outages?

Ponemon: As the survey shows, a lack of resources is a major issue. This could be attributed to a few things, including budgets that don’t align with the goal of providing a high-availability infrastructure. If CIOs and other senior management do not truly understand the cost of downtime in the data center, they likely haven’t allocated the appropriate budget for preventing and responding to outages.

HCI: What are the main implications for the healthcare sector in this report?

Ponemon: Despite their mission-critical focus, healthcare organizations noted the highest frequency of data center downtime with an average of three outages over the past two years. Comparatively, the financial industry reported the lowest frequency of downtime, with 1.8 outages in the same time period. This means that, although access to healthcare data like electronic medical records is just as important—if not more important— than access to our checkbooks, our healthcare data centers are years behind in terms of ensuring availability.

If a healthcare organization experiences a data center outage, it’s more than just an inconvenience. Medical personnel will be unable to access records—so, for instance, determining what each patient has been prescribed will be put on hold. There’s a potential for records like MRIs and doctors’ orders to be lost, as well as administrative problems like an inability to connect to reimbursement and/or billing. Downtime in the healthcare environment also means patients will be redirected elsewhere, resulting in lost revenue for the organization.

HCI: In comparison with the other industries, why do you think healthcare had the highest frequency and longest duration of outages?

Ponemon: Healthcare technology growth and is outpacing what legacy hospital data centers can deliver in terms of availability. Applications like electronic medical records and digital imaging have changed the face of the industry, but existing data centers have failed to adapt accordingly. The healthcare sector should model itself after another mission-critical industry, like the financial sector, which draws a clear line between access to data and profitability.

HCI: Your report noted a difference in perception between senior-level and rank-and-file respondents regarding data center outages, how can CIOs best persuade other senior management the need to implement data center systems and best practices that increase availability?

Ponemon: The key is to focus on the bottom line, as that’s what will resonate with senior management. Downtime, particularly in the healthcare sector, can add up to potentially thousands of dollars per minute. Think about the lost revenue from redirecting patients to other hospitals when access to data is lost or files are damaged due to power loss. There is also a liability concern if treatment is negatively impacted. Senior management understands the dependency on applications, but CIOs are best served to tie availability to a tangible, monetary figure their peers can comprehend. And if that doesn't work, the CIO can remind the hospital’s president that if the data center goes down, it will absolutely make the evening news, and that’s not a pleasant situation to be in.