At forward-thinking healthcare organizations seek to support new technologies, applications and services, such as advanced analytics to support precision medicine or population health initiatives, many are finding that their existing, older IT infrastructure may be an impediment to driving innovation forward. The volume of data that hospitals create and manage is growing exponentially, many health IT experts say, driven by the use of electronic health records (EHRs), diagnostic imaging systems, and mobile applications as well as federal regulations mandating the ongoing digitization of healthcare.
This unprecedented data growth, and the need to effectively leverage data to its potential, is proving to be a significant challenge for healthcare IT leaders. Effective data management and strategic data governance can be critical in order to move forward with operational improvements and to provide a high level of service and performance to clinician end users. To this end, many healthcare provider organizations have undertaken modernizing their IT infrastructure to meet future technology needs.
When Chad Eckes, executive vice president and chief financial officer, joined the Winston-Salem, N.C.-based Wake Forest Baptist Medical Center in March 2014 as CIO, he encountered significant legacy IT infrastructure challenges, and these challenges were impacting the organization’s ability to focus on its strategic priority of enhancing patient care versus maintenance and support. The aging IT infrastructure was difficult to support, there were sprawling data centers running out of space and critical projects were being delayed due to infrastructure constraints, he says.
“The prior CIO had chosen to direct investments in the IT organization toward two very large-scale IT software projects, an Epic EHR implementation and an Oracle PeopleSoft implementation, and had stopped putting proper investment into the data center, the networks and the core infrastructure there. There was no master architectural blueprint for the infrastructure, as a result, it grew haphazardly, and it was an aging and failing infrastructure,” he says.
Wake Forest Baptist Medical Center is an academic medical center and health system comprised of Wake Forest School of Medicine and an integrated clinical system with three hospitals and nearly 300 clinics serving 2.5 million residents in a 24-county area in northwestern North Carolina. The health system conducts 1.3 million patient clinic visits, 44,000 inpatient stays, 50,000 surgeries and 162,000 emergency department visits per year. The system also includes Wake Forest Innovations, a commercialization business. All told, the organization has 14,000 employees comprised of 1,200 physician faculty, 1,000 affiliated physicians and 1,900 students. In addition, the health system’s network also includes over 18 affiliated community hospitals, more than 100 clinics, physician practices and outpatient services serving one million patients annually.
As then-CIO, Eckes’ first order of a business in 2014 was conducting an assessment of the IT environment. “There were many applications issues that had to be addressed. One of the things that we uncovered was a mounting infrastructure risk that was impacting the organization, and the second was a severe issue with analytics and reporting.”
Chad Eckes
The assessment demonstrated that 64 percent of the IT infrastructure was over seven years old, Eckes says, and 12 percent was more than 11 years old, and he adds, “How that was manifesting itself on the operations of the company was that there were significant downtimes on a regular basis. We were showing an uptime percentage of 84 percent out of that infrastructure.“
The IT infrastructure at the time required 1,500 servers and multiple storage area networks. “All the servers were your legacy racked and stacked,” Eckes says. Further, the organization had made limited progress on server virtualization. “We were about 22 percent virtualized,” he says. And, Eckes adds, “We did not have a business continuity plan and disaster recovery plan for the systems, so that was core to our issues—when we were down, we were hard down, and that led to a lot of manually processing when taking care of patients.”
On the network side, the organization had many end-of-life products in the medical centers. “As an example, on our main campus we have 798 different networks, and 131 networks that had not received patch updates since 2010; not because we didn’t desire to, but because the products were so far end-of-life that we couldn’t patch them anymore,” he says.
For Wake Forest Baptist Medical Center’s executive leaders, modernizing the IT infrastructure was an imperative, and Eckes says organizational leaders considered two options. The first option for modernizing the IT infrastructure was what Eckes refers to as “the old-school way,” or traditional technology, and the second option was to implement a converged infrastructure.
“From a cost, risk and a speed-to-value perspective, all signs pointed to doing this with converged infrastructure,” he says.
After reviewing technology solutions providers, executive leaders decided to adopt a converged infrastructure approach, which is a data center management approach to consolidate computer, networking and virtualization silos into a single solution to support all of the organization’s applications. The health system worked with Dell EMC to develop a strategic plan for a new infrastructure and application platform, which resulted in the deployment of Dell EMC’s Vblock Systems to provide the infrastructure for the organization’s new software-defined data center, Wake Cloud.
“One of the key top reasons we choose this route was speed. Once the planning was done, it was 45 days and the Vblock System was designed to our specifications. Once the Vblock System was delivered to our doorstop, we were able to plug them in, and within two days, we were up and running and able to start migrating over our applications onto them,” and he adds, “That same process would have taken us months doing it in a traditional way.”
Implementing a converged infrastructure also required less staffing. “We would have had to add nine people to our infrastructure team to execute the same projects, doing it the ‘old school’ way. So, this saved us on the staffing front, and we were also able to target our retraining of our current staff on to the converged platform,” Eckes says.
All told, the process took about 18 months, with the biggest task in the migration involving transitioning 1,131 applications onto the converged infrastructure environment from the legacy environment without disrupting patient care.
Improving Efficiencies and Supporting IT Innovation
As a result of the modernization project, Wake Forest Baptist Medical Center has enhanced the performance of its Epic EHR and achieved significant efficiencies by reducing maintenance cost and improving uptime. “We’ve gone from an 84 percent uptime to close to a 100 percent uptime. And, when you’re talking about lives being dependent on the systems that we’re running or the bedside monitors that are hooked into our environment, it starts becoming priceless because it literally can be life or death,” Eckes, who transitioned to the CFO role in April 2015, says.
As a result of the project, the organization will reach 96 percent virtualization, up from 22 percent, in just 18 months, with enables virtualized desktops. Other efficiencies that have resulted from the project include driving maintenance and support costs down from 44 percent of the IT budget to just 10 percent, better utilization of the data centers and reducing 70 percent of the IT team’s time devoted to routine support to allow more time for creating added value. Further, the disaster recovery plan has improved with automatic failovers.
Further, the organization rolled out an upgrade of its Epic platform on top of the converged infrastructure environment which has enhanced performance, such as improved latency speed for better application response times, and an overall 30 percent performance improvement for end users. Saving seconds on every click inside Epic can add up to a significant amount of time saved for clinicians and staff throughout the day, Eckes points out.
IT infrastructure is a core part of the business of healthcare, says Dee Emon, vice president and CIO, and formerly chief clinical information officer, at Wake Forest Baptist Medical Center, and these improvements have been highly beneficial to clinicians and physicians. “It’s very important that our physicians and other care providers have information at their fingertips at all points of care—whether that’s at the patient’s bedside or in the operating room,” Emon says, adding that clinicians and staff now have faster, more secure access to medical records, diagnostic information and medical imaging.
With a modernized network infrastructure, the health IT teams are able to create more innovative solutions on top of that, Eckes adds. “For example, we have next-generation RFID solutions, and we’re literally badging patients and providers and we’re able to badge equipment, and by doing all of that, you can create auto log-in scenarios and create more personalized scenarios with the patient using data where it resides,” he says.
As an academic medical center, Wake Forest Baptist Medical Center executive leaders also wanted to better utilize the large pools of data it captured to support data analytics work. To this end, the organization also worked with its vendor partners to launch Wake Lake, a data lake platform to integrate with Wake Cloud to enable the health system to begin aggregating both structured and unstructured data in a centralized repository. According to Eckes, this will allow users, from clinicians to researchers, to perform powerful analytics projects.
Martin Sizemore, Wake Forest Baptist Medical Center’s associate vice president and chief data officer, says the IT infrastructure improvements solve many of the most common data center challenges—flexibility and speed. “Our goal was to have the same experience as if you wanted to stand up hardware on Amazon; we wanted that flexibility, we wanted that nimbleness—to be able to say, I need four processors and I need as much storage, and be able to spin that out very quickly to get something configured and ready to go for a researcher in the medical school or a clinician doing informatics, or just a business user who is trying to analyze their budget,” he says, adding, “So what used to take days, now takes minutes. We can also recover those virtual machines and that storage and use it for someone else, so it’s much more economical because we can redeploy resources quickly for bigger projects, for temporary development or for testing.”
One of the biggest lessons learned from this initiative, Eckes says, was devoting a significant amount of time on organizational change management with the infrastructure team. “This is a radically new way of thinking and there’s folks that have been making a career out of racking and stacking and implementing operating systems, and doing the normal care and feeding of servers. The new jobs that you have after going to converged infrastructure are much more interesting. The need for people doesn’t go away, but the roles change,” he says.
Eckes, who has worn both CIO and CFO hats, asserts as healthcare executive leaders move to cloud and converged infrastructure technologies, they should be prepared that these technologies essentially change the financials of IT.
“The way we used to do IT, especially in healthcare, is more project-based. You would decide to be put in a new system and that piece of software required X number of servers and X number of storage, and you would fund it off a CapEx (capital expenditure) project, and then you would worry about replacing the hardware down the road on another CapEx project. And you were always chasing the tail of capital dollars being available, and then you’d find that the replacements would be pushed out, because CapEx was squeezed,” he says. Deploying a converged infrastructure environment changes the discussion, he asserts, as “now it’s all about actually putting in capacity and planning for that capacity upfront so that it’s ready and waiting and you would call it more of a service-based approach to IT versus a project-based approach.”
Further he adds, “Our vision of the future is that software stays the same for our infrastructure and our internal consumers might say, ‘I want to spin up a new environment for this research study,’ and instead of using internal cloud computed storage, we can use a less expensive Amazon Web Services-based infrastructure. They still go through and utilize the same procurement process internally that manages that spin-off. Now all the sudden, from a finance perspective, you’re paying for the compute and storage, versus physical equipment.”
The Critical Role of Data Governance, as Healthcare Complexity Grows
Beyond issues related to modernizing aging infrastructure, healthcare provider organizations are grappling with changing market dynamics and trends, such as increasing digitalization, consumerism, regulatory impacts and healthcare delivery transformation. These market dynamics are creating demand for health IT and for more effective management of all the healthcare data.
Daniel Herman, informatics and technology practice area senior advisor at the Chicago-based The Chartis Group, a healthcare consulting firm, says as healthcare organizations continue to form different kinds of affiliations, whether through mergers and acquisitions or forming clinically integrated networks (CIN), this increases the need to focus on moving information from one place to another to access value and cost, and also provide patients access to their data as they go across these virtual networks. “We are seeing consistently this issue of governance and decision-making on how to balance supply and demand related to technology requests and capital investments. It’s an insatiable demand going on right now and there’s just limited amount of funds to go around, so looking at how we work with operational leaders to own those decisions just like they would own decisions related to investments, expansion or building programs or building new facilities and things like that,” Herman says.
Harold (Hal) Wolf, director with The Chartis Group and national leader for its information and digital health strategy practice, agrees, adding that this demand for technology “is such a critical and fundamental part of the building blocks that are happening inside the industry. There is this other significant demand going on in conjunction with personalized care in the digital space, so what’s beginning to happen, you’re seeing a tremendous amount of attention to governance inside organizations that are more complex that they even were just two years ago.”
Wolf continues, “The necessity to match IT investment and integrate it into the enterprise strategy is at an all-time high, because to gain the capabilities that are necessary for mobility, for population care management, for investment and analytics, you really have critical IT dependencies that have to be funded and they have to support the enterprise strategy, so the integration between the IT strategy and the enterprise strategy never before has been greater and it’s sitting inside that governance sphere.”
He also notes that healthcare leaders should approach technology projects as not strictly IT projects, but business projects supported by IT and critical to the success of the enterprise.
As an example, Herman points out that the consulting firm works with multi-entity, multi-region healthcare provider organizations. “With one organization, the first project was, ‘why are we spending so much on IT?’ And this is from the CFO and management team. After education was provided, the organization saw they were getting a lot of value from their spending.”
He continues, “And the next question was really the IT governance piece. The CEO of that health system really took ownership of that because he saw that the decisions related to it were very similar to the other decisions related to the prioritization of capital operating across the health system. As an example, for building programs that were already approved, the technology capital associated with that needed to be incorporated into the overall budget, and that wasn’t up for question, that was already approved.”
Ultimately, developing strategies for managing data, and addressing the technical as well as the cultural challenges, will be imperative as the transformation of healthcare continues and complexity only grows.
