The aftermath of an earthquake is something most businesses hope to never face, but to avoid getting caught off guard, or offline when disaster strikes, many businesses are realizing they need secure, reliable ways to store their sensitive data. For St. Charles Health System, based in Bend, Ore., an earthquake was only one of a myriad of issues, a tangled knot the information technology team wanted to smooth out. In the process of partnering with telecommunication company BendBroadband for its data storage solutions, St. Charles discovered that sometimes if you want it done right, you don’t always have to do it yourself.
Part of St. Charles’ problem was location: Nestled in the high desert of Central Oregon, bordered by mountains, Bend is somewhat cut off from Portland, where its data was located. One of the reasons it co-located its data in a different city was to prevent its compromise in the case of a regional disaster. Another reason was that, until recently, Bend was a relatively small town, and was not known for IT brainpower. But St. Charles discovered that co-locating its data in Portland put its computer assets and data out of easy reach. The Portland location also required redundant network circuit routing, and placed it in a facility next to an interstate freeway that posed the risk of hazardous material transports, fire and traffic accidents. Also, Portland is located in a zone that could be devastated by an earthquake.
Once considered a California problem by many Oregonians, the subduction zone just offshore from the west coast has become more of an Oregon concern. Whereas Portland is in a zone that could experience major damage in the event of an earthquake (and scientists from the Oregon Department of Geology and Mineral Industries say when, not if), Central Oregon is in a relatively safe part of the state. In fact, Apple is building a data center and Facebook is building its second data center in Central Oregon.
Given all of these concerns, and the advantages to actually siting its data close by, St. Charles decided it was time to bring its data back home. The company considered building its own data center right in Bend to cut out the middleman and do it itself. This idea was appealing in many ways, particularly the financial boost it would give the local economy.
Though the idea of building its own data center was appealing in some ways, it brought up another set of problems. St. Charles would be investing time, energy and resources into the ongoing management of a physical data center.
“Our business, if you will, is serving our patients, not investing and building out data center capability,” says Steve Janego, St. Charles’ chief technology officer.
So St. Charles sought an outside partner to host its data and discovered that its neighbor, BendBroadband, a locally owned telecommunications company, was in the process of planning a data center right in Central Oregon to be named the Vault, featuring cutting-edge security systems and green technology. BendBroadband, in turn, was searching for an anchor tenant.
Janego describes BendBroadband as “a local dog that has gained national attention as a technology innovator in a region that is not thought of like Silicon Valley.”
If St. Charles partnered with BendBroadband, it would have the unique opportunity to significantly influence security and design decisions, eventually resulting in a facility that was purpose-built for healthcare, from the ground up.
St. Charles decided to trust BendBroadband to carry through on its lofty goals: to build a data center facility with best-in-class security, redundancy and reliability that would be as sustainable and green as possible, while delivering a 30 percent reduction of St. Charles’ data storage energy costs. This would provide a return on investment over a five-year period of $7.1 million.
Multiple levels of security, redundant systems, scalability, sustainable design features (from the logical to the unusual) plus a savings of $7.1 million – could it be done?
It was paramount that St. Charles’ continued to comply with HIPAA, among the most stringent privacy and security regulations required by any industry. From the outset, BendBroadband committed to meeting or exceeding HIPAA’s requirements as well as those of the financial and other security-sensitive industries. That entailed building the Vault with a network operations center monitoring all aspects of system-wide performance with on-site staff 24/7, 52 HD video surveillance cameras and state-of-the-art security systems that allow flexible data logging and reporting.
Healthcare providers need granular control of who can access which cabinets for auditing compliances. Physical barriers to entry and biometric access controls allow only authorized personnel into selected areas of the building. According to Wade Holmes, BendBroadband VP of technology, the Vault is one of the only data centers in the nation that also has rack-level security, which means individuals must be scanned in to open a single rack. This allows BendBroadband to track an individual from the moment the person walks in the front door to the moment the rack door is opened.
Obviously, a healthcare system has no room for error when it comes to reliability. St. Charles conveyed this to HP, which BendBroadband tapped to design a concurrently maintainable MEP system – the best mechanical, engineering and power available. HP understood that a regional trauma center with people’s lives on the line – and the corresponding sensitive data – needed Tier III reliability.
Tier III certification requires the data center infrastructure to be concurrently maintainable, meaning that every component and element in the distribution paths for power, cooling and network can be removed from service on a planned basis without impacting any of the computer equipment. The benefit of Tier III certified facilities is the assurance that data will be kept safe and secure, with operations running without interruption. Tier III certification was a key objective for all parties.
Additionally, the Vault was built with scalability in mind. Usually data centers do not have the ability to operate 4 kW per cabinet, but the Vault’s cabinets are designed to handle up to 21 kW to accommodate four dense blade chassis in a single rack, or stack Cisco UCS systems floor to chimney. So rather than adding racks, St. Charles could scale within the rack. The cabinets were also designed to accompany a modular power distribution system capable of single- and three-phase power.
Green technology was not only important to St. Charles because it values sustainability, but was a key to cost reduction. Most data centers operate at a power usage effectiveness (PUE) ratio of 2.0 or higher. BendBroadband installed two 450-kW-capacity KyotoCooling systems, which utilize the chill night air of the high desert to cool the data center, and an uninterruptible power supply (UPS) flywheel system to help reduce the Vault’s PUE to less than 1.2.
In conventional data centers, cool air is pumped into the room from under a raised floor and drawn into the cabinets to cool off the equipment. In comparison, the Vault’s hot air containment design allows cooling only where necessary, saving energy and also supporting high-density virtualization – running several virtual servers in one physical server chassis, and several of these blade chassis in one custom-made Chatsworth cabinet. A chimney above each cabinet removes heat produced by the servers.
But when St. Charles’ new Nexus core switches from Cisco arrived at the Vault, they ran into a problem. The equipment didn’t fit into the cabinets.
The design team re-approached the floor layout: one way or another the equipment had to fit and still be cooled by drawing cold air in the front of the cabinets and exhausting the warmed air through chimneys. The team came up with the idea of arranging the equipment in pods that can handle the non-uniform equipment needs. Inside the pods, St. Charles can place a variety of non-uniform equipment, and each pod is still encapsulated by a chimney. The pods also allow for scalability, so they can continue to be a good long-term solution as St. Charles’ data continues to build-out.
Additionally, the sliding doors on the pods provide another layer of security, and the design team created an audit trail for each pod.
To Janego, this solution illustrates their partnership: “Allowing our team in on the ground floor of the design process was about as big a show of trust as you can dream of,” he says.
The design team also incorporated green technology into other aspects of the building design. From pervious pavement that allows water to seep naturally into the ground to wall insulation made from recycled denim jeans, it incorporated every sustainable building technique possible.
“Many of these investments make short-term business people scratch their heads,” Janego says. “But we want to invest in sustainable technology that would not only survive, but thrive in tough times, while making as small an impact to our environment as possible.”
It took St. Charles approximately nine months to move its data from the previous location to the Vault, with a huge amount of upfront planning. It completed the actual cutover in stages. St. Charles rehearsed and practiced the move with non-critical systems and moved its clinical systems last.
“Virtualization of our servers made the move possible,” Janego says. “With the limited downtime window we had, it was critical to use this technology to actually move our computing assets.”
At its former location, St. Charles had a primary fiber route and secondary route, and did fail-over testing periodically to ensure both 1-GB circuits worked as designed. St. Charles now has fully redundant 4-GB connections to the Vault at a much lower cost.
By April 2011, St. Charles’ data had been successfully shifted from its former location in Portland to the Vault, serving more than 3,000 caregivers. The health system also stores patient health records adding up to more than 1 million patient encounters since 2009.
Two months later in June, St. Charles was notified that the Vault received Tier III certification, including awards for both design and building construction from the Uptime Institute, a worldwide organization recognized for its credibility in validating enterprise and managed-service data centers.
More good news came to St. Charles in October 2011, when the Vault received LEED Gold, making it one of only five data centers worldwide to achieve this level of recognized sustainability.
The savings envisioned for this project have materialized for St. Charles. It reduced its co-location power costs by 30 percent. And now, just one year after moving into the Vault, it is still on target to receive a five-year ROI of $7.1 million, and is tracking to its planned capacity and build-out, as well.
Reflecting on the experience of transitioning from a data center in a different region to the partnership with BendBroadband, Janego says, “The Vault has provided St. Charles Health System the chance to focus on our core mission and attend to the needs of our health system. With the Vault in place, we are now focused on the most important things: the health of our patients and the community we serve.”
And when it comes to healthcare, that’s really what it’s all about.
About the author
Jenny Furniss is a freelance writer based in Portland, Ore. www.JennyFurniss.com For more on BendBroadband and the Vault, click here.
