The Rx for healthy networks and applications is a lifecycle management strategy that precedes rollout, and lasts through implementation and beyond.
Healthcare technology is transforming at an unprecedented rate. Physicians, nurses, clinicians, pharmacists, radiologists, emergency departments, local doctor’s offices, operating rooms, intensive care units, and insurance offices all must have instantaneous access to information from MRIs, X-rays, prescriptions and patient records to treat their patients. Considering that these individuals could be on different floors of a hospital, across a campus or scattered over several states, connecting them in real-time and in a cost-effective manner to the information they need is a monumental IT challenge.
The Rx for healthy networks and applications is a lifecycle management strategy that precedes rollout, and lasts through implementation and beyond.
Healthcare technology is transforming at an unprecedented rate. Physicians, nurses, clinicians, pharmacists, radiologists, emergency departments, local doctor’s offices, operating rooms, intensive care units, and insurance offices all must have instantaneous access to information from MRIs, X-rays, prescriptions and patient records to treat their patients. Considering that these individuals could be on different floors of a hospital, across a campus or scattered over several states, connecting them in real-time and in a cost-effective manner to the information they need is a monumental IT challenge.
When the network that serves a hospital’s medical personnel and patients, as well as insurance companies, also supports the storage and transfer of images used for diagnoses and treatment, network degradations or outages are simply not an option.
In response to federal regulations and guidelines, healthcare organizations have entered a new era in how medical information is collected, shared and accessed. Picture archiving and communication systems (PACS) and electronic medical records (EMR) applications are two of the most common and beneficial services now being deployed in healthcare networks. However, they come with some significant network challenges.
PACS, as imaging software, is very bandwidth intensive and can rapidly consume available capacity, while access to patient records in EMR applications needs to offer minimal delay and optimum availability. By developing an evidence-based data management approach for collecting and analyzing network data, the full value and benefits of the PACS and EMR applications can be realized, while simultaneously avoiding long implementation cycles, cost overruns or capacity issues.
The complexity of today’s distributed healthcare IT network environments combined with these new applications can introduce network problems that are difficult to pinpoint and resolve, and that negatively impact patient care. To mitigate these risks, the recommendation is to design a strategic lifecycle management process for PACS and/or EMR application deployment that includes: 1) A comprehensive assessment and analysis of the existing network; 2) Development of a thorough PACS or EMR rollout plan; 3) Appraisal of the impact of the implementation; and, 4) Establishment of processes for ongoing evidence-based data management.
Predeployment Audit Stage
During the predeployment phase of a PACS or EMR project, essential information needs to be gathered via an audit of the existing network, including bandwidth, applications, response time analysis and establishment of a network baseline prior to the introduction of new services. The following outlines the key information and statistics to gather, in order to baseline current network behavior and use it as evidence when formulating decisions and actions in the planning stage.
Create an inventory of the applications running over the network. This includes key details for optimizing the use of network resources, such as distinguishing business versus recreational use of the network, identifying applications that have been or will be retired, and pinpointing processes being performed at less than optimal times of the day. Value: This data will reveal bandwidth-consuming recreational use of the network, such as online gaming and streaming radio or video, as well as business activities such as downloads of security patches to desktops or server updates during peak times of the day that may be performed at a different time.
Evaluate bandwidth to ensure capacity availability for PACS or EMR services. Rank most and least utilized network segments, both in the campus LAN as well as over remote office WAN connections; trend activity and look for patterns in traffic behavior. Value: This information will be invaluable in “right-sizing” network segments to comfortably support the new services. (Note: If your hospital is charged by your service providers for WAN change orders, this takes on an even more important role in ordering the right bandwidth the first time.)
Create response time baselines of the hospital’s essential applications. Measure typical application response times for key applications. For instance, baseline the application nurses use to track schedules and hours worked, which may measure overall response time at 300 milliseconds, 220 milliseconds for network flight time, and 80 milliseconds for server think time. Value: This will help you understand your users’ perception of these applications’ performance prior to the introduction of PACS or EMR. If the reality is different post-implementation, you will know precisely by how much and where it is occurring—in the network or the application server.
Identify ancillary performance issues. No network is perfect—use this opportunity to do a little house cleaning. Look for packet loss, high application retransmits, previously undetected worms or viruses, or router misconfigurations. Value: Network anomalies may negatively impact service delivery of existing or new applications. Identifying them in the audit phase of the project gives you time to remediate them and avoid losing confidence in the PACS or EMR project during introduction and rollout. As an example, a medical center based in the northeast United States performed a network audit in advance of a PACS deployment and found eight workstations infected with a virus that were thought to have been removed from the network.
Decision and Planning Stage
The next lifecycle stage follows the carpenter’s maxim “measure twice, cut once,” by gauging the evidence collected from the predeployment audit to make some of the most important decisions during pilot testing and the eventual rollout of the PACS or EMR. Consider the following questions:
Should any current network traffic be moved or removed? Consider taking actions on items such as retiring applications that are no longer used but still in the network, business traffic that needs to be moved or scheduled to nonpeak times of the day, or configuration errors or misrouted applications that need to be corrected. One delicate task will be to remediate recreational use of the network by employees or contractors. For instance, eliminating streaming Internet radio to a computer at the 3rd floor pediatric nurses’ station.
Should bandwidth capacity be adjusted to accommodate PACS and/or EMR? This is critical because it affects the cost of WAN links. Consider what the network’s limits are for bandwidth upgrades: Does the segment need to be at 50 percent, 70 percent, 80 percent utilization and what parts of the network meet that parameter? Also, what assumptions are being made for added utilization from the PACS or EMR? Use the baselines established in the audit phase, plus projections for the PACS or EMR, to determine expected volume for network segments looking specifically at peak hours of traffic.
Were performance issues uncovered during the audit? If such issues as packet loss or high retransmits were uncovered in the audit phase, investigate and troubleshoot these problems now to avoid compounded problems later.
Will alarm thresholds be established? If yes, will they be by network segment, and if so, at what level, such as 80 percent overall utilization of gigabit Ethernet segments in the campus core, or 70 percent on T3 Internet connections. Also, what alarms are needed by application, based on the assumption for additional load, such as 25 percent utilization for PACS. If, in fact, the load ends up being higher, there may be congestion issues. Finally, will alarm thresholds be set against application response time? For example, if a nurse scheduling application’s response time expands beyond the average 300 milliseconds, an alarm should be triggered, because the PACS or EMR rollout may impact user experience of other business services.
Will a QoS policy be implemented with the introduction of the PACS and/or EMR? As much as any question, “What is the best way to deliver new applications?” is probably the most important. Establishing new virtual local area networks (VLANs) is one approach, while instituting a QoS policy is another. If implementing the latter, the number of classes to create will need to be decided upon, as well as which networked applications should be in each class. Here, the predeployment audit becomes invaluable because you know all your applications and can make the best decisions with all your stakeholders involved.
Three or four classes of service are common, with the highest priority class being given to the application least tolerant of latency—it may be VoIP (Voice over IP), for example, if it exists in the network or is expected to in the next six to 18 months, with the secondary priority being given to the PACS. The next priority level may go to the EMR and other patient-affecting applications, with the best effort class going to e-mail and Web surfing.
Rollout and Ongoing Management Stages
With the audit and planning stages complete, and decisions made and implemented, the initial deployment of the PACS or EMR will proceed. Real time monitoring and analysis will help mitigate impact on existing networked business services and help troubleshoot issues as they arise. Key to this phase is verification that the conclusions, assumptions and decisions made earlier in the project are effective in maintaining the quality of experience for doctors and nurses who will be reviewing images from the PACS, or accessing patient information quickly in the EMR. Pay particular attention to three key metrics:
Generated alarms: Adjust bandwidth or routing to ensure capacity does not create a bottleneck for any of your business services.
Response time: Monitor any response time changes for existing applications to avoid frustration on the part of your hospital staff, and to head off a small delay before its impact becomes severe and/or widespread.
QoS configurations: Quickly identify any misconfigurations and adjust business applications between classes as necessary or appropriate.
The widespread launch of a PACS or EMR marks the start of the ongoing management phase, which includes post-deployment impact assessment and troubleshooting activities to maintain a positive performance and high quality user experience. Many hospital IT organizations will find themselves serving as a sort of “referee” between the new applications and the existing services, troubleshooting network or application degradations, trending activity to plan adjustments in bandwidth, or initiating traffic engineering changes to move it to another time of day.
This ongoing proactive management of the performance of your hospital network before, during and after the introduction of a PACS or EMR helps avoid very costly and time consuming issues that often occur when decisions are either forgotten or made with incomplete information. Evidence-based data management lets you analyze and baseline the network, remediate discovered problems, and then rollout your new applications and services based on statistics from your own network. Good network and application performance health depends on an evidence-based management process throughout the application’s lifecycle.
For more information on integrated network performance management solutions from NetScout,
www.rsleads.com/701ht-200
Leveraging Information from Evidence-Based Management: A Case Study
Eastern Maine Medical Center is headquartered on a large campus in Bangor, Maine, with a 400+ bed medical center and more than 25 affiliate sites serving central, eastern and northern Maine. Two data centers, one primary and the other backup, support the network and application services for more than 5,000 healthcare professionals. Some of these services include enterprise resource planning, billing and account management, EMR, patient medical records and PACS.Recently, Eastern Maine Medical Center (EMMC), part of Eastern Maine Healthcare Systems, put their evidence-based data and network performance management system to good use. EMMS was engaged in ongoing troubleshooting and capacity planning between the data centers and the affiliate sites when it was discovered that another EMHS member hospital, Sebasticook Valley Hospital, was preparing to use a software solution to retain information on patient history and treatments.
While attempting pull down the database from the data center to a local server, in order to create a backup of patient information in the event of an interruption in service after the cut over, the IT staff at EMHS received some troublesome reports of slow response times from users at the Sebasticook location. The staff used the NetScout nGenius solution for evidence-based performance management to discover that during the time of the trouble calls, the normally 40 percent utilized T1 was spiking to 80 percent and sometimes 100 percent utilization, because the backup software was adding so much traffic volume. The network managers realized this was a temporary situation that did not require additional bandwidth. Instead, they simply selected a different time of day for the backups to avoid interruption of services to users during peak periods.
