Before you see Smoke

June 24, 2011
Every day that Marie Palladino works to prepare Deaconess Hospital for the year 2000, the problem appears to keep growing. One of her first steps

Every day that Marie Palladino works to prepare Deaconess Hospital for the year 2000, the problem appears to keep growing. One of her first steps after becoming Y2K coordinator last year was to conduct an equipment audit to identify all the electronic systems that potentially could greet the millennium with the equivalent of a cyber hangover. She passed out questionnaires to the Evansville, Ind., hospital staff but received only a headcount of PCs. "We made an assumption that end users knew more than they did," Palladino recalls. "They never looked at the biomed equipment or the network infrastructure, or anything else ’invisible.’" So Palladino walked through each department and noted each defibrillator, telephone, fax machine, network server, lighting system and security alarm. "The more I looked, the bigger the problem became," she says.

Then Palladino, an information services nurse analyst, tested an IV pump by setting its internal clock ahead to a date in 2000. Not only wasn’t it programmed to identify the millennium, the fact that 2000 is also a leap year totally jangled the pump’s silicon nerves. "It wouldn’t even let us go back to the present date, so we had to send it to the vendor to be fixed," Palladino sighs.

Deaconess is not alone in tripping over digital "gotchas." Y2K experts say pitfalls are the norm in any millennium preparedness effort. In fact, Deaconess is way ahead in the Y2K game compared with most healthcare organizations. Hospitals rank at the bottom of the list, along with cash-strapped educational institutions, as the least mobilized industry sector for addressing year 2000 problems, according to Lou Marcoccio, the Y2K research director for consultant GartnerGroup. He spends his days checking how well 15,000 companies in a variety of industries are faring with millennium projects. The prognosis for healthcare, he says, is grim. Few hospitals have created a detailed plan to attack the threat or actually fix code, he says. "Healthcare has been too slow, too late and too optimistic about Y2K," adds Jim Woodward, senior vice president of IS consultant CAP Gemini America, Iselin, N.J.

With slightly more than 15 months to go, that means ground zero for millennium crashes will likely be in healthcare’s biomedical equipment and computerized business systems. Experts agree that if you’re still working on millennium fixes, or worse, haven’t begun, IS triage may be mandatory: Focus on fixing critical systems because you may not have time to fix everything else in time.

Digital ’oops’
The millennium problem began, of course, decades ago when computer programmers started using two-digit shorthand to represent years. Thus, 1960 became "60" and 1998 became "98" so precious computer memory wasn’t wasted on extraneous information like the assumed "19."

Sometime around 1970, banks that sold 30-year mortgages became the first to say "Oops." Computers spit back mortgage applications that terminated in 2000 because the little cyber neurons assumed "00" was ancient history. From there, a succession of industries, ranging from insurers with multi-year policies, to brokerages selling long-term bonds, replaced or updated their computer systems to bring them into the 21st century.

Healthcare’s reach didn’t stretch so far into the future, so rewriting business and clinical applications wasn’t a priority until recently. That let the industry focus its IS efforts on modernization rather than reconfiguration. It also helped hospitals ignore the problem until the last minute.

Inaction may have even seemed prudent. After all, if other industries and hired-gun programmers learned time-saving tricks for fixing Y2K, the ramp-up time for latecomers like healthcare should be shorter. Unfortunately, hospital electronics are uniquely complex and the consequences of a missed med or a faulty pacemaker are far more dire than a checking account that doesn’t balance. Secondly, finding available Y2K experts to help your makeover efforts now may be like trying to find a competent accountant on April 14.

The result: Because IS departments face a "hard" deadline that can’t be rolled back, many healthcare companies are already close to being too late, says Woodward. Some hospitals have lulled themselves with rationalizations like "Technology is changing so rapidly, our stuff is probably Y2K compliant." Or, "Hardware and software vendors will help us fix the problem before it’s too late."

"These are blind views," Palladino asserts, after spending the last seven months uncovering plenty of systems that aren’t Y2K compliant and speaking with vendors who have yet to test their computer and software products--let alone figure out how to bring them up to date.

Other hospitals fully understand Y2K consequences, but find themselves strapped for resources. Most IS funds go to new systems that can deliver financial efficiencies necessary in a hotly competitive marketplace. Even with that, IS’s slice of the pie is tiny. The average IS expenditures across all industries is 6-8 percent of net operating revenue, according to Marcoccio. However, U.S. healthcare only spends about 2-3 percent for information technologies. So because millions of dollars and programming hours devoted to millennium problems won’t result in new system features or efficiencies, many IS departments face pressure from the top to devote only a minimal level of resources, Woodward observes.

The penny pinching that many hospital IS departments endure means IS staff receive as much as 20 percent less in salaries than their peers in other industries. Combine this with outdated computer infrastructures, and healthcare finds itself needing more time than other industries to modernize systems, anywhere from six to 36 months and $10 million to $80 million, depending on size and the age of the technology.

"Healthcare is in dire straits" regarding Y2K, according to Marcoccio. Gartner projects that companies only now starting to rewrite problem code are far enough behind that there’s a 50/50 chance they’ll experience a mission-critical system failure at the turn of the millennium. Companies just launching Y2K efforts face a 90 percent prospect of a critical failure.

Invisible threats
Some of the biggest fears hospitals face going into the new millennium are breakdowns in biomedical systems, including electrocardiograms, CT scanners, pacemakers, lung machines, IV pumps and other equipment with embedded microprocessors. According to research by GartnerGroup, about 2 percent of chips embedded in biomedical equipment fail Y2K tests.

Biomedical equipment vendors are setting up Web sites that list hardware that complies with Y2K. In addition, a number of public- and private-sector organizations, including the FDA (see "Resources," page 35) provide online Y2K-compliance databases. Yet some hospitals worry that even if one pacemaker, for example, passes a test and makes it on the list, there’s no guarantee that every model of that device will be compliant. Different components from different suppliers may go into a pacemaker product line throughout a given year, which in turn could alter Y2K compatibility.

So hospitals such as MacNeal Health Network, Berwyn, Ill., are creating testing labs, with servers, workstations and software that duplicates production systems, to identify noncomplying systems and certify that fixes don’t introduce new problems. The need to conduct their own tests rather than rely on compliance reports from hardware and software vendors exists because details are scant thus far, sometimes because vendors still are in the process of testing their own products for Y2K compliance. So even though they’re rushing to zero hour, many hospitals are taking a hard-nosed and time-consuming approach to biomedical-equipment testing. "We plan to test every piece of equipment," says David Printz, MacNeal’s CIO.

But unlike the ubiquitous PC, embedded processor machines don’t have an easy interface that lets users peer into the system clock of the chips and firmware, which adds an extra complication to testing Y2K worthiness. At MacNeal, the 100-member IS staff, plus biomedical support workers and department heads have become official biomedical equipment testers. Testing consists of resetting clocks to see if a pump or pacemaker starts to smoke when it sees zeros. Testers slap bright green stickers on machines that don’t flinch at the millennium. Test failures join a list of machines that need intervention, either replacement or--if the vendor offers it--a way to modernize the system clock.

Rather than devising its own test suite, Deaconess jump-started its certification efforts by following certification procedures used by General Motors, which the company makes available on its Web site. "They tell you step-by-step what to test. It’s the most detailed procedures I’ve seen," says Palladino.

This old IS department
If there’s any bright spot to the Y2K rush it’s that hospitals are cleaning their IS houses. Many hospitals use Big Iron computers and applications born in the 1960s, like the Shared Hospital Accounting System (SHAS) MacNeal ran until last year. "It should have died in the ’80s, but we kept it alive because we had other priorities for spending money," says Printz. "Year 2000 forced us to get rid of obsolete software. Replacing that product alone gave us dramatic performance improvements," he says.

The hospital also replaced a 20-year-old human resources/payroll application. Fortunately, hardware expenditures are relatively modest: A move to a client/server architecture in the early part of the decade provides the hardware foundation to run replacement apps.

Other hospitals are simplifying their purchasing procedures by paring the number of suppliers they use. For example, one organization stopped getting six different models of IV pumps, opting instead for two models, which will be easier to manage. Contracts for new equipment now regularly contain a clause requiring the vendor to insure Y2K compliance.

Nagging doubts
No matter how well prepared an IS department is, people like Printz and Palladino won’t rest easy until well into 2000. They lose sleep worrying about devices that slipped through the testing process or about suppliers who won’t be adequately prepared for Y2K. What happens if power doesn’t flow on January 3, the first workday of 2000? Or the software that routes calls from the local telephone switch stops working? Printz is heartened by the fact that the hospital ran efficiently for most of a day recently when a construction crew accidentally cut three underground power lines to the hospital. The hospital’s emergency generators kicked in without missing a heart beat. Nevertheless, emergency power won’t keep the hospital alive for an extended outage.

Palladino is philosophical about things beyond her control, but wrestles with concerns about human error. "What did we miss? What if some equipment we passed in our tests fails to work for some reason? Failures in the systems we feel comfortable will be the ones that get to me."

Online Resources

http://208.192.104.178/y2k/y2k_search.cfm: The U.S. government’s online database of biomedical-device compliance. The site consists of vendor responses to a Y2K survey sent out by the FDA.

http://www.mitre.org/research/cots/COMPLIANCE_CAT.html: A source list for software and hardware compliance.

http://pw2.netcom.com/~helliott/00_nfrms.htm: A master index for healthcare-oriented Y2K sites. Check here for names of consultants, "code factories," tool vendors and other aids.

http://www.rx2000.org:A comprehensive source for Y2K information for healthcare organizations.

http://www.uthscsa.edu/2000/test1.html:Maintained by the University of Texas Health Science Center at San Antonio, the site offers an excellent "toolkit" for testing procedures for embedded devices.

Lessons From The Front Lines

Y2K VETERANS AT HOSPITALS AND IS consulting firms warn that achieving year 2000 compliance is more complex, more time consuming and more expensive than most hospitals believe. If you’re still rushing to meet the millennium, here are some ways to turbocharge your project.

1. Get upper management’s commitment.In most cases, IS will lead the charge because it’s the first to comprehend the seriousness of year 2000 incompatibilities. Yet Y2K initiatives won’t survive unless chief executives and hospital boards also understand the patient care and legal risks inherent in incomplete modernization efforts. Sound easy? Don’t bet on it. GartnerGroup estimates that about 90 percent of hospital CIOs can’t get the Y2K funds they need because their CEOs don’t understand year 2000 risks.

What makes for successful Y2K projects? "The hospitals that are farthest along almost always have a strong person who really kicked ass early on," says Gartner’s Lou Marcoccio. The key: Present to board members clear explanations of the business and legal risks of Y2K to start. Later, delve into the technical details.

2. Get intradepartmental involvement.Establish a Y2K committee to create an overall project plan, set schedules and monitor progress. The eight-member committee at MacNeal provides Y2K presentations at managers’ meetings and publishes a Y2K update page on the organization’s intranet. Who to enlist? Systems managers, facilities staff, purchasing managers and operations managers, suggests Pat Riley, senior vice president for business development at Millennia III, a Westport, Conn., IS consultant.

3. Conduct triage.Hospitals that started on the Y2K problem a year or two ago had the luxury of doing complete equipment audits; hospitals behind in Y2K efforts may have to cut some corners. "Perform triage," says Woodward. "The first cut to make now is identifying mission critical systems." Once you decide which systems your organization can’t live without, arrange to have in-house or outside programmers write code patches to bring them up to par or contact the software vendor for Y2K-compliant versions of the application. Note that your task doesn’t end with merely reinstalling new software. You must convert legacy data entered under the plagued two-digit format into the four-digit format.

4. Consider outside help.Ask a Y2K consultant if a hospital can go it alone, and not surprisingly he or she will probably say no. "This is a specialized problem. The chance that you can do it at optimum efficiency is low," says Woodward.

If you decide to go out of house, your choices range from full-service consultants that take care of everything from the initial equipment audit to testing final fixes. Projects like these are typically priced on a per-project basis, ranging from $10 million for small organizations to $80 million or more for large hospitals.

Alternatively, hospitals can save money by relying primarily on staff, but hiring outside help for specific tasks, like writing code patches. Programmer fees range from $75 to $100 an hour for journeymen developers to $200 or more an hour for senior technical consultants. Y2K frenzy even spawned "code factories," companies that take code you’ve identified as rickety, update it and test it on site (see "Resources," on page 35 for contact information).

However, Printz says he hasn’t hired outside help--partly because he wants his in-house staff to understand system upgrades and partly because competent IS services are hard to come by. Palladino hasn’t hired help yet, but says she’ll review this decision when she budgets for next year.

5. Grade suppliers for Y2K awareness.Getting your company in millennium shape won’t spare you from headaches if companies you rely on grind to a halt at zero hour. Identify your top 20 most important suppliers and arrange to visit them to discuss first hand how they’re preparing for the future. Send questionnaires to your remaining suppliers to gauge their Y2K efforts. In both cases, track their improvements over time, and look for alternatives if you see suppliers dragging their feet, advises Riley.

6. Develop contingency plans.No matter how much you’ve prepared, make sure your master disaster plan and emergency systems are in shape. You may need them if the power fizzles or automated systems sputter. Also, identify which systems you can revert to manual processes in the event of a meltdown. Paychecks, for example, may need to be issued by hand once again.

Five Steps Toward the Millennium

NO MATTER IF YOU TACKLE Y2K YOURSELF or enlist the aid of a consultant, you’ll probably follow these steps:

Equipment Audit

What software and microprocessor-based hardware (ranging from PCs to biomed systems with embedded controllers) does your organization rely on? Include LANs and WANs and facilities equipment such as fire alarms.

Testing

Which systems crash or report erroneous data when you set their internal clocks ahead to important dates in the year 2000?

Master Plan

Using compliance test results, gauge the extent of your Y2K problem, estimate budgets, assign staff to fix problems and establish milestones.

Conversion

Replace hardware and software too obsolete to upgrade; convert code. Update legacy data based on two-digit date formats.

Test New Systems

Identify and fix incompatibilities introduced by new hardware and software.

Alan Joch is a contributing editor to Healthcare Informatics.

Sponsored Recommendations

ASK THE EXPERT: ServiceNow’s Erin Smithouser on what C-suite healthcare executives need to know about artificial intelligence

Generative artificial intelligence, also known as GenAI, learns from vast amounts of existing data and large language models to help healthcare organizations improve hospital ...

TEST: Ask the Expert: Is Your Patients' Understanding Putting You at Risk?

Effective health literacy in healthcare is essential for ensuring informed consent, reducing medical malpractice risks, and enhancing patient-provider communication. Unfortunately...

From Strategy to Action: The Power of Enterprise Value-Based Care

Ever wonder why your meticulously planned value-based care model hasn't moved beyond the concept stage? You're not alone! Transition from theory to practice with enterprise value...

State of the Market: Transforming Healthcare; Strategies for Building a Resilient and Adaptive Workforce

The U.S. healthcare system is facing critical challenges, including workforce shortages, high turnover, and regulatory pressures. This guide highlights the vital role of technology...