Faced with declining or flat revenues, shriveling reimbursements and escalating competition, hospitals and integrated delivery systems across the country are taking a harder look at a practice that is standard in most other industries: benchmarking. Managed care organizations too must begin to analyze internal and external data to survive, as medical costs continue to wreak havoc on HMO profits. Financial analysts are banking on the ability of HMOs to provide information to the public on costs, pricing and performance, and implement medical management programs to get ahead in the market. Yet the providers of care hold the key to data that is largely unmined, or in many cases, uncollected.
The good news: A wealth of companies offer products and services that can help organizations manage performance data and benchmark processes. The bad news: There’s little consensus on "best practices" or the methodology to measure them, and meaningful comparative data is scarce.
Benchmarking in its simplest definition is the discipline of using comparative data to examine and measure the processes within an organization. Such data may come from inside the organization, from marketplace studies or from state and national databases that track industries--but the goal is to identify areas for improvement. In healthcare that means lower costs, better quality and hopefully, healthier patients.
The intent of benchmarking is not necessarily to emulate other organizations, but more to stimulate "out of the box" thinking, according to Jack Partridge, vice president of the Chi Systems division of Superior Consulting, Southfield, Mich. "Innovative solutions utilized by other organizations serve as building blocks for new approaches and may, in fact, result in a new and improved solution. It is a classic approach to not reinventing the wheel."
According to healthcare decision support firm HCIA, Inc., Baltimore, if all U.S. hospitals performed as well as the 100 did in its annual survey of the nation’s top performing acute-care hospitals, the industry could save more than $24 billion, reduce inpatient mortality and complication rates by 22 percent and average length of stay by a half day, and increase profitability by more than 50 percent. The survey measures hospitals over nine operational and clinical performance indicators and compares them to other organizations in the same peer group (determined by number of beds and teaching or non-teaching affiliation). Results from the 1997 survey revealed that the South and the West produced the highest number of top-performing hospitals. Such results are probably more attributable to the effects of competition, managed care penetration and consumer demand in local markets than benchmarking, according to Dave Krone, senior director in HCIA’s Denver office. However, since the fourth quarter of 1997, industry analysts have been predicting that the easy cost-cutting measures in healthcare have already been achieved. Implementing benchmarking programs may be the path to any further progress in the financial health of the industry.
But cost is just half of the equation organizations need to examine. Benchmarks that look only at financial and utilization statistics such as length of stay and number of FTEs rankle the medical community, causing division not only between clinicians and administrators but also between the administrators and a public that is becoming increasingly concerned about medical management’s effect on the quality of care. Clinicians are data-driven and will use information that can help them improve practices, notes Cynthia Burghard, a research director specializing in medical management and clinical outcomes at The Gartner Group’s healthcare IT division, Wakefield, Mass. Still, Burghard issues a common warning: "Coming out with some external norm that physicians have to meet is a sure formula for disaster."
Culture clash
Before you address the practicalities of a benchmarking project--why you’re doing it, what you want to measure and how you’ll get the data--executives need to prepare for the inevitable resistance from the ranks. As one healthcare executive summarizes: "No one likes to be in the spotlight."
The difference in perspective is hard to get around: Clinicians want the freedom to make what they consider the best decisions possible for the patient, while administrators, no matter how committed they are to boosting quality of care and patient satisfaction, need to focus on the bottom line.
Mercy Health System, an integrated delivery system in Philadelphia, is in year two of a three-year contract with MECON, Inc., San Ramon, Calif., a software and consulting firm specializing in operational benchmarks for healthcare. As part of a broader continuous quality improvement program, Mercy hired MECON to help the IDS analyze its performance at both the department and system level. So far, there has been little improvement in individual departments there, and participation from the medical staff has been low, according to a Mercy administrator who spoke on condition of anonymity. "I suspect it’s a cultural thing," the administrator says. As a result, Mercy is now using hiring approvals as leverage to improve performance. Departments with unmet performance goals are in some cases having to wait to hire new staff until processes improve. The thinking, says the administrator, is that after the improvements take place an extra staff member may not be necessary. Call it what you will--encouragement, motivation, delayed gratification, bribery--in some organizations change is being force-fed.
Yet at St. Luke’s Medical Center in Milwaukee, clinicians have not battled a benchmarking program that began there four years ago to look for inefficiencies in the hospital’s cardiac services, oncology and orthopedics departments, according to Mike Panosh, VP of finance at Aurora Health Care, St. Luke’s parent organization. "The medical staff is pretty used to working with outside consultants," he says. In fact, says Panosh, some physicians initiated the changes themselves.
Length-of-stay rates began to drop early in the process, which Panosh attributes to a heightened awareness about performance. "We started seeing results immediately because everyone knew a consultant was working with us." Panosh cites 25 percent reductions in length of stay and close to that in cost per case for some physicians. Clinical pathways for cardiac patients have also been developed based on findings from the studies. The IDS hired HCIA to conduct the studies, using the hospital’s data and information obtained from interviewing staff members to set performance goals against a voluminous national database of benchmarks HCIA updates and maintains. The organization has discovered that less time in the hospital does not mean patients are worse off. Even with the dramatic reduction in length of stay in these high-utilization areas, Panosh defends the organization’s commitment to patient care, noting that quality monitors and patient satisfaction surveys have shown an improvement in quality since the organization began benchmarking.
Steve Shaha, research director at The Gartner Group’s healthcare IT division is skeptical of the motivations behind benchmarking. He says benchmarking is dangerous if the results are top-down mandates rather than guidelines or points of discussion: "Too often it’s adversarial. If a practitioner gets his data and it’s worse than a peer’s then it’s time to worry. The expectation is you need to change." Part of this is the nature of the beast--performance reviews can put anyone in a defensive posture. Still, Shaha says that benchmarking in healthcare continues to focus largely on financial performance, with little attention to clinical quality. Shaha, who holds a PhD in applied statistics and research methods and has led the development and implementation of more than 200 outcomes projects, says that benchmarking used appropriately can help set standards of care that will lower costs--yet he sees little evidence that this has occurred: "If benchmarks have had an impact on the cost of care it’s mostly by guns being held to the heads of providers."
No standards, no measures
Cultural conflicts aside, there are glaring practical reasons why benchmarking in healthcare may be more difficult--and less advanced--than in any other industry. It comes back to the same old problem hindering progress on many fronts in healthcare: a lack of standards--and not just in the way care is delivered but also how organizations operate and report and collect information. Variables such as payor mix or mortalities following discharge can produce numbers all over the map. Such differences between organizations even in the same town make the notion of comparing data almost laughable.
A classic example of how data is either inaccessible or useless is claims information, notes Burghard. Entering data into the incorrect field of the claims system, or finding whole pockets of data sitting in a shoebox somewhere is not uncommon, she says. And if you do manage to find the complete set of data, there is the problem of "cleaning" it for analysis. Many decision support systems on the marketplace adjust data for severity, taking into account complications and interventions that could make one physician’s outcome much different from another’s for the same diagnosis.
But severity adjusting is not an exact science, as Joan Guzik, director of program development and planning at NYU Medical Center can attest. The organization installed a decision support system from HealthShare Technology, Inc., Acton, Mass., last August to help analyze indicators like market share and costs by service or geographic area. Guzik admits that even with the system’s capability to severity adjust, analyzing at the micro level has revealed perplexing inconsistencies: "We had to do a lot of digging to figure out why that was happening."
Kevin Lieb, vice president at HCIA, agrees that finding reliable data that hospitals can use has proved laborious--but maybe because the process of benchmarking is still so new. "Healthcare is really in its infancy as a business," he says. "We’re probably a good 10 years from getting solid credible data."
Perhaps more vexing and widely controversial is the loose notion of clinical best practices, along with a lack of data and information on how nationally-reputable organizations treat patients and run their facilities. Explains Burghard: "All of the big academic medical centers think they have best practices. But they tend not to go outside of the individual organization."
While the the debate on best practices rages on (see "Wrestling Over Best Practices," May 1997), activity on a number of fronts is helping to identify and define them. The most visible of such initiatives is the Joint Commission on Accredited Healthcare Organization’s ORYX program that took effect this year, requiring hospitals and long-term care organizations to submit data on two to five clinical performance indicators by the end of the first quarter, 1999. Requirements for home care agencies, behavioral health and delivery networks are being established now. ORYX data will not affect an organization’s accreditation status, according to Deborah Nadzam, vice president of performance measurement at JCAHO--but that will likely change: "Over time as more becomes known about outcomes data, there may be more use of the data directly into accreditation."
More sophisticated data
The National Committee for Quality Assurance (NCQA)--the accrediting body for managed care organizations--has just released HEDIS 3.0/1998, the next version of the data set for measuring quality in health plans. NCQA spokesperson Brian Schilling says HEDIS meaures are becoming more sophisticated to include performance data on chronic illnesses like diabetes and cancer, as well as member satisfaction. NCQA posts some of the data collected from health plans on its Web site, and sells a complete database now in its second year, the Quality Compass.
Ultimately, efforts by groups like JCAHO, NCQA, FAACT (Foundation for Accountability) and the Presidental Advisory Commission on Consumer Protection and Quality in Healthcare should help in developing national benchmarks and standards of care. Says Lieb: "In the long run I think it (ORYX) will clean up all the benchmarking services out there."
However, just how relevant national or even regional benchmarks are to an organization is unclear. According to Shaha, benchmarking programs are most successful when they incorporate data at the national, organizational and individual provider level. He recommends the practice of "baselining:" assessing the organization’s internal status, setting targets and then measuring progress from the initial baseline over time. "Those are the guys that are saving three percent to 17 percent a year on costs and are statistically improving outcomes because they’re benchmarking to themselves," he says.
Understanding the barriers to benchmarking does not preclude success; beginning a benchmarking program now is better than waiting years for the optimum environment to materialize. "There are always ways to make informed decisions even if the data isn’t apples to apples," says Ron Webb, senior benchmarking specialist at the American Productivity & Quality Center, (APQC) a Houston-based nonprofit research firm that conducts benchmarking studies for its membership base of Fortune 1000 companies. Members that participate in a study on a particular business process all share in the results.
Decision support systems are crucial in helping companies crunch the data and generate reports, but many are still in the early stages of development and adoption in healthcare. Superior’s Partridge comments that the biggest problem with decision support systems is that people do not know how to use the information to make decisions.
Part of the solution is making the information easy to digest, according to Richard Siegrist, CEO of HealthShare and an adjunct lecturer at the Harvard School of Public Health. Development at his firm has focused not only on turning data into knowledge, but also presenting the data in a graphically-appealing way. Siegrist says the system is helping his customers glean valuable insight into market dynamics so they can better project services, contracts and other business opportunities: "Market share is an area that a lot of people look at because unless you get down to the detailed level, you don’t really know where you’re losing patients to another hospital, or if you start to market a program, where you want to focus," Siegrist explains.
Clinical outcomes data
To get a complete picture of performance, however, benchmarks must integrate cost and clinical outcomes information. Many healthcare information experts will agree that obtaining the precious outcomes data is the hardest part, for a host of reasons: lack of systems integration, low implementation of computer-based patient record systems, and difficulty in getting senior management support for funding costly outcomes management and research initiatives.
"I don’t see the market moving toward integrating the clinical and cost areas yet," says David Randall, HEDIS project director at Regence Blue Shield in Seattle. Part of the problem lies within HEDIS itself, he says, because the measures are not yet looking at outcomes.
Randall questions the credibility of the HEDIS program because of what he calls inconsistencies between results and accreditation, which make outcomes research a difficult sell.
Part of the buy-in problem, adds Burghard, is that for payors, the reward of investing in outcomes research is less immediate than for providers. And organizations need to commit not just technology dollars but people dollars. In some cases, she says, starting an outcomes management program requires hiring a department of people who understand statistics in healthcare. Organizations banking on survival over the long term may have little choice but to find the cash. "It needs to be a cost of doing business," Burghard says.
Selling change
According to Shaha, clinical outcomes and customer satisfaction have taken the back seat for too long when organizations analyze performance; even the notion of outcomes is often mistaken for utilization. "Length of stay does not mean you got a good clinical outcome," he says. "It just means you got out of the hospital."
Benchmarking of patient satisfaction and patient-reported outcomes is an emerging area that promises to take a central role in the outcomes picture as both employers and payors recognize the growing power of consumers. While there are several firms offering products to help collect and measure this data (see "Where’s the Data?," p. 46), the problem again goes back to a lack of consensus on a standardized survey instrument. The future for firms that can bring new ideas and sophisticated systems for patient-reported data is bright. Says Shaha: "Integrated outcomes is the science of the future and will be the determining factor for which organizations succeed and fail."
Healthcare is known for its abilities in research and analysis--but what about using such information to engineer change? The APQC’s Webb says it’s a rare week when he doesn’t get a call from a healthcare organization interested in a benchmarking study, yet few of those callers indicate a readiness to make change. Adds Lieb: "The biggest barrier is getting clients to believe in the benchmarks."
And Partridge cautions against using the data for quick-fix solutions rather than investigating the root of the problem--a mistake he says is common in healthcare benchmarking: "Unfortunately, organizations have misused benchmarking to establish goals and make staffing and cost cuts without making appropriate changes to the business processes." Echoes NYU Medical Center’s Guzik: "You have to be careful with how you use the data."
The biggest obstacle to change in the industry may be convincing healthcare professionals to look beyond the white picket fence. HealthShare’s Siegrist makes this blunt assessment: "Healthcare organizations are recognizing that the old line ’my organization is unique’ doesn’t work with payors or employers anymore." Healthcare should consider the fact that companies in other industries have accomplished much by keeping an open mind.
For example, starting in 1991 GE Medical Systems, Inc., began a study of the manufacturing, IS and business processes at other multinational firms such as Digital Equipment, Fuji, Xerox and Hewlett-Packard. The benchmarking program that came out of the study has resulted in savings of $21 million in facility, personnel and inventory reductions. The company has cut in half its number of warehouses, and now has the capability to view 99 percent of its inventory online, according to a report on the project by APQC.
Partridge suggests that healthcare could improve the process of medication flow throughout the hospital by looking at production processes in manufacturing. Dorothy Yu, partner with Coopers & Lybrand, Boston, gives the example of L.L. Bean’s reputation for customer loyalty as a case study for patient and member satisfaction. Still, while healthcare slowly but inevitably transforms itself from a cottage industry into a business, change and standardization must be tempered by the human compassion that is core to caregiving. Explains Lieb: "There’s an element in healthcare called the human life which tends to throw things off when making decisions."
When healthcare executives look to the power of benchmarking they are wise not to forget the ultimate customer--and it’s not the chairman of the board, the insurer or the employer. After all this collecting and measuring, shares one industry expert, the worst outcome of all would be if attaining an industry standard becomes more important than the personal care of the patient.
Where’s the Data?
FINDING THE DATA--BOTH internal and external--that you’ll need to begin a benchmarking program is no easy task. Although information systems are doing a better job of providing the level of detail, reporting and analysis healthcare organizations need, there are other considerations. "Often what you don’t know going into a project is how much information is out there, how willing companies are to share information and how accessible they are," says Dorothy Yu, partner with Coopers & Lybrand Consulting, Boston. Adds Ron Webb, senior benchmarking specialist at the American Productivity & Quality Center, Houston: "The healthcare environment is extremely competitive. People are leery of sharing information with other healthcare organizations." Coopers & Lybrand addresses the conflict by offering knowledge in return, says Yu: Every company that participates in a benchmarking study receives a copy of the final report.
But there are many public and private sources of data available on the Internet or from consultants and decision support vendors to supplement your research. For instance, Medicare maintains one of the largest databases of patient information in the industry--if not the largest. The Medpar database contains data for all Medicare beneficiaries using hospital inpatient services by state and DRG. Medpar covers the years 1990 to 1996 on these fields: total charges, covered charges, Medicare reimbursement, total days, number of discharges, and average total days.
The Healthcare Cost and Utilization Project, sponsored by the Agency for Health Care Policy and Research (AHCPR), Washington, D.C., has developed two databases, the Nationwide Inpatient Sample and the State Inpatient Database. The state-collected discharge data in the two products runs from 1988 to 1995, and is designed for research activities approved by the agency, according to the AHCPR’s Judy Ball. Ball says several hundred copies of the databases have been sold. The agency will continue to update it as new data becomes available. The National Committee for Quality Assurance (NCQA) sells its Quality Compass database of HEDIS results and other comparative health plan quality information to health plans, consultants and employers. Additionally, most states collect and store claims and other administrative data on patients. Some academic medical centers also maintain databases that can be requested for research purposes.
Once you get the data set, the next crucial step is refining and managing it. Helping to solve the proverbial "garbage in, garbage out" problem is where information systems will come into play. Says Kevin Lieb, vice president of healthcare benchmarking and decision support vendor HCIA, Inc., Baltimore: "As databases and reporting methodologies mature, you’ll see more effective decision-making."
Key Information Systems
THE SYSTEMS TO MEASURE AND MONITOR CARE ARE BECOMING MORE WIDELY AVAILABLE BUT MANY OF the products are still at the beginning of their life cycles. Cynthia Burghard, research director specializing in clinical outcomes and medical management at The Gartner Group’s healthcare IT division, Wakefield, Mass., outlines the following five types of outcomes systems with a sample list of vendors for each:
Hospital quality assurance and risk management systems: Perform functions like risk management, peer review, staff credentialing and utilization management.
Dorn Technology Group, Livonia, Mich., IST, Reston, Va., MIDS, Inc., Tucson, Ariz.
Risk Alert, Coral Gables, Fla.
Inpatient hospital outcomes systems: Help hospital track clinical performance and provide comparative data.
Apache Systems, Inc., Washington, D.C., Iameter, San Mateo, Calif., and Atlanta
MediQual Systems, Inc., Westborough, Mass., Quality Outcomes in Medicine, Boston
The MEDSTAT Group, Ann Arbor, Mich.
Patient-centered outcomes systems: Collect data from the patient to evaluate the process and outcome of healthcare from the patient’s perspective, measuring both quality and satisfaction.
Assist Technologies, Phoenix, Gallup, Inc., Princeton, N.J., HCIA, Baltimore
Health Outcomes Technologies, Doylestown, Pa., Solution Point (NCG Research), Nashville
Velocity Healthcare Informatics, Inc., (subsidiary of Object Products, Inc.), Minneapolis
Quality indicator systems:*Measure the process and quality of care from a delivery system perspective.
Codman Research Group, Andover, Mass., CSC Healthcare, Farmington Hills, Mich.
DataMedica, Minneapolis, HBOC/Amherst Product Group, Amherst, Mass.,
HBOC/GMIS Product Group, Malvern, Pa., HCIA, Baltimore, HPR, Cambridge, Mass.
O’PIN Systems (an Applied HealthCare Informatics, Inc. company), Bloomington, Minn.
Transition Systems, Inc., Boston, The MEDSTAT Group, Ann Arbor, Mich.
Value Health Sciences, Santa Monica, Calif.
* For additional vendors see (http://www.jcaho.org) under ORYX program.
Outcomes research systems: Study overall effectiveness of medical care including the effect of the healthcare process and the health and well being of the patient. Sometimes called "disease management." According to Burghard, there are no commercial systems on the marketplace to meet this need today.
Polly Schneider is senior editor at Healthcare Informatics.
