A recent study suggests that all electronic health record (EHR) systems may not be created equal as it relates to hospitals’ performance on Meaningful Use criteria. Certain vendors are more often associated with better Meaningful Use performance than others, according to a study published in the Journal of the American Medical Informatics Association.
The study, by Jay Holmgren of Harvard Business School, Julia Adler-Milstein, of the University of California San Francisco, School of Medicine, and Jeffery McCullough of the University of Michigan School of Public Health, assessed whether there was variation across EHR vendors in the degree to which hospitals using products from those vendors were able to achieve high levels of performance on Meaningful Use criteria.
As the study points out, the Medicare and Medicaid Electronic Health Record (EHR) Incentive Programs have sought to drive widespread EHR adoption among doctors and hospitals to realize quality and efficiency benefits. The programs were designed with two coordinated components: EHR Meaningful Use criteria to which providers and hospitals attest, and EHR certification criteria that vendors must meet to ensure that their products allow providers/hospitals to meet the Meaningful Use criteria. Prior research revealed that the majority of providers and hospitals have adopted a certified EHR and attested to Meaningful Use criteria. Less is known about variation across certified products in supporting providers’ ability to use EHRs, the researchers wrote.
The researchers also point out that prior studies have revealed variation in the quality of key capabilities of certified systems and the certification process itself has been criticized for only requiring that vendors meet a constrained set of functions that are known to them in advance.
For the study, the researchers created a cross-sectional national hospital sample from the Office of the National Coordinator for Health Information Technology (ONC) EHR Products Used for Meaningful Use Attestation public use file and the Centers for Medicare & Medicaid Services Medicare (CMS) EHR Incentive Program Eligible Hospitals public use file. The researchers used regression models to assess the relationship between vendor and hospital performance on six Stage 2 Meaningful Use criteria, controlling for hospital characteristics. And, the researchers also calculated how much variation in performance is explained by vendor choice.
The study results indicate that three hospitals involved in the study were in the top performance quartile for all Meaningful Use criteria, and all used Epic. Of the 17 hospitals in the top performance quartile for five criteria, 15 used Epic, one used MEDITECH, and one used a smaller vendor. For the 68 hospitals in the top quartile for four criteria, 44 (64.7 percent) used Epic, 8 (11.8 percent) used Cerner, and 6 (8.8 percent) used MEDITECH. For the 355 hospitals that were not in the top quartile of any criteria, 102 (28.7 percent) used MEDITECH, 89 (25.1 percent) used McKesson, 72 (20.3 percent) used Cerner, 51 (14.4 percent) used MEDHOST, and 24 (6.8 percent) used Epic.
The researchers found significant associations between specific vendors and level of hospital performance for all six Meaningful Use criteria. “Epic was associated with significantly higher performance on five of the six criteria; relationships for other vendors were mixed, with some associated with significantly worse performance on multiple criteria,” the study authors wrote. EHR vendor choice accounted for between 7 percent and 34 percent of performance variation across the six criteria, according to the study authors.
“In addition, the specific vendor appears to matter, with some vendors associated with significantly better or worse performance across multiple criteria. This suggests that hospitals may need to take vendor choice into account if they seek higher achievement on meaningful use criteria and cannot rely solely on the EHR certification process to ensure that all systems are equally capable,” the study authors wrote.
The fact the one vendor was significantly positively associated with multiple criteria while others reflected a mix indicates that vendors are making different design choices, and indicates that some vendors are simply doing a better job than others at creating systems that enable hospitals to achieve high levels of performance on meaningful use criteria, the study authors noted.
What’s more, the researchers wrote, “Our results also suggest that policymakers should consider strengthening the certification program to minimize such vendor-to-vendor differences or take steps to improve transparency about this variation.”
The study authors contend that the study findings have important implications for both policymakers and hospital leaders. “While all hospitals in our sample met Stage 2 meaningful use requirements, there was significant variation in achievement across certified vendors. This suggests that the current EHR certification process does not achieve a key intended goal of giving providers confidence that they are buying a ‘good’ EHR. This may be due to the fact that the certification process only tests a narrowly constrained set of ‘technical; meaningful use requirements and does so in a controlled laboratory setting,” the researchers wrote.
The study authors suggest that policymakers consider including more real-world trials to assess how EHRs perform in simulated clinical events that are not revealed to vendors beforehand. And, the study results may also indicate that the quality of vendor support services can impact Meaningful Use performance. “Policymakers may therefore wish to explore a certification process that includes an assessment of the quality of these services that holistically evaluates software and vendor support services,” the researchers wrote.
For hospital leaders, the study underscores the need to make careful choices when selecting an EHR vendor and offers some guidance on which products may be stronger, the study authors stated. However, given that vendor choice explained no more than one-third of performance variation, success may not be as simple as picking the “best” vendor. “Additional evidence revealing that hospitals have achieved disparate quality results using similar software systems further suggests the importance of organizational factors such as implementation and vendor services, IT staffing and support, training programs, and commitment to optimizing EHR use, which are likely more important to achieving high levels of performance,” the study authors wrote.
