CMIOs Report Frequent Clinical Decision Support Tool Malfunctions, Survey Finds

A survey of chief medical information officers (CMIOs) found that 93 percent of CMIOs reported experiencing at least one clinical decision support systems malfunction and 38 percent reported four or more malfunctions a year, according to a study published in the Journal of the American Medical Informatics Association.

In addition, the survey found that two-thirds of CMIOs experienced clinical decision support systems (CDSS) malfunctions at least annually.

The survey was a part of a study led by researchers at Brigham and Women’s Hospital in Boston examining the frequency of CDSS malfunctions—situations in which a CDSS does not function as it was designed and or expected to—and identifying patterns in malfunctions.

While evidence suggests that clinical decision support (CDS) tools, when used effectively, can improve healthcare quality, safety and effectiveness, there are risks associated with using CDS tools and safety issues have been reported, the researchers wrote.

“CDSS malfunctions are widespread and often persist for long periods,” the researchers wrote in the study. “The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate.”

To examine the issue of CDSS malfunctions, the researchers focused on four case studies of malfunctions that occurred at Brigham and Women’s Hospital and identified key factors that contributed to the malfunctions. The researchers hypothesized that the CDSS malfunctions that occurred at BWH are representative of the types of CDSS malfunctions that occur in self-developed and commercial EHR systems around the world.

One case study focused on a situation in which an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system. In the second case study, an alert for lead screening for children stopped working when the rule was inadvertently edited. For the third case study, a software upgrade of the electronic health record software caused numerous spurious alerts to fire. And, the fourth case study focused on a malfunction in an external drug classification system which caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one.

Based on these case studies, the researchers then conducted a preliminary survey of CMIOs to assess whether similar CDSS malfunctions occurred at other sites. CMIOs who responded to the survey reported that they had implemented a range of different CDS types, with nearly all having drug-drug interaction checking and drug-allergy alerts and a sizeable number having a range of other types of CDS.

“The most common contributing factors to CDSS malfunctions reported by the CMIOs mirror BWH’s experience: 18 of the survey respondents reported CDSS malfunctions occurring at the time of an upgrade of their EHR software, and 18 reported issues in connection with changes in data codes or data fields (both of which are known high-risk events for CDSS malfunctions,” the researchers wrote.

Inadvertently disabling rules, upgrades of other systems, database corruption, and other system malfunctions were also commonly reported causes of CDSS malfunctions, according to the survey.

When polled out how they had detected the malfunctions, the majority (83 percent) said detection came from reports from end users, followed by CMIOs identifying malfunctions during their own use of the EHR (48 percent). Many fewer CMIOs reported identifying CDSS malfunctions during ongoing system testing (31 percent) or when reviewing reports of CDSS performance (21 percent).

And, 62 percent of the CMIOs who responded to the survey expressed they were “not very confident” or “not at all confident” that their existing processes and procedures are sufficient to prevent or detect all CDSS malfunctions before they reach the user. And no CMIOs in the survey said they were totally confident in their existing processes and procedures to sufficiently prevent or detect all CDSS malfunctions.

In conclusion, the researchers stated that CDSS malfunctions occur commonly, in fact more frequently than previously thought, and that existing detection systems are inadequate to detect CDSS malfunctions before they reach users. “As CDSSs becomes more complex and widespread and clinicians increase their reliance on them, improved processes and tools for preventing and detecting CDSS malfunctions are essential,” the researchers wrote.

To address the issue, implementers of CDSS should continually and proactively monitor their CDSS for malfunctions, and EHR developers should provide system implementers with enhanced tools for reporting on and monitoring CDSSs. Such tools should proactively alert CDSS implementers to potential CDSS malfunctions that are identified using statistical or rule-based approaches, the researchers wrote.

In addition, policymakers also should make use of levers to encourage development and adoption of CDSS monitoring capabilities, and the researcher also asserted that as “proposals for increasing oversight of health IT safety are implemented and safety reporting systems for EHRs become available, data on CDSS malfunctions should be collected and analyzed as an important subcategory of health IT safety events.”