A recent initiative by Kaiser Permanente sought to demonstrate the value of artificial intelligence-based natural language processing (NLP) technology, as clinicians identified more than 50,000 patients with cases of aortic stenosis, a common heart valve disease.
Researchers trained NLP to comb through more than one million electronic medical records and echocardiogram reports to recognize certain abbreviations, words, and phrases that were indicative of aortic stenosis. The software identified nearly 54,000 patients with the condition, a process that would likely have taken years for physicians to perform manually, according to Kaiser Permanente.
In an e-mail Q&A, Matthew Solomon, M.D., Ph.D., a cardiologist at The Permanente Medical Group and a physician researcher at the Kaiser Permanente Division of Research in Oakland, Calif., discussed his team’s research findings, which were published in the Cardiovascular Digital Health Journal.
Healthcare Innovation: Before getting into the specific findings of your research, could you describe some of the ways that NLP has been used for clinical applications previously?
Solomon: AI techniques including NLP have slowly been adopted into health care settings over the past 5-10 years. Until recently, these techniques were used primarily in research settings, with applications including the identification of rare disease cohorts or outcomes that could not be obtained without extensive manual chart review. Increasingly, the techniques are being integrated into health system operations in ways that are moving us toward becoming true learning health systems.
What are some reasons that the integration of NLP has been slow in healthcare?
In general, the healthcare industry has overall been a slow adopter of technology, certainly slower than other industries. But with the explosion of technology into consumers’ daily lives, and novel technologies reaching the mainstream, this is changing. It takes significant investment to adopt, learn, and integrate new technology in healthcare settings, and many health systems do not have a proper long-term vision or focus, which has slowed adoption. Kaiser Permanente and its integrated care model has created opportunities for us to be a leader in this area. We were one of the first major healthcare systems to adopt an electronic medical record in the 2000s, and we are continuing to push the healthcare envelope and lead the way.
Is the large amount of unstructured data in healthcare a key reason that health systems should consider investing more in NLP-based approaches?
Although we live in the era of big data and data analytics, it remains very difficult in medicine to access all that data, and particularly to identify patients with complex conditions like valvular heart disease. That makes it difficult to study them, track practice patterns, and do population management. Currently, health systems track patients using diagnosis or procedure codes, which are mostly created for billing purposes. These can be very non-specific and are not useful for clinical care or research. Without accurate and systematic case identification, population management and research on valvular heart conditions and many other complex conditions isn’t possible. Natural language processing algorithms made it possible to teach a computer how to do this for us, which made this work in both research and operations possible.
Why are traditional approaches involving diagnosis codes for conditions like valvular heart disease less accurate and lacking in specificity? What are the impacts of that lack of specificity on population health efforts?
Diagnosis and procedure codes are designed for insurance billing, not clinical care, and are simply too nonspecific. They do not include detailed data about a specific medical condition. For example, a patient with moderate or severe aortic stenosis — a narrowing of one of the primary heart valves which warrants close follow-up — is entirely different than a patient with mild aortic valve disease. Those nuances are not included in diagnosis or procedure codes, so if a patient gets coded with “aortic valve disease,” it’s not clear exactly what type or severity of the disease the patient has. The code could have been applied for an entirely different clinical problem of the aortic valve. High-quality population management and research on valvular heart conditions aren’t possible without accurate and systematic case identification. Further, the data needed to properly characterize patients with valvular heart disease is buried in echocardiography reports. Like many radiology reports, these are often free-text fields that are heterogeneous and unstructured, which means they cannot be easily queried. The only solution to identify these patients is to either have an army of humans pore over 1,000,000 echocardiography reports, which is not feasible. Or we could develop NLP methods and teach a computer how to do that for us, which is what we did.
Can you briefly describe how your team developed and validated NLP algorithms to identify aortic stenosis (AS) cases and associated parameters from semi-structured echocardiogram reports?
For each data point to be abstracted, a separate NLP algorithm must be developed. First, it is developed and tested on a single data set. Once that algorithm is performing well, it is applied to a new data set. Typically, it won't work as well on the new data set, and adjustments to the algorithm need to be made. This is because there are corner cases in each data set that weren’t present in the former one. Once the algorithm is performing adequately, it is again tested on a new data set, and this process is repeated until the validation on a new data set performs with the accuracy you are looking for. In our case, that is greater than 95 percent positive and negative predictive value.
When you compared the NLP algorithms to the ICD-9/10 diagnosis code–based definitions, how much greater sensitivity did you find from the algorithms? Any surprising results?
The NLP algorithms were markedly greater in both their sensitivity and specificity for identifying aortic stenosis. However, the more important point is that we were able to identify the rich clinical details for each patient, such as aortic stenosis severity, and the specific quantitative parameters throughout the entire echocardiogram for each patient. No amount of administrative coding could ever capture that much information, which is critical from both the research and operational perspective. What is most astonishing is how powerful these methods can be once they are developed. We now are able to build what is likely to be the largest valvular heart disease population management program in the world.
Your paper also says that no published study has demonstrated the superiority of an NLP-based approach compared to using traditionally used administrative diagnostic codes across multiple coding eras. Do you think it is critical that this type of study be done around how other conditions are identified?
There is a vast amount of important clinical data buried within electronic medical records. But it remains out of reach to most researchers because it is unstructured and cannot be easily searched. AI-based technologies like NLP can overcome this challenge, improving the efficiency and feasibility of research by allowing us to build large cohorts of patients to study without manual chart review. This process also yields much larger datasets that can produce more precise and generalizable results that can be used to study other types of heart conditions, as well as for other types of diseases. So, yes, it would be ideal to continue this work for other health conditions as well, to improve our ability to conduct research and population management.
What would it take for Kaiser Permanente to start using them on a systemwide basis?
We are already doing so.
What are some next steps in your research?
Identifying patients with complex conditions that cannot be otherwise well characterized is only the start. Once we are able to build large cohorts for patients with various cardiovascular conditions, we can begin to do risk-based population management. This will allow us to not only focus resources on the sickest patients, but to also identify trends and novel predictors that affect outcomes or disease progression so that we can identify and intervene earlier to help the patients who may be most likely to progress to more severe disease.
