Open-Source AI Tools Show Promise in Comparative Study

A paper published March 14 in JAMA Health Forum suggests that an open-source AI tool can perform as well as a proprietary closed-source model on complex diagnoses. This is significant, the researchers say, because “institutions may be able to deploy high-performing custom models that run locally without sacrificing data privacy or flexibility.”

The challenger open-source AI tool called Llama 3.1 405B performed on par with GPT-4, a leading proprietary closed-source model, on 92 mystifying cases featured in the New England Journal of Medicine weekly rubric of diagnostically challenging clinical scenarios.

In discussing the performance of the two models, researchers said the findings suggest that open-source AI tools are becoming increasingly competitive and could offer a valuable alternative to proprietary models.

“To our knowledge, this is the first time an open-source AI model has matched the performance of GPT-4 on such challenging cases as assessed by physicians,” said senior author Arjun Manrai, Ph.D., an  assistant professor of biomedical informatics in the Blavatnik Institute at Harvard Medical School, in a statement. “It really is stunning that the Llama models caught up so quickly with the leading proprietary model. Patients, care providers, and hospitals stand to gain from this competition.”

The study’s lead author, Thomas Buckley, a doctoral student in the new AI in Medicine track in the Harvard Medical School Department of Biomedical Informatics, noted that one advantage is that open-source models can be downloaded and run on a hospital’s private computers, keeping patient data in-house. In contrast, closed-source models operate on external servers, requiring users to transmit private data externally. “The open-source model is likely to be more appealing to many chief information officers, hospital administrators, and physicians since there’s something fundamentally different about data leaving the hospital for another entity, even a trusted one,” said Buckley in a statement. 

Second, medical and IT professionals can tweak open-source models to address unique clinical and research needs, while closed-source tools are generally more difficult to tailor. “This is key,” said Buckley. “You can use local data to fine-tune these models, either in basic ways or sophisticated ways, so that they’re adapted for the needs of your own physicians, researchers, and patients.”

However, closed-source AI developers such as OpenAI and Google host their own models and provide traditional customer support, while open-source models place the responsibility for model setup and maintenance on the users. And at least so far, closed-source models have proven easier to integrate with electronic health records and hospital IT infrastructure.