The Challenge of AI Language Models in the Medical Field: Frequent Hallucinations in Clinical Summar | Generative ai in finance and banking | What are foundation models in generative ai | Microsoft generative ai tools download | Turtles AI
Highlights:
- A recent study found hallucinations in almost all medical summaries generated by GPT-4o and Llama-3.
- The identified hallucinations mainly involved symptoms, diagnoses, and medical instructions.
- GPT-4o showed more inconsistencies than Llama-3, which produced shorter and less detailed summaries.
- The research emphasizes the need for a more effective system to detect and manage AI hallucinations in clinical settings.
The use of large language models (LLMs) in the medical field raises significant safety concerns. A recent study by the University of Massachusetts Amherst reveals that AI models, including GPT-4o and Llama-3, frequently hallucinate in medical summaries, with potentially critical implications for patient care.
In recent years, the adoption of LLMs in healthcare has accelerated, partly due to the promise of alleviating clinician burnout through automated medical summaries. However, the medical community continues to express concern about so-called "hallucinations," situations in which an AI model produces incorrect or misleading information. The study conducted by the University of Massachusetts Amherst focused on this phenomenon, specifically analyzing the performance of two state-of-the-art LLMs: OpenAI’s GPT-4o and Meta’s Llama-3.
The research team collected 100 medical summaries produced by these models, equally divided between GPT-4o and Llama-3, with the intent to assess the reliability and accuracy of the generated information. The results were alarming: hallucinations were found in almost all of the analyzed summaries. According to Prathiksha Rumale, one of the study’s authors, the hallucinations mostly occurred in relation to symptoms, diagnoses, and medical instructions, highlighting the challenge of handling domain-specific knowledge in the medical field for language models.
The hallucinations identified in GPT-4o’s sample totaled 327, distributed among medical event inconsistencies, reasoning errors, and chronological inconsistencies. Specifically, 114 cases of incorrect reasoning and three chronological errors were reported, with most inconsistencies related to medical events. On the other hand, the summaries generated by Llama-3, although shorter and less detailed, showed 271 inconsistencies in medical events, 53 reasoning errors, and one chronological inconsistency. Tejas Naik, another author of the study, emphasized that the LLMs’ ability to generate fluent and plausible sentences, capable of passing the Turing test, could pose a risk, especially if these summaries are unfaithful to the original medical records.
The primary risk is that, if not detected, these hallucinations could lead to serious clinical errors. For instance, a model could mistakenly interpret a common cold as a throat infection, leading doctors to prescribe inappropriate medications and underestimate the potential risks of viral transmission. Even more severe could be the omission of a documented drug allergy, ignored by the model, which could result in a potentially fatal allergic reaction.
The research team concluded that there is a need to develop a more robust methodological framework for detecting and categorizing AI hallucinations in the clinical context, to improve the reliability of these tools. This recommendation opens a broader discussion on the integration of AI in the healthcare sector, where the accuracy and reliability of information are crucial for patient safety.
