Safeguarding Your Practice: Why AI Medical Summaries Should Be Human-Backed

AI use in medical practices has exploded in recent years. According to a survey by the American Medical Association, 66% of physicians used healthcare AI in 2024. With so much “buzz” around the topic, though, many professionals are skeptical about going all in. Concern around AI misuse has entered the public stage, with the US Patent and Trademark office releasing a memorandum on unsupervised AI use (“parties are responsible for the content of their filings”) and a US lawyer sanctioned for using ChatGPT.  

Why? AI tools can hallucinate. These “hallucinations” occur when an AI model provides information to a user that is incorrect, like an AI-generated image of a hand with 7 fingers. While an average human would spot the problem in seconds, the AI model can’t. 

Of course, the AI would perform better if the dataset was nothing but hands – that’s why custom software solutions designed for medical summaries often outperform more generalized LLM tools. However, healthcare and claims data is unique, and medical summarization still needs human oversight in order to remain compliant and protect patients. 

Why Protecting Patient Health Information is Important When Using AI

Protected health information (PHI) refers to any information about health status, health care, or payments for healthcare created by a healthcare provider or their business associate. If this information can be traced back to the individual patient, it’s PHI – but it’s not that simple. 

As technology evolves, even de-identified data can carry some risk. Dominant tech companies (like Meta or Google) collect so much data from users that they risk re-identifying de-identified patient information. For example, Google tracks location data from users. If a hospital gives Google access to de-identified electronic health records (no names, addresses, etc), Google could still use the timestamp and location data it collects to re-identify the patient. This was the case in Dinerstein v. Google.

In Dinerstein, Google and the University of Chicago partnered on a machine learning project to identify patient health problems and prevent readmissions. Mr. Dinerstein, who had stayed in the University of Chicago’s hospital twice during the project, alleged that geolocation data collected by Google via his smartphone could have been used to identify him as a patient. 

Despite being rejected at the U.S. Federal Court of Appeal, Dinerstein provoked a broader discussion around privacy and health data. Ultimately, Dinerstein was dismissed because Mr. Dinerstein could not prove he suffered concrete harm at the hands of Google. But the loss of his confidentiality wasn’t enough to satisfy the court, since HIPAA itself doesn’t give patients the private right to sue. But imagine the following scenarios:

• Dinerstein applies for insurance after the fact, and a Google-affiliated insurance company uses the confidential data to reject him.
• Dinerstein goes to the hospital for an extremely rare surgery and his insurer uses a third party tool built on Google’s API. Large language models (LLMs) are known to memorize specific training sequences verbatim, and since the situation is so rare, the system picks up clinical features that effectively identify him.
• His profile is flagged for increased risk and he is offered a higher rate upon renewal. 

In the two examples shared above, unsupervised AI use could expose providers to extra legal risk and cause patients harm. 

Medical Documentation in a New Regulatory Landscape 

Breaches of medical privacy are serious. As AI models become more embedded in claims, medical summarization, or enterprise workflows, it becomes more important than ever to choose a compliant claims documentation platform solution and maintain human in the loop oversight. 

Human validation (for example, to catch and prevent functional reidentification of a patient’s rare medical data) offer added safeguards. Expert human reviewers can see who viewed a case, what the AI tool suggested, and why a decision was made, as well as validating the medical data being used to come to that decision is accurate in itself. US lawmakers are increasingly legislating on these topics, too: the Physicians Make Decisions Act was adopted in California, Massachusetts, Texas, and Washington and effectively makes human oversight the rule, not the excpetion. The Act restricts AI tools from making final decisions on medical care, and requires transparency about AI use, further solidifying the industry’s stance on the benefits of human expertise in claims.

The Colorado Artificial Intelligence Act requires that developers and deployers of high risk AI tools (such as those involving healthcare) be transparent and disclose AI use in adverse customer scenarios; a strong first step towards ensuring continued human oversight over AI tools. 

Healthcare data is unique in its complexity, sensitivity, and level of regulation. With sensitive patient data on the line, organizations in claims need to know how to harness and optimize their AI systems to balance time saved with safe use.