TL;DR
An Ontario audit of 20 AI note-taking systems used in healthcare found that nearly half fabricated information, and many missed critical patient details. The findings raise concerns about AI accuracy in medical documentation.
The Ontario Office of the Auditor General has reported that 9 out of 20 AI note-taking systems approved for healthcare use routinely produce inaccurate or fabricated information, raising concerns about their safety and reliability.
The audit evaluated 20 AI systems used by healthcare professionals across Ontario, finding that nearly half of these systems inserted fabricated details into patient notes. Specifically, nine systems fabricated information, including false treatment suggestions, while twelve systems inserted incorrect medication data. Additionally, 17 of the systems missed key mental health details discussed during patient encounters, with six missing such information entirely or partially.
The evaluation process involved simulated doctor-patient recordings, which were reviewed alongside AI-generated notes. The report highlights that the scoring criteria for these systems heavily weighted factors such as domestic presence (30%) and security compliance (4%), while only 4% of the score was based on accuracy. This imbalance may have contributed to the selection of AI tools with poor performance in critical areas.
Why It Matters
The findings matter because they highlight potential risks to patient safety and data integrity when using AI for medical documentation. Inaccurate or fabricated notes could lead to misdiagnoses, incorrect treatments, and breaches of patient confidentiality. The report raises questions about the adequacy of current evaluation standards for AI tools in healthcare and the potential for harm if such errors go unnoticed.
AI note-taking system for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background
The use of AI note-taking systems in Ontario’s healthcare sector has expanded under the province’s AI Scribe program, which involves over 5,000 physicians. Previous studies have shown that AI models can produce unreliable medical information, but the Ontario audit provides concrete evidence of systemic issues. The evaluation methods used by the province have been criticized for their emphasis on non-clinical criteria, possibly allowing poorly performing systems to be approved.
“Inaccurate weightings could result in the selection of vendors whose AI tools may produce inaccurate or biased medical records or lack adequate protection to safeguard sensitive personal health information.”
— Office of the Auditor General of Ontario
“More than 5,000 physicians are participating in the AI Scribe program, and there have been no known reports of patient harms associated with the technology.”
— Ontario Ministry of Health spokesperson
As an affiliate, we earn on qualifying purchases.
What Remains Unclear
It is unclear whether the Ontario government will revise its evaluation criteria or impose mandatory accuracy attestations for AI systems. The extent of potential patient harm caused by current inaccuracies remains unconfirmed, and ongoing monitoring is needed to assess real-world impact.
As an affiliate, we earn on qualifying purchases.
What’s Next
The Ontario Ministry of Health is expected to review the audit findings and may implement stricter evaluation standards or oversight measures for AI tools. Further investigations into the clinical impact of these inaccuracies are likely, along with potential updates to procurement policies.
As an affiliate, we earn on qualifying purchases.
Key Questions
Are AI note-taking systems safe to use in Ontario healthcare?
Based on the audit, many systems have produced errors, raising concerns about safety. The government has not reported any patient harms so far, but the risks warrant caution and further review.
What specific errors were found in the AI systems?
Errors include fabricated information, incorrect medication details, and missed mental health notes. Some systems suggested treatments or details not discussed in the actual patient recordings.
Will the Ontario government change its evaluation process?
The audit criticizes the current scoring system’s emphasis on criteria other than accuracy, suggesting reforms are likely to improve AI vetting standards.
Could these errors lead to patient harm?
While no harm has been reported yet, inaccuracies in medical notes could potentially result in misdiagnoses or inappropriate treatments, posing risks to patient safety.
