Clinical AI chatbot misuse is a documented ongoing harm source not a theoretical risk as evidenced by ECRI ranking it the number one health technology hazard for two consecutive years

ECRI, the most credible independent patient safety organization in the US, ranked misuse of AI chatbots as the #1 health technology hazard in both 2025 and 2026. This is not theoretical concern but documented harm tracking. Specific documented failures include: incorrect diagnoses, unnecessary testing recommendations, promotion of subpar medical supplies, and hallucinated body parts. In one probe, ECRI asked a chatbot whether placing an electrosurgical return electrode over a patient's shoulder blade was acceptable—the chatbot stated this was appropriate, advice that would leave the patient at risk of severe burns. The scale is significant: over 40 million people daily use ChatGPT for health information according to OpenAI. The core mechanism of harm is that these tools produce 'human-like and expert-sounding responses' which makes automation bias dangerous—clinicians and patients cannot distinguish confident-sounding correct advice from confident-sounding dangerous advice. Critically, LLM-based chatbots (ChatGPT, Claude, Copilot, Gemini, Grok) are not regulated as medical devices and not validated for healthcare purposes, yet are increasingly used by clinicians, patients, and hospital staff. ECRI's recommended mitigations—user education, verification with knowledgeable sources, AI governance committees, clinician training, and performance audits—are all voluntary institutional practices with no regulatory teeth. The two-year consecutive #1 ranking indicates this is not a transient concern but an active, persistent harm pattern.

Supporting Evidence

Source: Thompson/CNBC 2026

The $1.8B AI telehealth startup's FDA warnings and lawsuits over AI-generated patient photos and deepfaked images represent a specific instance of clinical AI chatbot misuse at consumer scale. This is not a theoretical safety concern but an active regulatory and legal failure in a billion-dollar AI health deployment.