Chatbots beat doctors on paper. Then real patients used them and it fell apart.

Run time: 6:50

In today's episode:

1. Chatbot answers rated more empathetic than doctors, preferred 79% of the time
2. GPT-4 alone beat physicians by ~15 points on diagnostic vignettes
3. Google's AMIE matched or beat primary care docs in text consultations
4. Microsoft's system solved 85% of hard NEJM cases vs doctors' 20%
5. Oxford RCT: patients using LLMs did no better than a web search
6. Half of consumer chatbot medical advice carried potential for harm
7. ~1 in 6 US adults now use an AI chatbot for health advice monthly

TL;DR:

• On clean written cases, AI genuinely out-diagnoses doctors — that part is peer-reviewed, not hype.
• On real patients doing their own typing, that edge vanishes: an Oxford RCT (~1,300 people) found LLM users did no better than web search, because patients omit the detail that matters and models mix right advice with wrong.
• What's actually deployed is the opposite of a doctor replacement: clinician tools like OpenEvidence (used by ~65% of US doctors) keep a licensed human between the model and the prescription. No LLM is FDA-cleared to diagnose you solo.

Sources cited:

• JAMA Internal Medicine, Ayers et al.
• JAMA Network Open, Goh et al.
• Nature, Google DeepMind
• Microsoft AI
• Oxford, Nuffield Dept of Primary Care (Payne et al.)
• NBC News on OpenEvidence

Subscribe: YouTube

medAI Times is for educational and informational purposes only. The content does not constitute medical advice, diagnosis, treatment recommendation, or professional clinical guidance. Consult qualified healthcare professionals and refer to official sources before making clinical, research, regulatory, or business decisions.