『#464: Why LLM Unpredictability is a Liability in MedTech』のカバーアート

#464: Why LLM Unpredictability is a Liability in MedTech

#464: Why LLM Unpredictability is a Liability in MedTech

無料で聴く

ポッドキャストの詳細を見る
Artificial intelligence has officially entered the mainstream cultural zeitgeist, creating a wave of excitement—and a fair share of fatigue—across the medical device industry. In this episode, host Etienne Nichols sits down with Tyler Harmon, biomedical engineer and CEO of Iaso Automated Medical Systems, to cut through the marketing buzzwords. Together, they explore the technical realities behind the technology stack, shifting the conversation away from generic AI toward specific, actionable engineering frameworks.The discussion highlights a critical distinction between traditional machine learning models and consumer-oriented Large Language Models (LLMs). Harmon explains that while technologies like convolutional neural networks (CNNs) have successfully processed medical imaging for years, modern LLMs introduce an intentional element of randomness to mimic human conversation. This lack of predictability presents unique challenges for medical device developers who operate in a deterministic, safety-critical environment where reproducibility is paramount.Looking toward practical deployment, the episode addresses how companies can responsibly govern these tools both within their software architectures and their internal Quality Management Systems (QMS). From classifying external AI models as Software of Unknown Provenance (SOUP) under IEC 62304 to leveraging machine learning for early detection of Acute Respiratory Distress Syndrome (ARDS) in the ICU, this conversation serves as an essential guide for innovators looking to build the next generation of safe, compliant, and effective medical technologies.Key Timestamps00:05 – Introduction to the dual nature of AI in MedTech: embedded clinical algorithms versus internal process optimization.02:14 – Demystifying the math: Breaking down artificial intelligence into linear and non-linear algorithmic transformations.04:30 – The Turing Test, Markov chains, and why consumer LLMs are mathematically designed to be unpredictable.07:15 – Real-world success stories: How convolutional neural networks (CNNs) revolutionized emergency stroke triage.09:42 – Inside Iaso Automated Medical Systems: Using non-LLM machine learning to identify Acute Respiratory Distress Syndrome (ARDS) in critical care.12:10 – AI Governance in the QMS: Designing specialized Standard Operating Procedures (SOPs) and Machine Learning Management Systems (AIMS).15:35 – Evaluating recent FDA 510(k) clearances for LLM-adjacent technologies and managing third-party stacks as SOUP.Quotes"If we as innovators can't explain things to a more general audience, we generally don't understand them ourselves. And if you can't do that, it's probably not the best idea to be implementing it into your products." - Tyler Harmon"I am probably going to be the biggest advocate you'll ever talk to about 'doctors need enablement, not replacement.' We need to give them the tools, the force multipliers to tackle the challenges they're going to face this century." - Tyler HarmonTakeawaysClassify External AI as SOUP: Treat third-party language models and external tech stacks as Software of Unknown Provenance (SOUP) under IEC 62304 frameworks, implementing rigorous risk management boundaries to isolate the core medical device logic.Engineer Out Randomness: Recognize that consumer LLMs purposefully integrate randomness layers to maximize user engagement. For clinical safety, developers must utilize architectural harnesses or alternative machine learning methods (like CNNs or random forests) to force more deterministic outcomes.Establish an AI Management System: Expand your organizational compliance beyond standard Quality Management Systems (QMS) and Information Security Management Systems (ISMS). Implement specific AI standard operating procedures and work instructions to govern internal token usage and data handling.Prioritize Clinical Enablement Over Automation: Focus clinical software engineering on clearing workflow bottlenecks and flagging early-stage critical conditions (such as ARDS) to allow bedside clinicians to deploy their hands-on expertise faster.ReferencesBerlin Criteria: The formal, quantitative medical classification standard used by clinicians to diagnose and grade the severity of Acute Respiratory Distress Syndrome.IEC 62304: The international standard governing medical device software lifecycle processes, specifically detailing the management of Software of Unknown Provenance (SOUP).Connect with Etienne Nichols on LinkedIn to stay updated on the latest episodes and industry insights.MedTech 101 SectionUnderstanding Non-Linear Math and LLMsThink of a traditional medical device software algorithm like a standard thermometer tracking a fever. It follows a straight, predictable line: if the temperature input increases by one degree, the reading on the screen changes by exactly one degree. This is a linear system.Modern AI, like Large Language Models (LLMs), works more like a seasoned doctor ...
adbl_web_anon_alc_button_suppression_t1
まだレビューはありません