Here’s a quick wrap of the three papers we found interesting over the last few weeks with some take home points.

• 1:00 - Clinical knowledge in LLMs does not translate to human interactions
• 06:45 - From Tool to Teammate: A Randomized Controlled Trial of Clinician-AI Collaborative Workflows for Diagnosis
• 11:55 - Advancing Real-time Pandemic Forecasting Using Large Language Models: A COVID-19 Case Study

More details in the show notes on our website.

Episodes | Bluesky | info@medicalattention.ai

In this episode, we’re lucky to be joined by Alexandre Sallinen and Tony O’Halloran from the Laboratory for Intelligent Global Health & Humanitarian Response Technologies to discuss how large language models are assessed, including their Massive Open Online Validation & Evaluation (MOOVE) initiative.

0:25 - Technical wrap: what are agents?

13:20 - What are benchmarks?

• 18:20 - Automated evaluation

• 20:10 - Benchmarks

• 37:45 - Human feedback

• 44:50 - LLM as judge

Read more about the projects we discuss here:

• Meditron

• Learn about the MOOVE or contact our team if you'd like to be involved

• Listen to the LiGHTCAST including their recent excellent outline of the HealthBench paper

More details in the show notes on our website.

Episodes | Bluesky | info@medicalattention.ai

Here’s a quick wrap of the three papers we found interesting over the last few weeks with some take home points.

• 0:35 - Superhuman performance of a large language model on the reasoning tasks of a physician
• 06:20 - MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
• 11:45 - Identifying and mitigating algorithmic bias in the safety net

More details in the show notes on our website.

Episodes | Bluesky | info@medicalattention.ai