Summary

In this episode, I’m joined by Teresa Torres. Teresa is an author, speaker and a product discovery coach who I’ve been a fan of for quite a long time.

We chat about why assumptions look different depending on the level you’re working at, why showing your work is so important for alignment, and how using AI inside your courses can give students more opportunities to practice and receive feedback.

We also dig into AI evals - how Teresa uses them to measure the quality of her AI coaching tools, identify failure modes, and systematically improve their performance instead of just trusting that an LLM will get it right on the first try.

If you want to learn more about testing assumptions, designing better learning loops, and using AI without giving up your critical thinking skills, this episode is for you.

Takeaways

1. AI can create a safer space for learning. Teresa found that people are often willing to ask an AI questions they might feel embarrassed asking another person.

2. An AI coach works best when it is grounded in a clear teaching model. Teresa’s interview coach was effective because it was built on years of refined curriculum, rubrics, and explicit feedback criteria.

3. AI can dramatically increase opportunities for deliberate practice. Instead of waiting for an instructor, students can practice repeatedly and receive immediate, personalized feedback.

4. Building AI tools can improve the underlying curriculum. When an agent struggles with ambiguous instructions, it exposes gaps in how the material itself is taught.

5. Evals are simply a way to measure AI quality. Teresa uses evals to identify specific failure modes, track how often they occur, and test whether changes actually improve the agent.

6. Domain expertise still matters. Recognizing that an AI coach has given subtly bad advice often requires deep knowledge of the subject, not just technical skill.

7. Product teams should define what “good” looks like for AI. Generic quality metrics are not enough; teams need acceptance criteria and evals tied to the unique value their product is supposed to create.

8. AI changes the role of the teacher, not just the tools. Teresa is moving away from being the expert who simply delivers answers and toward creating environments where people can practice, get feedback, and learn for themselves.

Guest Links

ProductTalk Website: https://www.producttalk.org/

Teresa’s LinkedIn: https://www.linkedin.com/in/teresatorres/