What if your A/B test needed 67 years to reach statistical significance? Sam found out the hard way. Join Sam and Shifra as they demystify statistical testing for the real world of data work, where the stakes are lower, the data is messier, and your stakeholders definitely do not know what a p-value is.

We talk about:

• P-values, null hypotheses, and why 0.05 was basically made up
• Type 1 and type 2 errors through the lens of job interviews
• When A/B testing actually makes sense (hint: you need more than 10 visitors a day)
• T-tests, chi-square, ANOVA, and F1 scores explained without the jargon
• Why a suspiciously high model accuracy is actually a red flag
• The difference between statistical significance and practical significance

Chapters:

0:00 - The 67-year A/B test

0:22 - Welcome to everyone's favorite hobby

1:37 - Knowing how to interpret tests (not run them)

2:27 - Is the analysis actually important to the business?

3:37 - P-values refresher: what they are and aren't telling you

6:07 - Why a raw p-value isn't enough

7:40 - Null vs. alternative hypotheses explained

10:16 - Type one and type two errors (a.k.a. the costly mix-ups)

15:06 - Lift: measuring if your marketing actually did anything

18:53 - When you already have all the data, statistics isn't the tool

20:57 - Sample size, statistical significance, and the 67-year problem revisited

24:04 - Common A/B test types: t-tests, chi-square, and ANOVAs

26:44 - F1 scores, confusion matrices, and picking the right metric

29:19 - Central limit theorem and the magic number 30

31:31 - We never prove things — we just reject the null

34:51 - Premortems and deciding if a project is even worth doing

35:52 - When n is too small vs. too big (and why both are a problem)

38:00 - Effect size: the stat that doesn't care how big your sample is

41:39 - Regression, slope, and explaining it to real humans

47:07 - Spend your time on the right things, not the fanciest model

52:33 - Wrap-up and big takeaways