A request hits your server claiming to be Googlebot. Gary Illyes says making one look like that takes one line of code and about 30 seconds, and that impersonation is "unfortunately, very" common. He knows because he runs honeypot sites and reads the logs.

Gary explains why AI companies' hunger for training tokens made crawler identity urgent, how an unknown crawler can exhaust a site's bandwidth, and why most bad crawling today is ignorance rather than attack. He is blunt about robots.txt: it gives granular control, but it is a suggestion, and only network-level blocking actually stops a bot.

The conversation then separates two kinds of automated visitor, the autonomous crawler and the user-triggered fetcher, and Gary makes his arguable case that the second may not need to obey robots.txt. He shows how Googlebot and Google-Extended already let publishers allow Search while blocking AI use, explains how Web Bot Auth uses HTTP signatures to prove a request really came from Google, and why IPv4 scarcity makes the old IP-based verification hard for new crawlers.

Finally, he describes the idea he's toying with next, bot reputation, why one company deciding would be biased, whether authentication makes the open web less open, and what a site owner without a security team should actually do today.

Listen if you work in technical SEO, AI search, publishing, web security or infrastructure, or run a site where crawler traffic is becoming a real cost.