Don’t block the bots. Build the gate

Media companies have gotten pretty good at blocking AI bots. They have been less good at deciding which ones should be allowed in. Reports say most major publications almost universally block crawlers from major AI companies like OpenAI and Anthropic, and many small to midsize publishers mirror that, configuring their robots exclusion protocol settings, or robots.txt, to keep them out. It’s an understandable stance, and it might even be appropriate. But it also crudely compresses a complex…