Cloudflare AI crawler block
Find the managed block hiding in robots.txt.
Cloudflare can add a managed section to robots.txt. We flag its markers and the AI disallow rules so you know where the decision is coming from.
01
Detect managed markers
We look for the BEGIN and END Cloudflare Managed content markers instead of treating the file as a generic text blob.
02
Separate policy from symptoms
The result matrix shows whether citation bots and training bots are affected, not just whether one line says Disallow.
03
Choose the right fix
Review AI Crawl Control, your WAF rules, and the source robots file. Do not edit an injected section that will be overwritten.