What does your website tell AI crawlers?
Since September 15, 2026, Cloudflare sorts AI crawlers into search, agents and training, and blocks some by default. Enter your website to see what it says to each group today.
Common questions
What does this check read?
Your site's public robots.txt file, the headers on your home page, and whether an llms.txt file exists. It looks for rules naming known search, AI agent and AI training crawlers, plus Content Signals, the IETF Content-Usage line and RSL licence links.
Does 'Not stated' mean AI companies are using my content?
It means nothing on your site asks them not to. Well-behaved crawlers treat that as permission. Whether any particular company has crawled your site is a separate question your hosting or Cloudflare logs can answer.
Will blocking AI training hurt my Google ranking?
No. Google's training crawler, Google-Extended, is separate from Googlebot, and Google has said blocking it does not affect search. Cloudflare's new Disallow AI Training setting works the same way.
Should I block AI agents?
Usually not. Agents such as ChatGPT-User and Claude-User fetch a page because a person asked an assistant about your business. Blocking them means the assistant cannot answer questions about you.
Is this enforcement?
No. robots.txt and Content Signals are requests that responsible crawlers honour. Enforcement happens at your host or a service such as Cloudflare, which since September 15, 2026 can block or charge AI crawlers by category.
Related: what changed on September 15, the free chatbot disclosure scan and AI disclosure laws by state.