Is your site blocking AI crawlers?
Enter a domain. We read its robots.txt for every known AI crawler, visit as each answer bot to catch CDN and firewall blocks, and run the checks that decide whether AI answers can cite you.
How it works
- 1
Enter your domain, or a competitor's.
- 2
We fetch robots.txt, then request your homepage as each AI crawler would.
- 3
See every crawler allowed or blocked, and the checks that failed, worst first.
Fix it, then see whether it worked
AI visibility MCP serverBeing crawlable is the start. With Manifold connected, your agent checks you and your competitors, writes the robots.txt lines to fix a block, crawls the site for pages bots cannot read, and then asks the AI engines whether they name you now.
250 free credits every month. No card.
| This page | Your agent with Manifold | |
|---|---|---|
| Sites | One | Yours and every competitor's |
| Pages | The homepage and one sample | A full crawl of the site |
| Fixes | The checks that fail | The robots.txt lines and fixes, written for you |
| Proof | None | Live answers from six AI engines, before and after |
“Check whether [domain] blocks any AI crawler and write the robots.txt lines to fix it.”
“Compare AI crawler access for us and our three closest competitors.”
“Crawl our site and list the pages AI crawlers cannot read without JavaScript.”
Questions about the AI crawler checker
Which AI crawlers does it check?
Every known AI crawler in robots.txt, from OpenAI, Anthropic, Google, Perplexity, Apple, Meta and others, split into the ones that fetch for answers and the ones that collect training data. It also requests your homepage as the answer crawlers to catch a CDN or firewall that turns them away.
Should I block AI training crawlers?
That is your choice, and blocking them does not keep you out of AI answers. The crawlers that matter for being cited are the search and user-fetch ones, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. The checker never marks a training block as a failure.
Why would my CDN block AI crawlers?
Since July 2025 Cloudflare blocks AI crawlers by default on new sites, and firewalls often turn away unfamiliar bots. That kind of block never shows in robots.txt, so the checker visits as each bot to find it.
Do I need an llms.txt file?
It is reported but never failed. No AI engine has said it reads llms.txt, and Google says it does not use it, so it is not worth worrying about before the checks above. OAI-SearchBot explained
Is the AI crawler checker free?
Yes. You get 10 checks a day here, with no signup. For more, connect Manifold to Claude, ChatGPT or Cursor: the free plan gives your agent 250 credits every month, with no card.
More free tools
All free toolsAI visibility checker
Ask ChatGPT, Gemini and Google's AI Overview a buyer's question and see if they name your brand.
Try it freeAI Overview checker
See whether Google answers a keyword with an AI Overview, what it says and whose pages it cites.
Try it freeSubreddit finder
Type a topic and find the subreddits where it is discussed, with an example thread from each.
Try it free