# Is your site blocking AI crawlers?

> Enter a domain. We read its robots.txt for every known AI crawler, visit as each answer bot to catch CDN and firewall blocks, and run the checks that decide whether AI answers can cite you.

This is a free tool on the web page at https://www.manifoldmcp.com/free-tools/ai-crawler-checker: 10 runs a day per visitor, no signup, run by a person in a browser. An agent can do the same job, and more, by connecting Manifold's MCP server at `https://mcp.manifoldmcp.com/mcp` (250 free credits a month). See [AI visibility MCP server](https://www.manifoldmcp.com/mcp/ai-visibility.md).

## How it works

1. Enter your domain, or a competitor's.
2. We fetch robots.txt, then request your homepage as each AI crawler would.
3. See every crawler allowed or blocked, and the checks that failed, worst first.

## Fix it, then see whether it worked

Being crawlable is the start. With Manifold connected, your agent checks you and your competitors, writes the robots.txt lines to fix a block, crawls the site for pages bots cannot read, and then asks the AI engines whether they name you now.

| | Here | In your agent with Manifold |
| --- | --- | --- |
| Sites | One | Yours and every competitor's |
| Pages | The homepage and one sample | A full crawl of the site |
| Fixes | The checks that fail | The robots.txt lines and fixes, written for you |
| Proof | None | Live answers from six AI engines, before and after |

### Things to ask your agent

- Check whether [domain] blocks any AI crawler and write the robots.txt lines to fix it.
- Compare AI crawler access for us and our three closest competitors.
- Crawl our site and list the pages AI crawlers cannot read without JavaScript.

## Questions

### Which AI crawlers does it check?

Every known AI crawler in robots.txt, from OpenAI, Anthropic, Google, Perplexity, Apple, Meta and others, split into the ones that fetch for answers and the ones that collect training data. It also requests your homepage as the answer crawlers to catch a CDN or firewall that turns them away.

### Should I block AI training crawlers?

That is your choice, and blocking them does not keep you out of AI answers. The crawlers that matter for being cited are the search and user-fetch ones, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. The checker never marks a training block as a failure.

### Why would my CDN block AI crawlers?

Since July 2025 Cloudflare blocks AI crawlers by default on new sites, and firewalls often turn away unfamiliar bots. That kind of block never shows in robots.txt, so the checker visits as each bot to find it.

### Do I need an llms.txt file?

It is reported but never failed. No AI engine has said it reads llms.txt, and Google says it does not use it, so it is not worth worrying about before the checks above. [OAI-SearchBot explained](https://www.manifoldmcp.com/blog/oai-searchbot.md).

### Is the AI crawler checker free?

Yes. You get 10 checks a day here, with no signup. For more, connect Manifold to Claude, ChatGPT or Cursor: the free plan gives your agent 250 credits every month, with no card.

## More free tools

- [AI visibility checker](https://www.manifoldmcp.com/free-tools/ai-visibility-checker.md): Ask ChatGPT, Gemini and Google's AI Overview a buyer's question and see if they name your brand.
- [AI Overview checker](https://www.manifoldmcp.com/free-tools/ai-overview-checker.md): See whether Google answers a keyword with an AI Overview, what it says and whose pages it cites.
- [Subreddit finder](https://www.manifoldmcp.com/free-tools/subreddit-finder.md): Type a topic and find the subreddits where it is discussed, with an example thread from each.
- [All free tools](https://www.manifoldmcp.com/free-tools.md)