The answer to “should I block AI bots” depends on which bots: block the training bots if your writing is the product you sell, and keep AI search bots unless you want out of AI answers. OpenAI, Anthropic, Google and Apple name their training bots separately from their search bots, so those are separate decisions.
Before you touch robots.txt, look at what AI crawlers give back. On tlinky.com, our own WordPress link-tools site, AI referral traffic came to 35 visits in about one week, against hundreds of AI bot requests.
Should I block AI bots? Split the answer three ways
Block AI training bots if your writing is your product, and keep AI search bots and user fetchers unless you want out of AI answers. An AI training crawler is a bot that collects pages to train AI models. An AI search crawler is a bot that indexes pages so an assistant can cite them.
The third group, user-triggered fetchers like ChatGPT-User and Perplexity-User, opens one page because a person asked about it.
That split matters because most AI crawling is for training. Cloudflare found that 80% of AI crawling over a 12-month period was for training, 18% for search and 2% for user actions.
So the honest version of the question is three questions:
- Training bots: do you mind your pages helping train future models? If you sell writing, courses or original research, you probably do.
- AI search bots: do you want to appear, with a link, in ChatGPT search, Perplexity or Claude’s answers? Those links are where AI visits come from.
- User-triggered fetchers: do you want a person who asks an assistant about your page to get it? Note that robots.txt may not stop them: OpenAI and Perplexity both say their fetchers may skip it.
Our guide to AI crawlers on WordPress turns those answers into a setting for each kind of site, from publishers to stores.
Pros and cons of blocking AI bots
Blocking training bots, which made 80% of AI crawling in Cloudflare’s data, protects your writing from future training by companies that honour robots.txt, and costs little in visits. Blocking AI search bots or user fetchers costs citations and visits from those assistants.
| Pros | Cons | |
|---|---|---|
| Block training bots | OpenAI, Anthropic, Google and Apple each describe a training opt-out that’s separate from search | Fake crawlers and bots that ignore robots.txt aren’t stopped |
| Block AI search bots | Fewer requests to your server | Your pages drop out of that assistant’s answers and links |
| Block user-triggered fetchers | People can’t pull your page into an assistant | robots.txt may not work, so it takes a server or CDN rule |
What AI bots give back: crawls versus AI referral traffic
AI referral traffic is the visits people make after an AI assistant links to you, and it’s still small next to the crawling. On tlinky.com, AI assistants sent 35 visits in the week up to 28 October 2026.
Here’s how those visits compare with the crawlers shown on page one of the plugin’s crawler table that week:
| Company | Bot requests (about one week) | Visits sent | Requests per visit |
|---|---|---|---|
| OpenAI | 209, IP-verified (OAI-SearchBot 164, ChatGPT-User 40, GPTBot 5) | 29 from ChatGPT | about 7 |
| Anthropic | 656 by name, not IP-verified (Claude-User 404, ClaudeBot 252) | 5 from Claude | about 131 |
IP-verified means the request came from an address the company publishes. Our plugin doesn’t yet check Anthropic’s published list, so the Anthropic count may include fakes and isn’t directly comparable with OpenAI’s.
Gemini sent one visit; Google’s crawling isn’t in the table, because Googlebot also crawls for Search. The visits counted are unique visitors per page per day, and they landed on 14 pages, led by a YouTube Shorts downloader tool with 15.
We read these numbers in our Skyboot AI Markdown plugin’s AI Analytics, which shows verified crawler hits, fake crawlers and which assistants sent visitors to which pages (our documentation). One week on one site is a small sample.
The crawl-to-referral ratio is the number of crawls an AI company makes for each visit it sends back. Cloudflare measured it in July 2025 at about 5 for Google, 195 for Perplexity, 1,091 for OpenAI and 38,066 for Anthropic. As in Cloudflare’s data, Anthropic’s requests per visit on tlinky.com are far higher than OpenAI’s, though our numbers are much smaller.
The visits aren’t nothing. But on tlinky.com, AI assistants sent 35 visits in about one week, so don’t plan on them replacing search visits yet.
Does blocking AI crawlers hurt SEO?
Blocking Google-Extended doesn’t affect your Google Search inclusion or ranking, by Google’s own statement. Blocking Googlebot or an AI search bot does cost visibility, in Google or in that company’s AI answers. Each company describes its own split:
| If you block | What the company says happens |
|---|---|
| Google-Extended (Gemini training, and grounding, which means supporting Gemini’s answers with your pages) | Google: no effect on inclusion in Google Search, and not a ranking signal |
| GPTBot (OpenAI training) | OpenAI treats it separately from OAI-SearchBot, so ChatGPT search isn’t affected |
| OAI-SearchBot (ChatGPT search) | Your pages drop out of ChatGPT search answers, except as plain navigational links |
| ClaudeBot (Anthropic training) | Anthropic says the block signals that your future pages should be excluded from training; Claude-SearchBot and Claude-User are separate |
| Applebot-Extended (Apple training) | Apple says it doesn’t affect Apple’s search features |
Google-Extended isn’t a crawler at all; it’s a rule Google reads. Blocking it leaves Googlebot crawling as usual.
AI Overviews are the catch. Google’s rule for AI Overviews and AI Mode is that a cited page must be indexed and eligible for a snippet. Those features draw on Google Search’s index, so no AI-bot block keeps you out of them; only snippet controls or noindex do, and those affect normal search results too.
What a block can’t stop
A robots.txt block only binds bots that choose to obey it. The robots.txt standard, RFC 9309, treats its rules as a request that grants or refuses nobody access.
Three things stay out of its reach:
- Fake crawlers. A fake crawler is a bot that copies a well-known bot’s name to look legitimate. On tlinky.com, our plugin left 291 such requests out of its counts in about one week; it doesn’t block them, and a robots.txt rule can’t make them stop.
- User-triggered fetchers. Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt may not govern ChatGPT-User.
- Content already collected. Anthropic, for one, describes a ClaudeBot block as covering future content. It doesn’t mention data gathered before.
If you need a block that’s enforced rather than requested, it has to sit higher up. Search Engine Journal’s Ask an SEO column recommends blocking as high in the stack as you can: a firewall first, then a CDN, then server rules. Our Cloudflare AI bot guide covers the CDN route, and our PerplexityBot guide shows a tested nginx rule for a user fetcher.
Is a robots.txt block legally binding?
A robots.txt block isn’t legally binding by itself, and this isn’t legal advice. The standard treats its rules as a request. In the EU, a text and data mining opt-out is a rights holder’s reservation against automated analysis of their work. Copyright law allows one under Article 4(3) of the 2019 copyright directive.
Whether robots.txt counts as that opt-out is still being argued. A lawyer at the firm Garrigues notes that a Hamburg court tied a “machine-readable” reservation to something machines can act on automatically, and that the case is before Germany’s Federal Court of Justice. If the legal side matters to you, ask a lawyer where you operate.
Make the call, then measure it
Make the call in writing, one decision for each group: training bots, AI search bots and user-triggered fetchers. Note your AI visits and bot counts first, so you have a baseline. The robots.txt rules for each AI bot group cover training-only blocks in Yoast, Rank Math, AIOSEO or a physical file.
To do it from a settings screen instead, our plugin’s training-crawler setting blocks its listed training bots and never search or user bots (our documentation). It works only when WordPress generates robots.txt; if a physical robots.txt file exists, the plugin warns you and you edit that file instead.
A month later, check the same two numbers: verified bot hits and AI visits. Both live in your logs or analytics; finding AI crawler hits on WordPress walks through each place. If the visits matter to you, making your content easier for AI to cite is the other half of the job.
Frequently Asked Questions
Will blocking AI bots stop AI Overviews from using my content?
No. Google’s AI Overviews and AI Mode use pages from Google Search’s index, which Googlebot crawls. Google doesn’t name Google-Extended as a control for them. Google’s own controls for AI Overviews are snippet rules such as nosnippet and max-snippet, or noindex, and each of those also changes how the page shows in normal search.
Can I unblock AI bots later?
Yes. Remove the rule from robots.txt and the bots that read it will see the change; OpenAI says that takes about 24 hours for search. Unblocking doesn’t recall anything, and a block doesn’t remove anything already collected: Anthropic describes its block as covering future content only.
Do small sites need to worry about AI bots at all?
Yes, but the stakes are smaller than for big publishers. On tlinky.com, AI assistants sent 35 visits in about one week, and our plugin filtered 291 fake crawler requests in the same week. A decision per bot group and a monthly look at those two numbers is usually enough for a small site.
How we tested
- Site data: tlinky.com in the week up to 28 October 2026, read in Skyboot AI Markdown’s AI Analytics (version 3.0.0): bot requests with and without an IP match, fake crawlers and AI visits. Requests per visit are our arithmetic.
- Each company’s statement about search comes from its own crawler documentation, linked in the article.