How to let AI crawlers in and keep scrapers out
Every major AI vendor now splits its crawling into training, search, and user-initiated fetches. Here is how to say yes to the first kind of traffic you want and no to the rest.
The short answer
OpenAI, Anthropic, and Perplexity each run separate crawlers for separate jobs, so you can welcome the ones that send people to your site and refuse the ones that train models on it. Robots.txt only asks, though, and the documentation for two of the user-initiated fetchers says it may not apply to them. The firewall does the enforcing.
- A blanket rule blocks the bots you want along with the ones you do not, because each vendor splits crawling into training, search, and user-initiated fetching
- OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, so allow it if you want to be found there
- Google-Extended controls Gemini training and grounding, and Google states it does not affect inclusion in Google Search
- ChatGPT-User and Perplexity-User act for a person who asked a question, and their documentation says robots.txt may not apply
- Cloudflare and Vercel both ship switchable AI bot rules, and Cloudflare's docs set new defaults for new domains from 15 September 2026
- Google says llms.txt files can be ignored for Google Search
What people search for
The table below shows which bot names people look up. The figures come from Google Ads Keyword Planner, worldwide, as monthly searches, and Google-Extended came back as roughly 880 to 1,000.
| Search | Monthly searches | What the reader needs |
|---|---|---|
| gptbot | 1,600 | Whether to block OpenAI's training crawler |
| google-extended | 880 to 1,000 | What the token does and what it does not |
| perplexitybot | 480 | Whether it trains models, and how to allow it |
| how to rank in chatgpt | 390 | How to get into ChatGPT search answers |
| cloudflare block ai bots | 210 | Where the switch lives |
| robots.txt for ai | 170 | What to actually write |
The search that dwarfs them all is "llms.txt" at 33,100 a month, and we come back to it near the end because the evidence for it is thinner than the demand.
Three jobs a crawler can do
The table below show which bot does which job. Training bots collect pages to build or tune models. Search bots index pages so an assistant can cite them later. User-initiated fetchers load a page because someone just asked about it.
| Company | Training | Search | User-initiated |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None, per its docs | PerplexityBot | Perplexity-User |
| Google-Extended token | Googlebot | No entry in the page we read |
OpenAI also lists OAI-AdsBot, which checks the safety of pages submitted as ads on ChatGPT, and says its data is not used to train foundation models.
Each vendor describes its bots in its own documentation. OpenAI's bot page says GPTBot crawls content for training and that disallowing it means a site's content should not be used in training. Anthropic's help article describes ClaudeBot as training collection, Claude-User as retrieval for user queries, and Claude-SearchBot as analysis that improves search results. Perplexity's crawler page says PerplexityBot surfaces and links sites in search results and is not used to crawl content for AI foundation models.
robots.txt is a request
The robots exclusion standard, RFC 9309, says its rules "are not a form of access authorization" and that crawlers are requested to honour them. It also warns that listing paths in robots.txt exposes them publicly, so never put the path to a private admin area in the file to keep it hidden.
Two vendors go further and say the standard does not cover their user-initiated fetchers. OpenAI writes that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity writes that Perplexity-User generally ignores robots.txt rules, because a user requested the fetch. OpenAI adds that ChatGPT-User is not used for crawling the web in an automatic fashion, and Perplexity says Perplexity-User is not used to collect content for training AI foundation models.
Anthropic's article adds a trap for anyone who reaches straight for the firewall. It notes that blocking IP addresses may not reliably opt you out, because the block impedes the crawler's reading of your robots.txt. Use robots.txt for the bots that honour it, and keep IP-level rules for traffic you need to stop.
A robots.txt that welcomes search and refuses training
This file allows the three search crawlers and disallows the training crawlers from OpenAI and Anthropic, plus Google's Gemini token.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlRead the Google-Extended line before you keep it. Google's crawler documentation says the token manages whether crawled content may be used to train future Gemini models, and that the control covers both training and grounding, meaning content from the Search index supplied to the model at prompt time, in Gemini Apps and the Gemini API on Vertex AI. It also says Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and that it has no separate user agent string, since crawling runs under existing Google user agents. That last point means a user agent rule cannot single it out, so robots.txt is the lever to use.
Timing also varies. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust. RFC 9309 tells crawlers not to use a cached copy for more than 24 hours unless the file is unreachable. Change the file, then wait a day before you judge the result.
Decide what you want before you block
| Your goal | Allow | Block |
|---|---|---|
| Appear in ChatGPT, Claude, and Perplexity answers, keep training out | OAI-SearchBot, Claude-SearchBot, PerplexityBot | GPTBot, ClaudeBot, Google-Extended |
| Appear everywhere, including Gemini | All of the above, and leave Google-Extended alone | Nothing |
| Stay out of AI products altogether | Googlebot, for normal Search | Every bot in this guide, the Search Console control set to exclude, plus a firewall rule for the fetchers that may ignore robots.txt |
The third row has a cost. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says disabling Claude-SearchBot reduces indexing for search results, and Perplexity describes PerplexityBot as the crawler that surfaces sites in its results. If nobody in your market finds you through AI answers yet, blocking costs little. If they do, it costs you the referral. Our guide to LLM tracking shows how to check which situation you are in.
The Google side, in three levers
| Lever | What it controls | What it leaves alone |
|---|---|---|
| Googlebot in robots.txt | Whether Google can crawl and index you for Search | Nothing, since blocking it removes you from Search |
| Google-Extended in robots.txt | Gemini training and grounding | Search inclusion and ranking |
| Search generative AI control in Search Console | Inclusion in AI Overviews, AI Mode, and generative AI in Discover | Training, and it is not a ranking signal |
Google's help page for the control says it rolled out to all sites on 31 August 2026 and defaults to including your site. Choosing to exclude takes effect within days. Separately, Google's guide to generative AI features says a page needs to be indexed and eligible to show with a snippet for these features, with no extra technical requirements.
The firewall layer
A firewall rule does what robots.txt cannot, which is stop a request before your server answers it. We cover two platforms here.
Vercel
Vercel's bot management offers a Bot Protection ruleset that is available on all plans, inactive by default, and excludes verified bots. It does not work behind a reverse proxy such as Cloudflare. A separate AI bots managed ruleset covers training, search, and user-fetch bots with log and deny modes, and it is also off by default. The list is maintained by Vercel. To block only training bots, write a custom WAF rule that matches their user agents, and run it in log mode first so you can read the traffic.
Cloudflare
Cloudflare's AI Crawl Control is available on all plans. It sets allow or block rules per crawler and tracks which crawlers follow your robots.txt. The AI bot policy page sorts crawlers into three categories: Search, Agent, and Training. Each takes one of three settings: block on all pages, block on pages with ads, or allow. You reach it through Security Settings and Configure AI bot policies.
New sites get different defaults. Cloudflare's documentation says that from 15 September 2026, new domains default to a setting where bots classified as Training or Agent are blocked on pages that display ads while Search stays allowed, and that mixed-purpose crawlers used for both Training and Search are also blocked. It lists the legacy Block AI bots option as deprecating on 15 September 2026. That page still phrases the change in future tense, so we cannot tell from the page whether it has reached your zone. Open your dashboard and read what is set.
Allow rules for the bots you want
Perplexity recommends whitelisting PerplexityBot and Perplexity-User in your WAF by combining a user agent match with its published IP ranges. Anthropic publishes a list to identify its crawlers too. Pair the name with the address wherever a vendor gives you both.
Read the traffic before you deny anything
Both platforms let you watch before you act. Vercel's rulesets have a log mode, and Cloudflare's AI Crawl Control shows which AI services access your content and which crawlers follow your robots.txt. Give it a week. Write down which bots turn up, which of them hit paths you disallowed, and whether any user-initiated fetchers appear at all.
A bot that hits a disallowed path despite the rule belongs in a firewall rule. A bot that obeys needs nothing more from you. A solo founder with a five-page site can finish that review in one sitting, and a publisher with a large archive should sort the log by path first.
The llms.txt reality check
"llms.txt" draws 33,100 searches a month, so the question deserves a straight answer. Google's guide to generative AI features lists llms.txt files among the things you can ignore for Google Search. The OpenAI, Perplexity, and Google crawler pages we read say nothing about the file affecting crawling or citation. Several documentation sites publish one of their own, including OpenAI, Perplexity, Cloudflare, Lovable, and Next.js, so the format is in use. We found no vendor documentation saying it changes whether you appear in an answer.
Treat it as optional. If you have the time after the robots.txt and firewall work, add one. Do not expect traffic from it.
How to rank in ChatGPT, as far as the documents say
OpenAI's documentation gives one concrete lever, which is to keep OAI-SearchBot allowed. It does not publish a ranking formula. Nothing we read guarantees a citation. What you can do is remove your own blockers: allow the search bot, keep the page public, and publish content that answers a specific question better than the page next to it. Our guide on getting mentioned by LLMs covers the content side, and why a brand goes unmentioned in ChatGPT covers the diagnosis.
Prompts to paste into Claude or v0
Write app/robots.ts (or public/robots.txt if this is not Next.js) that allows
OAI-SearchBot, Claude-SearchBot and PerplexityBot, disallows GPTBot, ClaudeBot
and Google-Extended, allows everything else on public pages, and lists my sitemap.Check whether any middleware, firewall config or hosting rule in this project
blocks requests by user agent. List each rule and the bots it affects.Make sure no page under /admin or /api is linked from public pages, and that
neither path is listed in robots.txt as a hiding place.Then test. A user agent check with curl shows whether a rule that matches on the name alone is firing.
curl -I -A "GPTBot" https://yourdomain.com/Rules that combine a name with an IP range will not fire for a request from your own machine, so read your firewall logs for the real traffic.
Where your content lives
Crawler access only pays off when there is something worth reading. If your posts live in a CMS that publishes straight to your site, all of the above applies to those URLs as well. Our guide to an agent-friendly website covers the on-page side, and Contengi's integrations show how posts reach a Claude or v0 site through a signed webhook.
The checklist
| Check | How | Pass looks like |
|---|---|---|
| Googlebot allowed | Open /robots.txt | No Disallow rule covering public pages |
| Search bots allowed | Same file | OAI-SearchBot, Claude-SearchBot, PerplexityBot not blocked |
| Training decision made | Same file | GPTBot, ClaudeBot, Google-Extended match your choice |
| No secrets listed | Read the file as a stranger would | No private paths named |
| Firewall on purpose | Vercel or Cloudflare settings | Rulesets set deliberately, log mode first on Vercel |
| Cloudflare defaults | Security Settings, Configure AI bot policies | Categories match the choice you made |
| Search Console control | Settings, Search generative AI | Include, unless you decided otherwise |
| Wait a day | Recheck after 24 hours | Vendors have had time to adjust |
| Fetchers that ignore robots.txt | Firewall logs | You decided whether to allow or block them |
Frequently asked questions
Should I block GPTBot?
GPTBot is OpenAI's training crawler, and OpenAI says disallowing it means a site's content should not be used in training generative AI foundation models. OAI-SearchBot, which surfaces sites in ChatGPT search, is a separate bot with its own robots.txt entry, so you can disallow one and allow the other.
Will blocking Google-Extended hurt my Google ranking?
Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It does control whether crawled content is used for Gemini training and for grounding in Gemini Apps and Vertex AI, so blocking it opts you out of both.
Does robots.txt stop ChatGPT-User and Perplexity-User?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiates the action, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. A firewall rule is the only way to stop them.
How do I get my site into ChatGPT search answers?
OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so keep it allowed in robots.txt and allow roughly 24 hours for changes to take effect. The documentation we read publishes no ranking formula and promises no citations.
Do I need an llms.txt file?
Google lists llms.txt files among the things you can ignore for Google Search, and the OpenAI, Perplexity, and Google crawler pages we read do not say the file affects crawling or citation. Add one if you want to, but do not expect it to change your traffic.
Does Cloudflare block AI bots by default?
Cloudflare's documentation says that from 15 September 2026, new domains default to blocking Training and Agent bots on pages that display ads, with Search allowed. The page still uses future tense, so check the AI bot policies in your own dashboard to see what your domain has.
Is Vercel's AI bots ruleset switched on by default?
No. Vercel's AI bots managed ruleset is off by default, with log and deny modes, and the Bot Protection ruleset is inactive by default too. Start in log mode and read the traffic before you deny anything.