# How to let AI crawlers in and keep scrapers out > Which AI crawlers to allow and which to block: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, with a robots.txt to copy. Source: https://contengi.com/blog/robots-txt-for-ai Author: Stefan Maritz Published: 2026-09-30 Every major AI vendor now splits its crawling into training, search, and user-initiated fetches. Here is how to say yes to the first kind of traffic you want and no to the rest. ## The short answer OpenAI, Anthropic, and Perplexity each run separate crawlers for separate jobs, so you can welcome the ones that send people to your site and refuse the ones that train models on it. Robots.txt only asks, though, and the documentation for two of the user-initiated fetchers says it may not apply to them. The firewall does the enforcing. - A blanket rule blocks the bots you want along with the ones you do not, because each vendor splits crawling into training, search, and user-initiated fetching - OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, so allow it if you want to be found there - Google-Extended controls Gemini training and grounding, and Google states it does not affect inclusion in Google Search - ChatGPT-User and Perplexity-User act for a person who asked a question, and their documentation says robots.txt may not apply - Cloudflare and Vercel both ship switchable AI bot rules, and Cloudflare's docs set new defaults for new domains from 15 September 2026 - Google says llms.txt files can be ignored for Google Search ## What people search for The table below shows which bot names people look up. The figures come from Google Ads Keyword Planner, worldwide, as monthly searches, and Google-Extended came back as roughly 880 to 1,000. Search Monthly searches What the reader needs gptbot 1,600 Whether to block OpenAI's training crawler google-extended 880 to 1,000 What the token does and what it does not perplexitybot 480 Whether it trains models, and how to allow it how to rank in chatgpt 390 How to get into ChatGPT search answers cloudflare block ai bots 210 Where the switch lives robots.txt for ai 170 What to actually write The search that dwarfs them all is "llms.txt" at 33,100 a month, and we come back to it near the end because the evidence for it is thinner than the demand. ## Three jobs a crawler can do The table below show which bot does which job. Training bots collect pages to build or tune models. Search bots index pages so an assistant can cite them later. User-initiated fetchers load a page because someone just asked about it. Company Training Search User-initiated OpenAI GPTBot OAI-SearchBot ChatGPT-User Anthropic ClaudeBot Claude-SearchBot Claude-User Perplexity None, per its docs PerplexityBot Perplexity-User Google Google-Extended token Googlebot No entry in the page we read OpenAI also lists OAI-AdsBot, which checks the safety of pages submitted as ads on ChatGPT, and says its data is not used to train foundation models. Each vendor describes its bots in its own documentation. [OpenAI's bot page](https://developers.openai.com/api/docs/bots) says GPTBot crawls content for training and that disallowing it means a site's content should not be used in training. [Anthropic's help article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) describes ClaudeBot as training collection, Claude-User as retrieval for user queries, and Claude-SearchBot as analysis that improves search results. [Perplexity's crawler page](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) says PerplexityBot surfaces and links sites in search results and is not used to crawl content for AI foundation models. ## robots.txt is a request The robots exclusion standard, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), says its rules "are not a form of access authorization" and that crawlers are requested to honour them. It also warns that listing paths in robots.txt exposes them publicly, so never put the path to a private admin area in the file to keep it hidden. Two vendors go further and say the standard does not cover their user-initiated fetchers. OpenAI writes that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity writes that Perplexity-User generally ignores robots.txt rules, because a user requested the fetch. OpenAI adds that ChatGPT-User is not used for crawling the web in an automatic fashion, and Perplexity says Perplexity-User is not used to collect content for training AI foundation models. Anthropic's article adds a trap for anyone who reaches straight for the firewall. It notes that blocking IP addresses may not reliably opt you out, because the block impedes the crawler's reading of your robots.txt. Use robots.txt for the bots that honour it, and keep IP-level rules for traffic you need to stop. ## A robots.txt that welcomes search and refuses training This file allows the three search crawlers and disallows the training crawlers from OpenAI and Anthropic, plus Google's Gemini token. ``` User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / Sitemap: https://yourdomain.com/sitemap.xml ``` Read the Google-Extended line before you keep it. [Google's crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) says the token manages whether crawled content may be used to train future Gemini models, and that the control covers both training and grounding, meaning content from the Search index supplied to the model at prompt time, in Gemini Apps and the Gemini API on Vertex AI. It also says Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and that it has no separate user agent string, since crawling runs under existing Google user agents. That last point means a user agent rule cannot single it out, so robots.txt is the lever to use. Timing also varies. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust. RFC 9309 tells crawlers not to use a cached copy for more than 24 hours unless the file is unreachable. Change the file, then wait a day before you judge the result. ## Decide what you want before you block Your goal Allow Block Appear in ChatGPT, Claude, and Perplexity answers, keep training out OAI-SearchBot, Claude-SearchBot, PerplexityBot GPTBot, ClaudeBot, Google-Extended Appear everywhere, including Gemini All of the above, and leave Google-Extended alone Nothing Stay out of AI products altogether Googlebot, for normal Search Every bot in this guide, the Search Console control set to exclude, plus a firewall rule for the fetchers that may ignore robots.txt The third row has a cost. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says disabling Claude-SearchBot reduces indexing for search results, and Perplexity describes PerplexityBot as the crawler that surfaces sites in its results. If nobody in your market finds you through AI answers yet, blocking costs little. If they do, it costs you the referral. Our guide to [LLM tracking](https://contengi.com/blog/llm-tracking) shows how to check which situation you are in. ## The Google side, in three levers Lever What it controls What it leaves alone Googlebot in robots.txt Whether Google can crawl and index you for Search Nothing, since blocking it removes you from Search Google-Extended in robots.txt Gemini training and grounding Search inclusion and ranking Search generative AI control in Search Console Inclusion in AI Overviews, AI Mode, and generative AI in Discover Training, and it is not a ranking signal [Google's help page for the control](https://support.google.com/webmasters/answer/16908024) says it rolled out to all sites on 31 August 2026 and defaults to including your site. Choosing to exclude takes effect within days. Separately, [Google's guide to generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says a page needs to be indexed and eligible to show with a snippet for these features, with no extra technical requirements. ## The firewall layer A firewall rule does what robots.txt cannot, which is stop a request before your server answers it. We cover two platforms here. ### Vercel [Vercel's bot management](https://vercel.com/docs/bot-management) offers a Bot Protection ruleset that is available on all plans, inactive by default, and excludes verified bots. It does not work behind a reverse proxy such as Cloudflare. A separate AI bots managed ruleset covers training, search, and user-fetch bots with log and deny modes, and it is also off by default. The list is maintained by Vercel. To block only training bots, write a custom WAF rule that matches their user agents, and run it in log mode first so you can read the traffic. ### Cloudflare Cloudflare's [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) is available on all plans. It sets allow or block rules per crawler and tracks which crawlers follow your robots.txt. The [AI bot policy page](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) sorts crawlers into three categories: Search, Agent, and Training. Each takes one of three settings: block on all pages, block on pages with ads, or allow. You reach it through Security Settings and Configure AI bot policies. New sites get different defaults. Cloudflare's documentation says that from 15 September 2026, new domains default to a setting where bots classified as Training or Agent are blocked on pages that display ads while Search stays allowed, and that mixed-purpose crawlers used for both Training and Search are also blocked. It lists the legacy Block AI bots option as deprecating on 15 September 2026. That page still phrases the change in future tense, so we cannot tell from the page whether it has reached your zone. Open your dashboard and read what is set. ### Allow rules for the bots you want Perplexity recommends whitelisting PerplexityBot and Perplexity-User in your WAF by combining a user agent match with its published IP ranges. Anthropic publishes a list to identify its crawlers too. Pair the name with the address wherever a vendor gives you both. ## Read the traffic before you deny anything Both platforms let you watch before you act. Vercel's rulesets have a log mode, and Cloudflare's AI Crawl Control shows which AI services access your content and which crawlers follow your robots.txt. Give it a week. Write down which bots turn up, which of them hit paths you disallowed, and whether any user-initiated fetchers appear at all. A bot that hits a disallowed path despite the rule belongs in a firewall rule. A bot that obeys needs nothing more from you. A solo founder with a five-page site can finish that review in one sitting, and a publisher with a large archive should sort the log by path first. ## The llms.txt reality check "llms.txt" draws 33,100 searches a month, so the question deserves a straight answer. Google's guide to generative AI features lists llms.txt files among the things you can ignore for Google Search. The OpenAI, Perplexity, and Google crawler pages we read say nothing about the file affecting crawling or citation. Several documentation sites publish one of their own, including OpenAI, Perplexity, Cloudflare, Lovable, and Next.js, so the format is in use. We found no vendor documentation saying it changes whether you appear in an answer. Treat it as optional. If you have the time after the robots.txt and firewall work, add one. Do not expect traffic from it. ## How to rank in ChatGPT, as far as the documents say OpenAI's documentation gives one concrete lever, which is to keep OAI-SearchBot allowed. It does not publish a ranking formula. Nothing we read guarantees a citation. What you can do is remove your own blockers: allow the search bot, keep the page public, and publish content that answers a specific question better than the page next to it. Our guide on [getting mentioned by LLMs](https://contengi.com/blog/answer-engine-optimisation-for-small-business-how-to-get-mentioned-by-llms) covers the content side, and [why a brand goes unmentioned in ChatGPT](https://contengi.com/blog/why-is-my-brand-never-mentioned-in-chatgpt-responses-when-people-ask-about-my-industry) covers the diagnosis. ## Prompts to paste into Claude or v0 ``` Write app/robots.ts (or public/robots.txt if this is not Next.js) that allows OAI-SearchBot, Claude-SearchBot and PerplexityBot, disallows GPTBot, ClaudeBot and Google-Extended, allows everything else on public pages, and lists my sitemap. ``` ``` Check whether any middleware, firewall config or hosting rule in this project blocks requests by user agent. List each rule and the bots it affects. ``` ``` Make sure no page under /admin or /api is linked from public pages, and that neither path is listed in robots.txt as a hiding place. ``` Then test. A user agent check with curl shows whether a rule that matches on the name alone is firing. ``` curl -I -A "GPTBot" https://yourdomain.com/ ``` Rules that combine a name with an IP range will not fire for a request from your own machine, so read your firewall logs for the real traffic. ## Where your content lives Crawler access only pays off when there is something worth reading. If your posts live in a CMS that publishes straight to your site, all of the above applies to those URLs as well. Our guide to an [agent-friendly website](https://contengi.com/blog/how-to-build-an-agent-friendly-website) covers the on-page side, and [Contengi's integrations](https://contengi.com/integrations) show how posts reach a Claude or v0 site through a signed webhook. ## The checklist Check How Pass looks like Googlebot allowed Open /robots.txt No Disallow rule covering public pages Search bots allowed Same file OAI-SearchBot, Claude-SearchBot, PerplexityBot not blocked Training decision made Same file GPTBot, ClaudeBot, Google-Extended match your choice No secrets listed Read the file as a stranger would No private paths named Firewall on purpose Vercel or Cloudflare settings Rulesets set deliberately, log mode first on Vercel Cloudflare defaults Security Settings, Configure AI bot policies Categories match the choice you made Search Console control Settings, Search generative AI Include, unless you decided otherwise Wait a day Recheck after 24 hours Vendors have had time to adjust Fetchers that ignore robots.txt Firewall logs You decided whether to allow or block them ## Frequently asked questions ### Should I block GPTBot? GPTBot is OpenAI's training crawler, and OpenAI says disallowing it means a site's content should not be used in training generative AI foundation models. OAI-SearchBot, which surfaces sites in ChatGPT search, is a separate bot with its own robots.txt entry, so you can disallow one and allow the other. ### Will blocking Google-Extended hurt my Google ranking? Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It does control whether crawled content is used for Gemini training and for grounding in Gemini Apps and Vertex AI, so blocking it opts you out of both. ### Does robots.txt stop ChatGPT-User and Perplexity-User? Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiates the action, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. A firewall rule is the only way to stop them. ### How do I get my site into ChatGPT search answers? OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so keep it allowed in robots.txt and allow roughly 24 hours for changes to take effect. The documentation we read publishes no ranking formula and promises no citations. ### Do I need an llms.txt file? Google lists llms.txt files among the things you can ignore for Google Search, and the OpenAI, Perplexity, and Google crawler pages we read do not say the file affects crawling or citation. Add one if you want to, but do not expect it to change your traffic. ### Does Cloudflare block AI bots by default? Cloudflare's documentation says that from 15 September 2026, new domains default to blocking Training and Agent bots on pages that display ads, with Search allowed. The page still uses future tense, so check the AI bot policies in your own dashboard to see what your domain has. ### Is Vercel's AI bots ruleset switched on by default? No. Vercel's AI bots managed ruleset is off by default, with log and deny modes, and the Bot Protection ruleset is inactive by default too. Start in log mode and read the traffic before you deny anything.