--- title: "GPTBot, ClaudeBot, PerplexityBot: AI bots in robots.txt" description: Which AI crawlers to allow in robots.txt. GPTBot, ClaudeBot, PerplexityBot and others explained, with a paste-ready block that blocks training only. --- [Insights | ROAS Expert](https://roasexpert.com/blog) # [GPTBot, ClaudeBot, PerplexityBot: AI bots in robots.txt](https://roasexpert.com/blog/ai-crawlers-robots-txt) Written by [Rishabh Gupta](https://roasexpert.com/blog/author/rishabh-gupta) | October 20, 2026 By Rishabh Gupta, founder of ROAS Expert. Last updated 19 October 2026. If you want AI assistants to recommend your store or product, allow the search and user-triggered bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot) and decide separately whether to block the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Amazonbot, Bytespider). Each company documents these as independent switches, so blocking training does not have to cost you visibility. The paste-ready blocks are below. In short - AI bots come in three roles: training crawlers, search index crawlers and user-triggered fetchers. Each company documents them as independent switches - Allow the search and user-triggered bots if you want to be recommended. Block training only if your legal or content position says so - Cloudflare has blocked AI crawlers by default for new domains since July 2025. A 403 at the edge makes your robots.txt irrelevant - Paste-ready blocks for plain robots.txt and for Shopify's robots.txt.liquid are below **3**roles an AI bot can have, and three separate decisionstraining, search, user-triggered **24h**for a robots.txt change to apply at OpenAI, Meta and AmazonAmazon may cache for 30 days **1M+**Cloudflare customers had switched on the AI crawler block before it became the defaultCloudflare, July 2025 On this page 1. [Three kinds of AI bot, three different decisions](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#three-kinds-of-ai-bot-three-different-decisions) 2. [The bots, by company, from their official pages](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#the-bots-by-company-from-their-official-pages) 3. [Paste-ready robots.txt: allow discovery, block training](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#paste-ready-robots-txt-allow-discovery-block-training) 4. [Paste-ready robots.txt: allow everything](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#paste-ready-robots-txt-allow-everything) 5. [Cloudflare: you may be blocking these bots without knowing](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#cloudflare-you-may-be-blocking-these-bots-without-knowing) 6. [Shopify: robots.txt.liquid](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#shopify-robots-txt-liquid) 7. [How to check which AI bots are hitting your site](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#how-to-check-which-ai-bots-are-hitting-your-site) 8. [Where ROAS Expert fits](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#where-roas-expert-fits) 9. [Questions](https://roasexpert.com/blog/ai-crawlers-robots-txt?hs_amp=true#questions) ## Three kinds of AI bot, three different decisions Most robots.txt mistakes come from treating "AI bots" as one thing. The official docs split them into three roles. **Training crawlers** - Fetch pages in bulk to build datasets for future models - GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent - Blocking is a licensing decision. It changes nothing about whether you appear in answers today **Search index crawlers** - Build the index an assistant searches when a user asks something - OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, meta-webindexer - Block these and you remove yourself from the answers **User-triggered fetchers** - Fetch one page because a person asked a question right now - ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User - Several may not follow robots.txt at all. Blocking produces broken answers, not privacy For a store or SaaS company that wants to be found, the position is simple. Allow the search and user-triggered bots. Block training only if your legal or content position says so. The two are independent in every bot family listed below. ## The bots, by company, from their official pages | Bot token | Operator | Role | Our default | Note | | --- | --- | --- | --- | --- | | GPTBot | OpenAI | Training | Block if you want | No effect on ChatGPT search | | OAI-SearchBot | OpenAI | Search | Allow | Blocking removes you from ChatGPT search answers | | ChatGPT-User | OpenAI | User-triggered | Allow | May ignore robots.txt | | ClaudeBot | Anthropic | Training | Block if you want | Excludes future content from training | | Claude-SearchBot | Anthropic | Search | Allow | Blocking stops indexing for search | | Claude-User | Anthropic | User-triggered | Allow | Blocking stops retrieval for a user query | | PerplexityBot | Perplexity | Search | Allow | Not used for training | | Perplexity-User | Perplexity | User-triggered | Allow | Generally ignores robots.txt | | Googlebot | Google | Search | Allow, always | Blocking removes you from Google | | Google-Extended | Google | Training and grounding control | Block if you want | No effect on Search or ranking | | Bingbot | Microsoft | Search and Copilot | Allow | Copilot is controlled by meta tags, not robots.txt | | Applebot | Apple | Search for Siri, Spotlight, Safari | Allow | Follows your Googlebot rules if unnamed | | Applebot-Extended | Apple | Training control | Block if you want | Pages stay in search results | | CCBot | Common Crawl | Open archive, a common training input | Block if you want | Others impersonate it; verify by IP | | meta-externalagent | Meta | Training | Block if you want | | | meta-webindexer | Meta | Search | Allow | Helps Meta AI cite and link you | | meta-externalfetcher | Meta | User-triggered | Allow | May bypass robots.txt | | Amazonbot | Amazon | General crawler, may train | Block if you want | Caches robots.txt up to 30 days | | Amzn-SearchBot | Amazon | Search including Alexa | Allow | Not used for training | | Amzn-User | Amazon | User-triggered for Alexa | Allow | May not follow every directive | | Bytespider | ByteDance | Crawler, compliance disputed | Block | No documented search role for Western stores | Highlighted rows are the bots that put you in front of users. Everything else is a licensing call. Details and sources per company follow. ### OpenAI Source: [OpenAI's crawler overview](https://developers.openai.com/docs/bots). - GPTBot. Training. "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." - OAI-SearchBot. Search. Surfaces sites in ChatGPT's search features. OpenAI says robots.txt changes take about 24 hours to apply. - ChatGPT-User. User-triggered. Visits a page when a user asks a question or uses an action. OpenAI says it "is not used for crawling the web in an automatic fashion" and that robots.txt rules may not apply. OpenAI states that each robots.txt setting is independent, so blocking GPTBot does not affect ChatGPT search. The current UA strings carry version numbers, for example GPTBot/1.4 and OAI-SearchBot/1.4, so match on the token, not the full string. ### Anthropic Source: [Anthropic's crawler article](https://support.claude.com/en/articles/8896518-about-claude-s-web-crawler). - ClaudeBot. Training. Disallowing it "signals that the site's future materials should be excluded from our AI model training datasets." It honors robots.txt and supports Crawl-delay. - Claude-SearchBot. Search. Disallowing it "prevents our system from indexing your content for search optimization." - Claude-User. User-triggered. Disallowing it "prevents our system from retrieving your content in response to a user query." Anthropic advises against IP blocking and says rules must be added per subdomain. ### Perplexity Source: [Perplexity's bots guide](https://docs.perplexity.ai/guides/bots). - PerplexityBot. Search. Perplexity says it is "not used to crawl content for AI foundation models" and recommends allowing it so you appear in results. - Perplexity-User. User-triggered. Perplexity says it "generally ignores robots.txt rules" because a user asked for the page, and that it isn't used for training. ### Google Source: [Google's common crawlers page](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers). - Googlebot. Search, including Discover and other Search features. Never block this. - Google-Extended. Training and grounding control. It has no separate user agent string. It's a robots.txt token that controls whether content Google already crawls may be used to train Gemini models and for grounding. Google says it does not affect inclusion in Search and is not a ranking signal. ### Microsoft and Bing Source: [Bing Webmaster Blog, September 2023](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat). Bing doesn't publish a separate Copilot crawler. Its blog says content in the Bing index may appear in Bing Chat (now Copilot) answers and may be used for training, and the controls are meta tags, not a user agent. NOCACHE limits Copilot to URL, title and snippet. NOARCHIVE excludes the page from Copilot answers and training. Both leave normal Bing results unchanged. So the decision on Bingbot is binary: allow it, or lose Bing and Copilot together. ### Apple Source: [About Applebot](https://support.apple.com/en-us/119829). - Applebot. Search for Siri, Spotlight and Safari. Allowing it lets you appear there. If you have no Applebot rules, it follows your Googlebot rules. - Applebot-Extended. Training control only. It doesn't crawl. Disallowing it opts you out of training Apple's foundation models, and Apple says pages that disallow it "can still be included in search results." ### Common Crawl Source: [commoncrawl.org/ccbot](https://commoncrawl.org/ccbot). CCBot builds an open web archive that anyone can download. Common Crawl is a non-profit and doesn't say who uses the data, but it is a common input to training sets. Block with User-agent: CCBot. Common Crawl warns that other crawlers impersonate CCBot and publishes IP ranges for verification. ### Meta Source: [Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers). - meta-externalagent. Training and product improvement. - meta-webindexer. Search. Meta says allowing it helps Meta AI cite and link to your content. - meta-externalfetcher. User-triggered. Meta says it "may bypass robots.txt because it performs fetches that were requested by the user." - facebookexternalhit. Link previews on Facebook, Instagram and Messenger. Not an AI bot, keep it allowed. ### Amazon Source: [developer.amazon.com/amazonbot](https://developer.amazon.com/amazonbot). - Amazonbot. General crawler that "may be used to train Amazon AI models." - Amzn-SearchBot. Search experiences including Alexa. Not used for training. Follows rules for other search bots if it isn't named. - Amzn-User. User-triggered live fetches for Alexa. Not used for training and may not follow every directive. Amazon's bots may cache robots.txt for up to 30 days. ### ByteDance Bytespider is ByteDance's crawler. Its official documentation is on the Toutiao webmaster platform in Chinese, which I haven't verified directly, so I'm relying on [ppc.land's summary](https://ppc.land/bytespider/): the token is Bytespider, ByteDance says it follows the Robots Exclusion Protocol, and independent reports dispute how well it complies. It has no documented search or user-triggered role that would benefit a Western store, so most sites block it. ## Paste-ready robots.txt: allow discovery, block training This keeps you visible in ChatGPT search, Claude, Perplexity, Google, Bing, Siri, Meta AI and Alexa, and declines training use where a token exists for it. Put the block above your existing rules and keep your sitemap line. ``` # Search and user-triggered fetchers: allow User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Applebot User-agent: meta-webindexer User-agent: meta-externalfetcher User-agent: Amzn-SearchBot User-agent: Amzn-User Allow: / # Training crawlers: block User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent User-agent: Amazonbot User-agent: Bytespider Disallow: / # Everyone else, including Googlebot and Bingbot User-agent: * Disallow: /cart Disallow: /checkout Disallow: /account Sitemap: https://www.yoursite.com/sitemap.xml ``` Two notes. Googlebot and Bingbot stay under the wildcard group with your normal disallows, since blocking them would remove you from search and Copilot together. And Bing's training control is a meta tag, not robots.txt, so add `` on pages you want kept out of Copilot answers, knowing it also removes them from those answers. ## Paste-ready robots.txt: allow everything If you're happy for your content to be used for training, which many small brands are because the exposure is worth more to them than the licensing position, this is enough: ``` User-agent: * Disallow: /cart Disallow: /checkout Disallow: /account Sitemap: https://www.yoursite.com/sitemap.xml ``` Named bots with no group of their own fall back to the wildcard. A bot that finds its own group ignores the wildcard entirely, so a leftover GPTBot block from 2023 will still apply. Delete it. ## Cloudflare: you may be blocking these bots without knowing On 1 July 2025 Cloudflare announced that it would block AI crawlers by default. Its [press release](https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) says every new domain is asked at sign-up whether to allow AI crawlers, with blocking as the default, and that more than one million customers had already turned on the one-click block introduced in September 2024. Existing zones weren't switched automatically but a lot of site owners clicked the block when it was offered. Check this first If Cloudflare sits in front of your site, open AI Crawl Control and look at the Crawlers tab before touching robots.txt. An edge 403 is served before your file is ever read. That block is enforced at the edge with a 403, so your robots.txt is irrelevant if Cloudflare is turning the bot away first. Check it in the dashboard under AI Crawl Control, where the Crawlers tab lists each bot with its operator and its current Allow or Block action. Cloudflare's own [managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) feature, if you've enabled it, prepends rules that disallow Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, which lines up with the "block training" list above. Make sure the search and user-triggered bots are set to Allow in the Crawlers tab, or the robots.txt you just wrote never gets a chance. ## Shopify: robots.txt.liquid Shopify generates robots.txt for you, and its defaults don't block any of the bots above. To add rules, create templates/robots.txt.liquid in your theme. [Shopify's docs](https://shopify.dev/docs/storefronts/themes/seo/robots-txt) say this replaces the generated output, so loop over robots.default\_groups to keep Shopify's rules and add your own outside the loop. A minimal version that keeps defaults and blocks training bots: ``` {%- for group in robots.default_groups -%} {{- group.user_agent }} {%- for rule in group.rules -%} {{ rule }} {%- endfor -%} {%- if group.sitemap != blank -%} {{ group.sitemap }} {%- endif -%} {%- endfor -%} User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot User-agent: meta-externalagent User-agent: Amazonbot User-agent: Bytespider Disallow: / ``` Save a copy of your current /robots.txt first and diff it against the rendered output, as Shopify recommends. For the broader picture see [Shopify agentic storefronts](https://roasexpert.com/blog/shopify-agentic-storefronts). ## How to check which AI bots are hitting your site *Step 01***Read the crawler report** On Cloudflare: AI Crawl Control, then Metrics. Group by crawler, read the status code chart (403 blocked, 200 served) and the most popular paths. *Step 02***Or grep the logs** Without Cloudflare, pull 30 days of access logs and count hits and status codes for each token: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Applebot, CCBot, the Meta and Amazon tokens, Bytespider. *Step 03***Verify the real ones** OpenAI publishes IP lists at openai.com/gptbot.json, searchbot.json and chatgpt-user.json; Common Crawl at index.commoncrawl.org/ccbot.json. A "GPTBot" from a residential IP is not GPTBot. *Step 04***Separate crawls from referrals** Humans arriving from chatgpt.com or perplexity.ai show up in analytics, not crawler logs. See [AI assistant traffic in GA4](https://roasexpert.com/blog/ai-assistant-traffic-in-ga4). *Step 05***Re-check after every change** OpenAI, Meta and Amazon say to allow about 24 hours. Amazon may cache robots.txt for up to 30 days. ## Where ROAS Expert fits Crawler access is the first thing we check in [agentic SEO](https://roasexpert.com/agentic-seo) engagements, because a 403 from Cloudflare or a stale GPTBot block undoes everything else. Once the bots can get in, the work shifts to what they find: clean product pages, structured data and [product feeds for AI shopping](https://roasexpert.com/blog/product-feeds-for-ai-shopping). An [audit](https://roasexpert.com/audit) starts at $1,000 and includes a log review for every bot on this page. **Not sure which bots your site is turning away?** An agentic SEO audit includes a log review for every bot on this page, the Cloudflare settings and the robots.txt you should actually ship. From $1,000. [Book a 30-minute call](https://roasexpert.com/contact)[See the audit](https://roasexpert.com/audit) Want to see which of these crawlers your robots.txt lets in today? The [free AI SEO audit](https://tools.roasexpert.com/?utm_source=roasexpert&utm_medium=blog&utm_campaign=ai-crawlers) reads your robots.txt and reports access for 11 AI bots, GPTBot to PerplexityBot, in about twenty seconds. To write the file without losing your existing rules, the [robots.txt builder for AI crawlers](https://tools.roasexpert.com/robots-ai?utm_source=roasexpert&utm_medium=blog&utm_campaign=robots-ai) fetches your current robots.txt, shows what it does to each of the 11 bots above and lets you flip them one by one. ## Questions ### Should I block GPTBot? Only if you don't want your content used to train OpenAI's models. OpenAI says blocking GPTBot does not affect ChatGPT search, which is controlled by OAI-SearchBot, so you can block training and stay visible. ### What is the difference between GPTBot and ChatGPT-User? GPTBot crawls the web to collect training data. ChatGPT-User fetches a single page because a user asked ChatGPT something that needs it, and OpenAI says it isn't used for automatic crawling. ### Does blocking Google-Extended hurt my Google rankings? No. Google says Google-Extended does not affect inclusion in Search and is not a ranking signal. It only controls use of your content for Gemini training and grounding. ### Why are AI bots getting 403 errors on my site? Most often because Cloudflare is blocking them at the edge. Since July 2025 new Cloudflare domains block AI crawlers by default. Check the Crawlers tab in AI Crawl Control and set the search and user-triggered bots to Allow. ### Do user-triggered bots respect robots.txt? Not reliably. OpenAI, Perplexity, Meta and Amazon all say their user-triggered fetchers may ignore robots.txt because a person requested the page. Blocking them in robots.txt mostly results in a failed answer rather than privacy. [View full post](https://roasexpert.com/blog/ai-crawlers-robots-txt) ```json { "@context" : "http://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Rishabh Gupta" }, "dateModified" : "2026-11-03T10:29:46.894Z", "datePublished" : "2026-10-20T04:35:36Z", "headline" : "GPTBot, ClaudeBot, PerplexityBot: AI bots in robots.txt", "image" : { "@type" : "ImageObject", "height" : 60, "url" : "/hs/hsstatic/content_shared_assets/static-1.4092/img/default-amp-logo.png", "width" : 60 }, "mainEntityOfPage" : "https://roasexpert.com/blog/ai-crawlers-robots-txt", "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "height" : 60.0, "url" : "https://cdn2.hubspot.net/hubfs/24114652/Logo.svg", "width" : 60.0 }, "name" : "Insights" } } ```