GPTBot, ClaudeBot, PerplexityBot: AI bots in robots.txt

By Rishabh Gupta, founder of ROAS Expert. Last updated 19 October 2026.

If you want AI assistants to recommend your store or product, allow the search and user-triggered bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot) and decide separately whether to block the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Amazonbot, Bytespider). Each company documents these as independent switches, so blocking training does not have to cost you visibility. The paste-ready blocks are below.

Three kinds of AI bot, three different decisions

Most robots.txt mistakes come from treating "AI bots" as one thing. The official docs split them into three roles.

For a store or SaaS company that wants to be found, the position is simple. Allow the search and user-triggered bots. Block training only if your legal or content position says so. The two are independent in every bot family listed below.

The bots, by company, from their official pages

OpenAI

Source: OpenAI's crawler overview.

• GPTBot. Training. "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."
• OAI-SearchBot. Search. Surfaces sites in ChatGPT's search features. OpenAI says robots.txt changes take about 24 hours to apply.
• ChatGPT-User. User-triggered. Visits a page when a user asks a question or uses an action. OpenAI says it "is not used for crawling the web in an automatic fashion" and that robots.txt rules may not apply.

OpenAI states that each robots.txt setting is independent, so blocking GPTBot does not affect ChatGPT search. The current UA strings carry version numbers, for example GPTBot/1.4 and OAI-SearchBot/1.4, so match on the token, not the full string.

Anthropic

Source: Anthropic's crawler article.

• ClaudeBot. Training. Disallowing it "signals that the site's future materials should be excluded from our AI model training datasets." It honors robots.txt and supports Crawl-delay.
• Claude-SearchBot. Search. Disallowing it "prevents our system from indexing your content for search optimization."
• Claude-User. User-triggered. Disallowing it "prevents our system from retrieving your content in response to a user query."

Anthropic advises against IP blocking and says rules must be added per subdomain.

Perplexity

Source: Perplexity's bots guide.

• PerplexityBot. Search. Perplexity says it is "not used to crawl content for AI foundation models" and recommends allowing it so you appear in results.
• Perplexity-User. User-triggered. Perplexity says it "generally ignores robots.txt rules" because a user asked for the page, and that it isn't used for training.

Google

Source: Google's common crawlers page.

• Googlebot. Search, including Discover and other Search features. Never block this.
• Google-Extended. Training and grounding control. It has no separate user agent string. It's a robots.txt token that controls whether content Google already crawls may be used to train Gemini models and for grounding. Google says it does not affect inclusion in Search and is not a ranking signal.

Microsoft and Bing

Source: Bing Webmaster Blog, September 2023.

Bing doesn't publish a separate Copilot crawler. Its blog says content in the Bing index may appear in Bing Chat (now Copilot) answers and may be used for training, and the controls are meta tags, not a user agent. NOCACHE limits Copilot to URL, title and snippet. NOARCHIVE excludes the page from Copilot answers and training. Both leave normal Bing results unchanged. So the decision on Bingbot is binary: allow it, or lose Bing and Copilot together.

Apple

Source: About Applebot.

• Applebot. Search for Siri, Spotlight and Safari. Allowing it lets you appear there. If you have no Applebot rules, it follows your Googlebot rules.
• Applebot-Extended. Training control only. It doesn't crawl. Disallowing it opts you out of training Apple's foundation models, and Apple says pages that disallow it "can still be included in search results."

Common Crawl

Source: commoncrawl.org/ccbot. CCBot builds an open web archive that anyone can download. Common Crawl is a non-profit and doesn't say who uses the data, but it is a common input to training sets. Block with User-agent: CCBot. Common Crawl warns that other crawlers impersonate CCBot and publishes IP ranges for verification.

Meta

Source: Meta web crawlers.

• meta-externalagent. Training and product improvement.
• meta-webindexer. Search. Meta says allowing it helps Meta AI cite and link to your content.
• meta-externalfetcher. User-triggered. Meta says it "may bypass robots.txt because it performs fetches that were requested by the user."
• facebookexternalhit. Link previews on Facebook, Instagram and Messenger. Not an AI bot, keep it allowed.

Amazon

Source: developer.amazon.com/amazonbot.

• Amazonbot. General crawler that "may be used to train Amazon AI models."
• Amzn-SearchBot. Search experiences including Alexa. Not used for training. Follows rules for other search bots if it isn't named.
• Amzn-User. User-triggered live fetches for Alexa. Not used for training and may not follow every directive.

Amazon's bots may cache robots.txt for up to 30 days.

ByteDance

Bytespider is ByteDance's crawler. Its official documentation is on the Toutiao webmaster platform in Chinese, which I haven't verified directly, so I'm relying on ppc.land's summary: the token is Bytespider, ByteDance says it follows the Robots Exclusion Protocol, and independent reports dispute how well it complies. It has no documented search or user-triggered role that would benefit a Western store, so most sites block it.

Paste-ready robots.txt: allow discovery, block training

This keeps you visible in ChatGPT search, Claude, Perplexity, Google, Bing, Siri, Meta AI and Alexa, and declines training use where a token exists for it. Put the block above your existing rules and keep your sitemap line.

# Search and user-triggered fetchers: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amzn-SearchBot
User-agent: Amzn-User
Allow: /
# Training crawlers: block
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Sitemap: https://www.yoursite.com/sitemap.xml

Two notes. Googlebot and Bingbot stay under the wildcard group with your normal disallows, since blocking them would remove you from search and Copilot together. And Bing's training control is a meta tag, not robots.txt, so add <meta name="robots" content="noarchive"> on pages you want kept out of Copilot answers, knowing it also removes them from those answers.

Paste-ready robots.txt: allow everything

If you're happy for your content to be used for training, which many small brands are because the exposure is worth more to them than the licensing position, this is enough:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Sitemap: https://www.yoursite.com/sitemap.xml

Named bots with no group of their own fall back to the wildcard. A bot that finds its own group ignores the wildcard entirely, so a leftover GPTBot block from 2023 will still apply. Delete it.

Cloudflare: you may be blocking these bots without knowing

On 1 July 2025 Cloudflare announced that it would block AI crawlers by default. Its press release says every new domain is asked at sign-up whether to allow AI crawlers, with blocking as the default, and that more than one million customers had already turned on the one-click block introduced in September 2024. Existing zones weren't switched automatically but a lot of site owners clicked the block when it was offered.

That block is enforced at the edge with a 403, so your robots.txt is irrelevant if Cloudflare is turning the bot away first. Check it in the dashboard under AI Crawl Control, where the Crawlers tab lists each bot with its operator and its current Allow or Block action. Cloudflare's own managed robots.txt feature, if you've enabled it, prepends rules that disallow Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, which lines up with the "block training" list above. Make sure the search and user-triggered bots are set to Allow in the Crawlers tab, or the robots.txt you just wrote never gets a chance.

Shopify: robots.txt.liquid

Shopify generates robots.txt for you, and its defaults don't block any of the bots above. To add rules, create templates/robots.txt.liquid in your theme. Shopify's docs say this replaces the generated output, so loop over robots.default_groups to keep Shopify's rules and add your own outside the loop. A minimal version that keeps defaults and blocks training bots:

{%- for group in robots.default_groups -%}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{%- endfor -%}
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /

Save a copy of your current /robots.txt first and diff it against the rendered output, as Shopify recommends. For the broader picture see Shopify agentic storefronts.

How to check which AI bots are hitting your site

Where ROAS Expert fits

Crawler access is the first thing we check in agentic SEO engagements, because a 403 from Cloudflare or a stale GPTBot block undoes everything else. Once the bots can get in, the work shifts to what they find: clean product pages, structured data and product feeds for AI shopping. An audit starts at $1,000 and includes a log review for every bot on this page.

Want to see which of these crawlers your robots.txt lets in today? The free AI SEO audit reads your robots.txt and reports access for 11 AI bots, GPTBot to PerplexityBot, in about twenty seconds.

To write the file without losing your existing rules, the robots.txt builder for AI crawlers fetches your current robots.txt, shows what it does to each of the 11 bots above and lets you flip them one by one.

Questions