Should You Block GPTBot? A Decision Guide

Whether to block GPTBot depends on what you sell. A publisher protecting paid content refuses every AI crawler. A business that wants to be recommended permits the retrieval crawlers — OAI-SearchBot, PerplexityBot, Claude-SearchBot — because blocking them removes citation eligibility entirely. Training crawlers are a separate decision. This page gives the agents, the directives and the verification test.

See pricing

Before the first directive. Two things are required: write access to robots.txt at the domain root, and read access to the server access logs. robots.txt is a plain-text file at /robots.txt that names user-agents and states which paths each one is permitted to fetch. A user-agent is the identifying token an automated client sends with every request. A crawl directive is one Allow or Disallow line applied to the agents named above it. AI crawlers divide into two classes, with a third group beside them. A retrieval crawler fetches a page so an assistant can quote and link it in an answer. A training crawler copies text into a corpus used to build a later model version. A user-triggered fetcher retrieves one page because a person asked an assistant about it.

Step 1

Should you block GPTBot?

A business seeking recommendation permits the retrieval crawlers. Refusing the training crawlers is a separate decision that does not affect citation. A publisher protecting paid content refuses both classes. The two business models produce opposite correct answers.

The decision turns on where the revenue sits. A newspaper, a course seller and a research firm earn money from the text itself, so every copy taken without payment subtracts from saleable inventory, and refusing both classes protects it. A plumbing company, a law firm and a dental practice earn money from the call that follows a recommendation, so refusing the retrieval crawlers removes the recommendation and protects nothing at all.

GPTBot is a training crawler. OpenAI documents it as the agent that crawls content for training generative AI foundation models. Refusing GPTBot leaves OAI-SearchBot free to fetch the same pages and ChatGPT search free to cite them. Two agents, two lines, two independent decisions.

Crawler permission precedes every other technique in ai search optimization best practices, and a page formatted for an agent that was refused returns nothing.

Step 2

Retrieval crawlers and training crawlers are different

Retrieval crawlers fetch pages so an assistant can cite them in a live answer. Training crawlers collect text for future model versions. Blocking a training crawler preserves citation eligibility; blocking a retrieval crawler removes it entirely.

Retrieval crawlers and training crawlers are different
Property	Retrieval crawler	Training crawler
Reason for the fetch	to compose or index a live answer	to add text to a corpus
Effect of permitting	the page becomes eligible for citation and a link	the text enters a future model version
Effect of refusing	the page stops appearing in that assistant's answers	the text stays out of the corpus, citation intact
When the effect lands	at the next answer	at the next model release
Documented examples	OAI-SearchBot, PerplexityBot, Claude-SearchBot	GPTBot, ClaudeBot, CCBot

Read the table as five statements. A retrieval crawler fetches to compose or index a live answer, and a training crawler fetches to fill a corpus. Permitting a retrieval crawler makes the page citable, and permitting a training crawler places its text in a later model. Refusing a retrieval crawler ends citation on that assistant, and refusing a training crawler leaves citation intact. A retrieval decision lands at the next answer, and a training decision lands at the next model release. OAI-SearchBot, PerplexityBot and Claude-SearchBot are the documented retrieval agents, and GPTBot, ClaudeBot and CCBot are the documented training agents.

Anthropic's naming does not mirror OpenAI's, and the difference costs people citations. Anthropic documents ClaudeBot as the agent collecting web content for model training. Claude-SearchBot is the agent that navigates the web to improve search result quality, and Claude-User is the agent that reaches a website when a person asks Claude a question. Citation on Claude runs through Claude-SearchBot and Claude-User, never through ClaudeBot.

Step 3

The agents, and what each one does

Fourteen tokens carry this decision: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider and CCBot. Two of the fourteen carry no operator documentation.

The agents, and what each one does
Token	Operator	Class	Refusing it removes citation?	Primary source, checked 21 Sep 2026
`GPTBot`	OpenAI	training	no	OpenAI — Bots
`OAI-SearchBot`	OpenAI	retrieval	yes, in ChatGPT search answers	OpenAI — Bots
`ChatGPT-User`	OpenAI	user-triggered fetch	the directive is not honoured	OpenAI — Bots
`PerplexityBot`	Perplexity	retrieval	yes, in Perplexity results	Perplexity — Bots
`Perplexity-User`	Perplexity	user-triggered fetch	the directive is not honoured	Perplexity — Bots
`ClaudeBot`	Anthropic	training	no	Anthropic — crawler article
`Claude-SearchBot`	Anthropic	retrieval	yes, in Claude search results	Anthropic — crawler article
`Claude-User`	Anthropic	user-triggered fetch	yes, the directive is honoured	Anthropic — crawler article
`anthropic-ai`	🔴 unattributed	🔴 undocumented	🔴 unverified	none located
`Google-Extended`	Google	training control token	no	Google — crawler documentation
`Applebot-Extended`	Apple	training control token	no	Apple — About Applebot
`meta-externalagent`	Meta	training and product indexing	yes, in Meta product indexing	Meta — web crawlers
`Bytespider`	ByteDance	🔴 undocumented	🔴 unverified	none reachable
`CCBot`	Common Crawl	open corpus collection	no	Common Crawl — CCBot

GPTBot crawls content for training OpenAI's foundation models, so refuse it at no citation cost. OAI-SearchBot surfaces websites in ChatGPT search results, and OpenAI states opted-out sites are not shown, so permit it. ChatGPT-User visits a page when a person asks ChatGPT a question, so permit it. PerplexityBot surfaces and links websites in Perplexity results, so permit it. Perplexity-User visits a page to answer a user's question, so permit it. Neither ChatGPT-User nor Perplexity-User is bound by the directive, since both operators exempt user-initiated fetches. ClaudeBot collects web content contributing to Anthropic's training, so refuse it at no cost to citation. Claude-SearchBot improves the accuracy of Claude's search responses, so permit it. Claude-User reaches a website when a person queries Claude, so permit it. Google-Extended governs Gemini training alone and Applebot-Extended governs Apple foundation-model training alone, so decide both separately. meta-externalagent trains Meta's foundation models and indexes content for Meta products in one agent, so permit it for Meta visibility or refuse it to withhold training. CCBot builds Common Crawl's open repository, a training input for third parties, so refuse it at no cost to citation. anthropic-ai and Bytespider are the two unverified tokens, and this page recommends no directive for either.

Step 4

The directives to use

Publish one robots.txt group permitting the retrieval and user-triggered agents, and one refusing the training agents. Repeat every site-wide Disallow line inside each named group, because a crawler obeys the single most specific group naming it and ignores all others.

# robots.txt — a business seeking recommendation
# Retrieval and user-triggered agents: permitted.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: meta-externalagent
Disallow: /wp-admin/
Disallow: /cart/

# Training agents: refused.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
Disallow: /

# Every other crawler.
User-agent: *
Disallow: /wp-admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

The file does five things. The first group names seven agents that fetch for live answers — OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User and meta-externalagent — and opens the site to them apart from two administrative paths. Those two Disallow lines are load-bearing: Google documents that only one group is valid per crawler and other groups are ignored, so an omitted line exposes /wp-admin/ to all seven. The second group refuses the whole site to six training agents: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Bytespider. The third group restates those restrictions for every unnamed crawler. The Sitemap line belongs to the file, not to a group.

Three placements deserve a second look. meta-externalagent is permitted because Meta documents one agent doing two jobs, training foundation models and indexing content directly. ClaudeBot is refused as Anthropic's training crawler, and Claude-SearchBot and Claude-User carry the citations. Bytespider is refused with no documented record of compliance, so enforcement belongs at the CDN.

A publisher collapses both groups into one refusal naming all thirteen documented tokens: the seven retrieval and user-triggered agents plus the six training agents.

# robots.txt — a publisher protecting paid content
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: meta-externalagent
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
Disallow: /

Step 5

What Google-Extended actually controls

Google-Extended governs whether content trains Gemini models. It does not control Googlebot, Search indexing, or AI Overview eligibility. Blocking Google-Extended leaves organic ranking and AI Overview citation entirely unaffected, which surprises most people who set it.

Google states the limit in its crawling documentation: using Google-Extended does not affect a site's inclusion in Search, and it is not a ranking signal. AI Overviews and AI Mode sit inside Search, and Google names robots.txt directives for Googlebot as the control over Search crawling. The nosnippet, data-nosnippet, max-snippet and noindex controls govern what those surfaces display.

Google-Extended is a control token, not a crawler. Google documents that it has no separate HTTP request user agent string, that crawling runs under existing Google user agents, and that the token works in a control capacity. Applebot-Extended behaves the same way: Apple documents it as determining how data already crawled by Applebot is used, and states that pages disallowing it stay eligible for search results.

Step 6

How to verify a crawler reached your site

Check server access logs for the user-agent string. GPTBot, OAI-SearchBot and PerplexityBot each identify themselves. A directive change takes effect on the crawler's next visit, which ranges from hours to six weeks.

One other root-level file is routinely confused with this one. llms.txt proposes an index of a site's key documents and carries no access semantics at all — it grants nothing and blocks nothing. Google states it ignores the file. robots.txt remains the only file that controls whether a retrieval crawler reaches a page.

Run five checks against the raw access log:

Open the access log covering the 30 days since the robots.txt edit.
Filter each request line for the token, matched case-insensitively.
Confirm the declared reference URL inside the user-agent string.
Compare the earliest matching timestamp against the date of the edit.
Repeat the filter weekly until every permitted agent appears once.

Six declared strings carry the tokens. GPTBot sends GPTBot/1.4; +https://openai.com/gptbot. OAI-SearchBot sends OAI-SearchBot/1.4; +https://openai.com/searchbot. ChatGPT-User sends ChatGPT-User/1.0; +https://openai.com/bot. PerplexityBot sends PerplexityBot/1.0; +https://perplexity.ai/perplexitybot. Perplexity-User sends Perplexity-User/1.0; +https://perplexity.ai/perplexity-user. CCBot sends CCBot/2.0 (https://commoncrawl.org/faq/). Two tokens never appear in any log, Google-Extended and Applebot-Extended, because neither one is a crawler. A user-agent string is self-declared, so the log test confirms arrival rather than identity.

Running this test across an unfamiliar site is the opening pass of an AI visibility audit.

Questions

Common questions

Every answer below ships in the raw HTML, so an assistant reading this page without running a script still receives it.

Does blocking GPTBot remove a site from ChatGPT?

GPTBot is OpenAI's training crawler. ChatGPT search draws on OAI-SearchBot, a separate agent on a separate line. Refusing GPTBot while permitting OAI-SearchBot keeps a site citable in ChatGPT search answers and its text out of OpenAI's training corpus.

What happens to a site that refuses every AI crawler?

Every retrieval surface drops it. ChatGPT search, Perplexity and Claude search stop citing refused pages, and the text stays out of later model versions. Publishers accept the first outcome to secure the second.

Is anthropic-ai a real user agent?

Anthropic's crawler documentation names three agents: ClaudeBot, Claude-User and Claude-SearchBot. The token anthropic-ai circulates in published robots.txt files and third-party directories, and Anthropic does not list it. This page recommends no directive for it.

Selecting a provider to set and re-check these directives is covered in how to choose an ai seo agency. The practice they belong to is defined in What Is Generative Engine Optimization?.

Notes and sources. Seven primary sources carry every token on this page, each checked 21 September 2026. OpenAI documents four agents — GPTBot for training generative AI foundation models, OAI-SearchBot for surfacing websites in ChatGPT search results, ChatGPT-User for pages visited when users ask ChatGPT questions, and OAI-AdsBot for ad landing-page safety validation — and states that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers (OpenAI — Bots). Perplexity documents PerplexityBot as designed to surface and link websites in Perplexity search results, and Perplexity-User as a fetcher that robots.txt rules do not reliably govern because a user requested the page (Perplexity — Bots). Anthropic documents ClaudeBot as collecting web content that contributes to model training, Claude-SearchBot as navigating the web to improve search result quality, and Claude-User as accessing websites when individuals ask Claude questions, and states that its bots honour robots.txt directives (Anthropic — Does Anthropic crawl data from the web?). Google states that using Google-Extended does not affect a site's inclusion in Search and is not used as a ranking signal in Search (Google — Things to know about Google's web crawling), that Google-Extended has no separate HTTP request user agent string and operates in a control capacity (Google — Google common crawlers), that robots.txt directives for Googlebot are the control for how a site is crawled for Search including its AI features (Google — AI features and your website), and that only one group is valid for a particular crawler while other groups are ignored (Google — robots.txt specifications). Apple documents Applebot-Extended as an opt-out from training Apple's foundation models and states that pages disallowing it are still included in search results (Apple — About Applebot). Meta documents meta-externalagent as used for training foundation AI models or improving products by indexing content directly (Meta — Web crawlers). Common Crawl documents CCBot and the user-agent string CCBot/2.0 (https://commoncrawl.org/faq/) (Common Crawl — CCBot).

🔴 Two tokens carry no primary source. ByteDance publishes no reachable documentation for Bytespider; the reference URL inside its own user-agent string resolves inside China only, and no operator statement of robots.txt compliance was located on 21 September 2026. Anthropic's current documentation does not name anthropic-ai; the token appears only in third-party crawler directories and in copied robots.txt files. This page names both and recommends a directive for neither.

Next step

Find out which assistants name you today.

The audit runs 40 checks across five categories and queries seven assistants for your citation baseline. Findings in ten business days.

See pricing