Should You Block AI Crawlers? Weighing the Trade-Off

"Should you block AI crawlers" is really two separate questions wearing one robots.txt disallow line: do you want your content used to train future language models, and do you want your content retrieved and cited when someone asks a chatbot a question today. Those are different bots doing different jobs, and a single blanket rule often answers the wrong one. Before touching the file, it helps to know which crawler does which job, because "block AI crawlers" as a strategy is really "block these several kinds of AI crawlers, in this specific way, while treating a different kind on purpose."

The short version: blocking a training crawler mostly costs you influence over how a future model describes your brand or topic, which is real but slow-moving and hard to measure. Blocking a retrieval crawler costs you appearing in AI Overviews, ChatGPT's search-style answers, or Perplexity's answers today, which behaves much more like disappearing from an index. Conflating the two, which the naming conventions make easy to do, is the single most common mistake publishers make when they decide to "just block the AI bots."

Training Bots and Retrieval Bots Are Not the Same Bot

A training crawler visits your pages, once or on its own periodic schedule, to add your text to a corpus that eventually gets baked into a future model version through pretraining or fine-tuning. Once that snapshot is taken, disallowing the crawler tomorrow does not retroactively remove your content from a model already trained on it. The block only affects future crawls and future model generations, not the one already shipped.

A retrieval crawler, by contrast, fetches a page in something close to real time: a user asked a question, the system decided your page might hold the answer, and it grabbed the current version of the page to summarize or quote from, usually with a citation link back to you. Block this crawler and the effect shows up immediately, the next time someone asks that question. This is the crawler that behaves like a de-facto AI search index, and it is the one worth thinking hardest about before you touch it.

Vendors are inconsistent about labeling the two. Some publish a single user agent that does both jobs; others publish two or three separate tokens, one per function. Common Crawl's bot is a bulk archival crawler reused by many different model builders rather than one company's dedicated tool, so disallowing it opts you out of an entire family of downstream uses at once, not just a single vendor's plans.

What the Major Tokens Actually Control

Treat this as a lookup, not something to memorize. Vendors add tokens as they ship new products, so the categories below matter more than any single name.

• GPTBot, operated by OpenAI, crawls to help train future OpenAI models. Disallowing it opts your content out of that training pool going forward; it has no effect on ChatGPT's ability to browse a specific page live when a user asks it to.
• OAI-SearchBot, also OpenAI, fetches and indexes pages specifically to surface and cite them in ChatGPT's search-style answers. Disallowing this token is the one that actually removes you from being cited there.
• ChatGPT-User fires when someone inside a ChatGPT conversation asks it to visit a specific URL, or a connected action reaches out on the user's behalf. Blocking it breaks that on-demand fetch, not the training or search-index behavior above.
• Google-Extended controls whether your content can be used to train Gemini models and Vertex AI's generative features. It is a separate, standalone token from Googlebot, which handles ordinary Search crawling and indexing, and the distinction is worth its own section below.
• ClaudeBot and anthropic-ai, from Anthropic, crawl for model training and improvement, functioning as that company's counterpart to GPTBot.
• Applebot-Extended is an opt-out specifically for using crawled content in Apple's generative features, kept separate from the original Applebot that has long powered Siri and Spotlight search results.
• PerplexityBot fetches pages to ground and cite Perplexity's answers, acting as that product's retrieval layer in the same way OAI-SearchBot does for ChatGPT.
• Bytespider and meta-externalagent are training-oriented crawlers from ByteDance and Meta respectively; treat a decision about either one the same way you would treat GPTBot.
• CCBot belongs to the non-profit Common Crawl project, whose archive is reused by many model builders as raw training data. Blocking it is a broad, one-time opt-out rather than a single-vendor decision.
• Amazonbot is a general-purpose crawler feeding Alexa and shopping features, not an LLM-training bot exclusively, so it deserves a lower priority unless you have a specific reason to disallow it.

The Google-Extended Versus Googlebot Confusion

Because Google is the search engine publishers already worry about most, this is the mistake with the highest stakes attached to it. Googlebot is the crawler that builds Google's search index and determines what ranks; disallowing it removes you from Google Search entirely, which is rarely what anyone actually intends. Google-Extended is a separate, standalone token that only controls whether your content can be used to improve Gemini's underlying models and Vertex AI's generative APIs. Google's own documentation states that toggling Google-Extended does not change your ranking or your eligibility to appear in Google Search results.

The detail that catches people out: Google has said that AI Overviews draw on the regular Search index that Googlebot builds, not on the separate Gemini-training relationship that Google-Extended governs. That means disallowing Google-Extended does not pull your pages out of AI Overviews, because there is no dedicated robots.txt token for opting out of that particular Search feature. If avoiding an AI Overviews citation is actually your goal, blocking Google-Extended will not achieve it; the only lever that affects that outcome is the same one that governs ordinary Search inclusion, and touching that one has much bigger consequences.

This is worth confirming in your own server logs rather than assuming. Search the access log for "Google-Extended" and "Googlebot" separately, confirm which one your rule actually matches, and re-check after any edit that a broad wildcard has not caught the wrong crawler along with the one you meant to stop.

What Blocking Training Bots Costs You, Honestly

Blocking training crawlers is close to a free option for most publishers. It does not change current search visibility or current AI-answer visibility, since the effect is forward-looking only, and even then it only shows up in whichever future model generation gets trained after the block takes effect, and only if the vendor faithfully filters out the URLs you disallowed. The cost, such as it is, only shows up if you were counting on broad model training as a brand-awareness channel: some publishers deliberately allow training crawlers because being described accurately by many independently trained models functions a little like being cited across reference material. That is a bet on future relevance, not a certainty, and nobody can measure its return today.

What Blocking Retrieval Bots Costs You, Honestly

Blocking a retrieval crawler is a same-day decision with a same-day effect. The next time someone asks a chatbot the kind of question your page answers, the system will not have your page available to answer from, and if it does answer anyway from another source, it will not cite you. Publishers who depend on organic clicks are increasingly watching AI-driven citations as an early, if still unproven, source of qualified visits alongside organic search, and blocking a retrieval bot forecloses that channel outright rather than letting it develop on its own.

Before disallowing a retrieval-oriented token, ask honestly whether a meaningful share of your audience already discovers you through a chatbot's answer, whether the platform's direct answer still sends the reader to your page for the full detail the way a search snippet often does, and whether you would accept the same loss if a conventional search engine, rather than an AI assistant, were the one making the decision to exclude you.

Writing and Testing the Rules Correctly

Robots.txt lives at the root of your domain, one file per site, and it is read as a series of user-agent blocks each followed by their own rules. A common pattern for blocking several training-oriented bots at once looks like this:

User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: /

You do not have to block a bot from your entire site. Disallowing a single path, such as a subscriber-only or licensed section, protects the content that actually needs it while leaving the rest open to citation, a common middle path for publishers who want AI-citation traffic on free content but not on paywalled reporting sold elsewhere.

After editing the file, fetch it directly at the domain root and confirm it renders as plain text rather than a redirect or an error page. Check each user-agent token's spelling carefully; matching is forgiving of case but not of a misspelled name, and a token that does not exactly match what the vendor documents is simply ignored rather than flagged as an error.

Enforcement Beyond Robots.txt

Robots.txt is a request, not a lock. It works only because reputable operators choose to honor it, and that voluntary compliance has already been publicly questioned for at least one major AI vendor, with site operators reporting crawler traffic that kept requesting disallowed paths. If you need actual enforcement rather than a polite ask, look past the text file.

• Verify a crawler's identity against the IP ranges or reverse-DNS pattern the vendor documents for that specific bot, rather than trusting the declared user-agent string alone. A request claiming to be a known crawler from an address outside that vendor's published range is not actually that crawler, since a user-agent header is trivial for anyone to forge.
• A web application firewall or edge network can block or challenge traffic by verified bot identity at the network layer, which is the actual enforcement point that a text file cannot provide by itself.
• Rate-limit or challenge, with a script check or similar, any request pattern that looks automated but fails to match a verified crawler, instead of trusting the user-agent header on its own.
• Some publishers layer a machine-readable rights reservation, such as the protocol built around the European Union's text-and-data-mining exceptions, on top of robots.txt, which asserts a legal opt-out rather than a request, though enforcing it still generally means pursuing the operator directly.
• A terms-of-service clause prohibiting automated scraping gives a contractual basis to act against a non-compliant crawler, but only if you are prepared to pursue it; it is a legal backstop, not a live technical control.

A Decision That Fits Your Situation, Not a Universal Answer

There is no single correct posture, but three cover most publishers in practice. Blocking everything suits content that is licensed, paywalled, or otherwise monetized in a way that any reuse, training or citation, competes directly with revenue. Allowing everything suits publishers for whom being described accurately by AI systems, and cited in AI-generated answers, matters more than controlling reuse, which is common when reach and brand mention are the actual goal. Splitting by function is the middle path: disallow training-only crawlers such as GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, Bytespider, and CCBot, while allowing retrieval crawlers such as OAI-SearchBot, ChatGPT-User, and PerplexityBot, staying eligible for today's citations while opting out of tomorrow's model weights.

Whichever posture you choose, revisit it periodically. New tokens appear as vendors ship new products, and a robots.txt file written a year ago may already be silent on a crawler that did not exist when you wrote it, which functions as an implicit allowance by omission, since the file only blocks what it actually names.