robots.txt recipes for AI crawlers
AI companies' crawlers differ in how far they honor robots.txt, so it only works if you know each company's crawler names and list them separately. Block training crawls only, keep search visibility open, slow crawling down or block every published name: click a card to copy the lines to paste into robots.txt. Each card carries one line on what that company's documentation says its crawler honors. The strip on each card has one block per User-agent group: red blocks, green allows, yellow only slows the crawl.
OpenAI assigns training crawls to GPTBot. This rule does not reach OAI-SearchBot, which handles visibility in ChatGPT search.
User-agent: GPTBot
Disallow: /OpenAI's documentation says the two names can be allowed or blocked separately in robots.txt. Search-side changes take about 24 hours to take effect.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Blocks only OAI-SearchBot, which handles visibility in ChatGPT search. Per OpenAI's documentation, changes take about 24 hours to take effect.
User-agent: OAI-SearchBot
Disallow: /Anthropic says ClaudeBot supports the non-standard Crawl-delay extension and gives this value (1) as its example.
User-agent: ClaudeBot
Crawl-delay: 1Crawl-delay is a non-standard extension, so it goes only in the ClaudeBot group, where support has been stated. The other two groups are names OpenAI says can be handled separately.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Crawl-delay: 1Anthropic says ClaudeBot follows robots.txt.
User-agent: ClaudeBot
Disallow: /These are visits where a ChatGPT user's question opens the page. OpenAI's documentation says that because these actions are initiated by a user, robots.txt rules may not apply.
User-agent: ChatGPT-User
Disallow: /Of the four, ChatGPT-User may not be bound by the rules. This works less as a block than as a public statement of the site's intent.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /FAQ
- Does blocking in robots.txt stop the crawling?
- The standard that defines the file's rules (RFC 9309, September 2022) states plainly that robots.txt is a set of rules crawlers are asked to follow, not a mechanism that decides access rights. Even companies that follow the rules differ in how far they apply them.
- If the User-Agent says GPTBot, is it really OpenAI?
- Anyone can imitate a name (User-Agent). OpenAI publishes the IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User as JSON files. Checking the connecting address against those lists filters out requests that only borrow the name.
- How do I verify Google's agents?
- Google attaches Web Bot Auth signatures to some requests from its agents and publishes the address of its public key list. Checking that signature on your server or CDN tells you whether the agent is really Google's. Google also notes that not every request carries a signature.
- What about visits robots.txt cannot filter?
- When an agent visits on a person's instruction, robots.txt alone cannot filter it out. IP range checks and request signatures also work only for companies that already identify themselves. So put request caps where a single request makes the server do a lot of work, such as search, filter combinations and large file downloads. Without logging the User-Agent, the connecting address and the request path, there is no way to trace who did what when something goes wrong.