The AI Crawler Directory: GPTBot, OAI-SearchBot, ClaudeBot and Your robots.txt

Most robots.txt advice is written for publishers worried about content theft. If your name is the product, the calculation inverts: the crawler is not stealing your work, it is delivering your reputation. You still need to know exactly who is knocking.

Why does one AI company send three different crawlers?

Because an AI answer is assembled from three different supply chains, and each has its own bot. OpenAI is the cleanest example, documented at platform.openai.com/docs/bots:

• GPTBot collects public web content that may be used to train future models. This is the long game: what GPTBot reads today can shape what a model "knows" from memory next year.
• OAI-SearchBot builds and refreshes the index behind ChatGPT search. This is the medium game: it determines whether your pages are candidates for citation when ChatGPT searches the web.
• ChatGPT-User fetches a page on demand when a user's question sends ChatGPT to your site right now. This is the short game: the live visit during a conversation that mentions you.

Each responds to its own name in robots.txt, and blocking one does not block the others. That separation is not bureaucracy; it is a control panel. It lets you say "do not train on me, but do cite me," or the reverse. Most site owners never touch the panel and get the default: whatever their CMS or CDN decided for them. If you want the mechanics of what each pipeline does with your name once the crawler leaves, that is the subject of training data vs retrieval.

The AI crawler directory

The named crawlers that matter for a personal site in 2026, with what blocking each one actually costs you:

Three sources anchor this table and are worth bookmarking: OpenAI's bot documentation, Anthropic's crawler help page covering ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity's crawler documentation, which also publishes the IP ranges each of its agents uses. Note the odd one out: Google-Extended is not a crawler at all. Googlebot does the fetching, and Google-Extended is a robots.txt token that tells Google whether the fetched content may feed Gemini training.

What should a personal site put in robots.txt?

Two honest configurations cover almost every individual. Pick by intent, not by fashion.

Configuration one: fully open

For anyone whose goal is to be known, recommended and named by machines. This is the PEO default:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

No AI-specific rules at all. Every declared crawler, training or retrieval, reads everything. Your only jobs are to keep the sitemap current and make sure nothing upstream, a CDN default, a WordPress plugin, a bot-protection rule, is silently blocking crawlers your robots.txt welcomes. Check your server logs or your CDN's bot dashboard for the user agents above; absence of visits is a finding, not a comfort.

Configuration two: cite me, do not train on me

For those uneasy about model training but unwilling to disappear from AI answers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Understand the trade before you ship it. Blocking training bots means future models are less likely to know you from parametric memory, so your entire AI presence rides on retrieval: being found and cited at question time. That is a legitimate strategy, but it is a bet on one door instead of two.

What about the crawlers this table leaves out?

The directory above covers the agents with direct consequences for whether AI engines can name you. A second tier exists and deserves a sentence each. Meta publishes crawlers for its AI training, controllable by user agent like the rest. Apple offers Applebot-Extended, which works like Google-Extended: not a separate crawler but a token telling Applebot whether fetched content may feed Apple's AI training. And there is a long tail of aggressive scrapers, ByteDance's Bytespider being the most frequently named, that have been widely reported to ignore robots.txt entirely, which is a useful reminder of what the file is and is not. For an individual running a PEO strategy none of these change the decision much: the engines your buyers actually ask are the ones in the table.

The other omission is deliberate: this directory will age. Labs rename agents, split them, and launch new ones without ceremony, which is why the configurations above control bots by name but default open, and why a twice-yearly review of the file belongs on your calendar in a way it never did before 2024.

Do AI crawlers actually obey robots.txt?

The declared ones do, on the record. OpenAI, Anthropic and Perplexity all document robots.txt compliance for their named agents, and Anthropic states its bots also respect the crawl-delay extension. But keep two caveats in view. First, robots.txt is a convention with no enforcement layer; a scraper that ignores it faces no technical barrier, and undeclared crawling remains a live controversy in the industry. Second, compliance is per-agent: a rule for GPTBot says nothing to OAI-SearchBot. The practical routine is to verify rather than trust: both OpenAI and Perplexity publish the IP ranges their bots crawl from, so a suspicious visitor claiming to be PerplexityBot can be checked against Perplexity's published JSON before you conclude anything.

How do I see which AI crawlers actually visit my site?

Stop guessing and read the guest book. Three ways in, from easiest to most precise:

• Your CDN or host's bot analytics. Cloudflare, Vercel, Netlify and most managed WordPress hosts now break out AI crawler traffic by name in their dashboards. Five minutes, no setup, and you immediately know whether GPTBot has ever seen your about page.
• Raw access logs. Filter for the user agent strings in the table above. Look at which paths each bot requests and how often. A retrieval bot hitting your homepage weekly but never reaching your best article is a routing problem you can fix with internal links and a current sitemap.
• A canary page. Publish a new page, submit the updated sitemap, and time how long each crawler takes to fetch it. That interval is your personal answer to "how fast does new material about me become citable," and it varies enough between sites to be worth measuring rather than assuming.

Whatever you find becomes the baseline for the framework below. Crawl data tells you whether machines can read you; only visibility testing tells you what they say. You need both numbers.

A decision framework in four questions

1. Is your name the product? If clients, employers or investors might ask an engine about you, run configuration one. Visibility is the whole point, and the mechanics of getting recommended by ChatGPT all assume the door is open.
2. Do you sell the content itself? Paywalled courses, licensed research, book manuscripts: block training bots on those paths specifically, keep your about page and public writing open. robots.txt rules take a path, not just a domain.
3. Are you managing a live reputation problem? Blocking crawlers does not remove what models already learned; it only stops corrections from getting in. Sites with something to fix should open the doors wider, then fix the source material.
4. Not sure? Default open, then measure. Watch your logs for AI user agents, and track your AI visibility monthly so the decision runs on data instead of vibes.

Review the file twice a year. New agents keep appearing, and the guide Search Engine Land maintains on AI crawlers is a reasonable changelog for the category. Fifteen lines of plain text decide whether machines can read you at all; audits of that file, and everything downstream of it, are part of our PEO service.