Personal websites
If you run a personal blog, portfolio, family site or small creator page, crawler blocking starts with naming the right user agents and understanding that robots.txt is a request, not a lock.
Do this carefully
Do not copy a random blocklist into production. Search crawlers, AI search crawlers, training crawlers and user-triggered fetchers can have different purposes. Blocking the wrong agent can hide your pages from tools you still want.
Server logs show who actually visits. Keep a short review cycle: add rules, watch traffic, check whether important pages still appear where you expect, then adjust.
| User agent | Company | Why it matters | Careful note |
|---|---|---|---|
GPTBot | OpenAI | Commonly used for OpenAI web crawling controls. | Block if you do not want this crawler to fetch your public pages for AI-related use. Verify current OpenAI docs. |
ChatGPT-User | OpenAI | User-triggered browsing/fetching agent in many discussions. | Blocking can stop ChatGPT users from asking it to read your page. This is not the same intent as broad training crawling. |
Google-Extended | A control Google documents for some generative AI use cases. | This is not the same as blocking Google Search. Do not block Googlebot unless you want search visibility affected. | |
ClaudeBot | Anthropic | Anthropic crawler name seen in crawler-control discussions. | Use only after checking Anthropic’s latest documentation. |
Claude-SearchBot | Anthropic | Search-related Claude crawler name seen in public guidance. | Consider the visibility tradeoff before blocking search-style agents. |
PerplexityBot | Perplexity | Perplexity crawler/user-agent discussed by publishers. | Blocking may affect how your content is discovered or cited by Perplexity-style answer engines. |
CCBot | Common Crawl | Public web crawler used for large web datasets. | Often blocked by publishers who do not want pages copied into public crawl corpora. |
Bytespider | ByteDance | Crawler associated with ByteDance. | Useful to review if you want a broad AI-crawler blocklist. |
How crawlers find you
Before you can block a crawler, it helps to know how it arrives. Automated fetchers rarely stumble onto a page at random; they follow a fairly predictable chain of discovery. The descriptions below are general and simplified, and the exact behaviour of any named bot can change, so verify against each provider's official documentation before acting.
Crawlers find URLs by following links from other pages, reading your XML sitemap, checking your robots.txt, and drawing on large public link datasets. A brand-new page usually becomes reachable the moment something public links to it.
A crawler sends an HTTP request identifying itself with a user-agent string. Your server can inspect that string and decide how to respond, though user agents can be imitated, so treat them as a signal rather than proof.
Well-behaved crawlers re-visit periodically to pick up changes, and many respect crawl-delay hints or their own rate limits. Poorly behaved ones may fetch aggressively, which is one reason rate limiting matters alongside polite rules.
Three kinds of bots
Many blocking mistakes come from treating every bot as identical. The three categories below overlap in mechanics but differ in purpose, and blocking one can have very different consequences from blocking another. Naming and behaviour evolve, so confirm the current role of any specific agent before you rely on this framing.
| Type | What it does | Effect of blocking |
|---|---|---|
| Search crawler | Indexes pages so they can appear in a search engine's results. | Blocking can remove you from that search engine. Usually not what you want. |
| AI-training crawler | Fetches pages to build or improve training datasets for AI models. | Blocking asks the provider not to use your public pages for training. This is the main target for most people here. |
| AI live-retrieval bot | Fetches a page on demand when a user asks an AI assistant to read or cite it. | Blocking can stop your page from being read or cited in real time, which affects visibility in answer engines rather than training. |
Training crawlers and live-retrieval bots are the ones most people mean by "AI bots", but some also power discovery and citation. Decide deliberately whether you want to be trained on, cited, both, or neither, because those are separate choices.
A layered approach
Think of blocking as defence in depth rather than a single switch. Each layer catches cases the others miss, and each has honest limits. Start with the polite, low-risk layers and add stricter ones only where the tradeoff is worth it.
A plain-text file that asks named crawlers not to fetch listed paths. Compliant bots honour it; non-compliant ones ignore it. It is the standard first step and the least disruptive.
A page-level meta tag or an HTTP response header can request that content not be indexed or used in certain ways. Support varies by bot, so treat directives like noai as requests, not guarantees.
A web application firewall or rate limiter can block or throttle traffic by user agent, IP range or behaviour. Unlike robots.txt, this actually enforces at the server, though it needs care to avoid catching legitimate visitors.
Content behind a login is not publicly fetchable, which is the most reliable way to keep casual crawlers out entirely. The cost is friction for real users, so reserve it for material that genuinely needs it.
Managed platforms add another option. Some providers, such as Cloudflare-type services, offer built-in controls that identify and block categories of AI bots for you. These can be convenient, but you still depend on the provider's detection accuracy and definitions, so verify what a given toggle actually does before trusting it.
# robots.txt — a polite first layer (verify current agent names)
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
# Keep normal search working: do NOT blanket-block Googlebot
User-agent: Googlebot
Allow: /# HTTP response header example (server or WAF config)
X-Robots-Tag: noai, noimageaiWhy blocking is imperfect
It would be misleading to present any of these techniques as airtight. They meaningfully reduce unwanted crawling, but they do not make your content unreachable. Setting expectations correctly is part of doing this responsibly.
Aim to make unwanted AI crawling meaningfully less convenient and to document your stated preference clearly. That is achievable. Guaranteed, universal exclusion is not.
Watch your logs
Rules are only useful if you check whether they are working. Your server access logs record who requested what, when, and with which user-agent string. A short, regular review tells you which bots are actually visiting and whether your rules are being honoured. Remember that user agents can be imitated, so treat the logs as evidence to interpret rather than absolute truth.
Scan for known AI user-agent strings such as GPTBot, CCBot, ClaudeBot, PerplexityBot and Bytespider. Note their request volume and which paths they fetch.
Watch for agents that request your site heavily after you have disallowed them, unfamiliar agents fetching many pages quickly, or generic browser strings hitting URLs no human would.
If a compliant bot ignores a rule, re-check your syntax. If a bot genuinely ignores robots.txt, consider a firewall or rate-limit rule, accepting the added risk of false positives.
# Rough example: count requests by AI-related user agent
grep -Ei "GPTBot|CCBot|ClaudeBot|PerplexityBot|Bytespider" access.log \
| awk '{print $NF}' | sort | uniq -c | sort -rnStep-by-step rollout
Because a wrong rule can quietly hurt legitimate visibility, roll out changes gradually and watch the effect at each step. The sequence below favours reversibility over speed, which is the right priority for most personal and small sites.
Related reading
Compare AI models before choosing a workflow.
ChatGPT Alternatives chatgpt-alternatives.comFind neutral alternatives when one AI provider is not enough.
AI Subscription Guide aisubscriptionguide.comLearn which AI subscription fits your real work.
OpenAI crawler docs platform.openai.com/docs/gptbotCheck the current OpenAI crawler and user-agent guidance.
Google crawler docs developers.google.com/search/docs/crawling-indexing/overview-google-crawlersVerify Google crawler names, indexing behavior and AI-related controls.
Anthropic support support.anthropic.com/Check current Claude account, data and crawler guidance.
FAQ
You can reduce exposure, but you cannot guarantee full control once photos are public. Keep sensitive photos private, review platform settings, avoid public high-resolution uploads and use crawler controls for your own website.
Only after removing details you do not need for the task. Names, addresses, phone numbers, birth dates, signatures, employer secrets and references can often be replaced with placeholders.
Be careful. Medical letters, passports, school documents, legal papers, bank files and family records can contain sensitive data about other people. Use approved privacy settings or do not upload them.
It depends on the product, account type and settings. Some providers offer controls, team plans or API terms that treat data differently. Check the current settings before typing sensitive information.
Not with one technical switch. robots.txt can request that compliant crawlers stay away, but it is voluntary and does not remove copies that already exist elsewhere.
robots.txt is a technical convention, not a contract by itself. It can support your policy position, but legal questions depend on jurisdiction, terms, copyright, contracts and enforcement.