Snippet builder
Use these examples as a starting point, not as a permanent truth. User-agent names, products and crawler behavior change, so always check official pages before deploying.
Static tool
Select the crawlers you want to request not to crawl. This generator runs in your browser only. It does not send data anywhere. Before deploying, verify the current official user-agent names.
# AI crawler controls: verify current user-agent names before deploying.
# This is voluntary. robots.txt is not a legal lock.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /Warning: blocking user-triggered or search-related agents can reduce AI-assisted discovery, previews or citations.
Copy-paste block
Below is a starter block covering several AI-related crawlers. User-agent names change and vary by product, so treat this as a template and verify each name against the operator's official documentation before you deploy it.
# Example only. Verify every user-agent name against official docs.
# robots.txt is voluntary. It does not remove copies that already exist,
# and it only affects crawlers that choose to obey it.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Applebot-Extended
Disallow: /| User-agent (verify) | Associated with | Typically used for |
|---|---|---|
| GPTBot | OpenAI | Crawling web content that may be used for model training |
| Google-Extended | A control token to limit use of content for certain AI products | |
| CCBot | Common Crawl | Building the open Common Crawl archive used by many downstream datasets |
| ClaudeBot / anthropic-ai | Anthropic | Crawling related to Claude; verify current tokens in Anthropic docs |
| PerplexityBot | Perplexity | Crawling for its answer and search features |
| Bytespider | ByteDance | Broad crawling associated with ByteDance products |
| Applebot-Extended | Apple | A control token to limit use of content for Apple AI training |
Some of these are true crawler user-agents and others are control tokens rather than bots that fetch pages. Names, meanings and behavior change, so confirm each one on the operator's official documentation before relying on it.
What it does not do
robots.txt is a convention. Well-behaved crawlers read it and comply, but it is not enforced by the web itself. A crawler that ignores it, or that is not on your list, can still fetch public pages. It is a signal of intent, not a technical barrier.
Adding a disallow rule today does nothing about content that was already crawled, cached, archived or copied elsewhere. Nor does it affect data people paste into AI tools by hand. Think of it as reducing future exposure, not undoing the past.
Stronger layers
A web application firewall or server rule can actually block or challenge requests by user-agent, IP range or behavior, rather than just asking politely. This is enforcement rather than a request, though it needs maintenance as crawler identities change and can occasionally catch legitimate visitors.
Page-level directives such as a noindex robots meta tag or X-Robots-Tag header influence how compliant crawlers treat a page. Like robots.txt, they rely on cooperation and do not stop a crawler that ignores them.
Requiring a login, rate-limiting, or putting sensitive material behind authentication keeps it off open, crawlable pages entirely. Gating access is usually far more reliable than any voluntary text file for content that truly must stay private.
Verify it works
https://yourdomain.com/robots.txt directly in a browser and confirm the file is served correctly at the site root.This page is educational information, not legal advice, and user-agent names shown here may be out of date. Always confirm the current tokens and behavior on the operator's official documentation before relying on them.
Related reading
Compare AI models before choosing a workflow.
ChatGPT Alternatives chatgpt-alternatives.comFind neutral alternatives when one AI provider is not enough.
AI Subscription Guide aisubscriptionguide.comLearn which AI subscription fits your real work.
OpenAI crawler docs platform.openai.com/docs/gptbotCheck the current OpenAI crawler and user-agent guidance.
Google crawler docs developers.google.com/search/docs/crawling-indexing/overview-google-crawlersVerify Google crawler names, indexing behavior and AI-related controls.
Anthropic support support.anthropic.com/Check current Claude account, data and crawler guidance.
FAQ
You can reduce exposure, but you cannot guarantee full control once photos are public. Keep sensitive photos private, review platform settings, avoid public high-resolution uploads and use crawler controls for your own website.
Only after removing details you do not need for the task. Names, addresses, phone numbers, birth dates, signatures, employer secrets and references can often be replaced with placeholders.
Be careful. Medical letters, passports, school documents, legal papers, bank files and family records can contain sensitive data about other people. Use approved privacy settings or do not upload them.
It depends on the product, account type and settings. Some providers offer controls, team plans or API terms that treat data differently. Check the current settings before typing sensitive information.
Not with one technical switch. robots.txt can request that compliant crawlers stay away, but it is voluntary and does not remove copies that already exist elsewhere.
robots.txt is a technical convention, not a contract by itself. It can support your policy position, but legal questions depend on jurisdiction, terms, copyright, contracts and enforcement.