How to Audit robots.txt for AI Crawlers Without Blocking Search Engines

Executive Summary

Managing automated web crawling policies has evolved from a simple search engine optimization administrative task into a core challenge of modern technical architecture, intellectual property governance, and brand discoverability. For nearly three decades, the Robots Exclusion Protocol served a singular, predictable objective: it informed traditional web crawlers such as Googlebot and Bingbot which directories were safe to traverse and index for standard organic search results.

In the modern web ecosystem, that simplicity has completely broken down. Websites today are visited by a fragmented landscape of autonomous systems. These include traditional search engine indexers, real-time retrieval-augmented generation (RAG) search agents such as ChatGPT Search, PerplexityBot, and Claude-Web, and aggressive offline training scrapers such as GPTBot, ClaudeBot, and Bytespider that harvest billions of tokens to train commercial foundation models.

Organizations now face a difficult technical dilemma. If you block crawlers aggressively using blanket wildcard rules, you inadvertently sever your discovery pipeline in traditional search engines and conversational AI answer engines. If you leave your infrastructure completely open, your proprietary data, documentation, and original research are extracted wholesale without attribution, compensation, or computational rate limits.

The purpose of this guide is to provide an exhaustive, engineer-to-engineer technical blueprint for performing a comprehensive robots.txt audit for AI crawlers. By mastering the formal IETF RFC 9309 specification, understanding the mechanics of modern crawler token differentiation, preventing headless browser rendering failures, and integrating simulation infrastructure like OllaGraph, technical teams can establish an airtight content governance perimeter that blocks unwanted model scrapers while ensuring flawless discovery across global search engines. To measure how your crawl policies affect downstream AI citation rates, pair this audit with our frameworks on measuring AI Search Visibility Score and optimizing your Citation Readiness Score, or test for post-deployment crawl bugs using AI crawler regression testing.

Key Takeaways

• Three Operational Tiers: Automated web agents are divided into three distinct operational tiers: traditional search indexers (Googlebot, Bingbot), real-time conversational search agents (OAI-SearchBot, PerplexityBot, ChatGPT-User), and offline model training scrapers (GPTBot, ClaudeBot, Bytespider, CCBot). Policy decisions must treat these tiers independently.
• Single Group Matching (RFC 9309): The Internet Engineering Task Force (IETF) RFC 9309 standard enforces a strict “Single Group Matching” rule. When a crawler finds a user-agent block specifically matching its token, it completely ignores the generic wildcard group (User-agent: *). This creates severe security and privacy vulnerabilities if disallow rules are not duplicated across all specific blocks.
• Render Dependency Protection: Headless browser rendering engines require unimpeded access to JavaScript bundles, CSS stylesheets, layout assets, and public client API endpoints. Blocking asset paths in an attempt to protect data results in blank DOM trees, broken Core Web Vitals, and total de-indexing across both search engines and AI citation engines.
• Google-Extended Disconnect: Google-Extended is strictly a governance token for Gemini and Vertex AI training pipelines as documented in Google Search Central. Disallowing Google-Extended has zero negative effect on traditional Google Search indexation and does not remove your content from Google Search AI Overviews.
• Edge WAF Precedence: Edge Web Application Firewalls (such as Cloudflare, Fastly, or AWS WAF) that employ broad “Block AI Scrapers” toggles operate at Layer 7 before robots.txt is ever evaluated. If a WAF drops an AI search bot with a 403 Forbidden status, your robots.txt allow rules are rendered completely useless.
• Programmatic Simulation: Accurate, reliable verification requires programmatic simulation tools. Running live crawler simulations with platforms like OllaGraph allows engineering teams to verify how headless browsers and answer engines render, fetch, and extract text from production endpoints before deploying policy changes.

1. Problem Statement

The central technical challenge facing engineering and SEO teams today is the collision between legacy crawling protocol assumptions and modern AI data ingestion workflows.

When a system administrator, DevOps engineer, or technical lead attempts to prevent artificial intelligence companies from scraping their web property, the instinctive reaction is often to deploy a broad disallow rule across the entire domain:

User-agent: *
Disallow: /

When teams realize this eliminates their organic search traffic overnight, they attempt to create layered exceptions for specific bots. However, because robots.txt parsing is governed by the strict longest-match and single-group selection mechanics of RFC 9309, these layered configurations frequently introduce subtle, catastrophic failures.

The Four Primary Failure Modes

1. Search Visibility Degradation: Search engines like Googlebot and Bingbot no longer rely solely on static HTML parsing. They run modern headless rendering engines that execute JavaScript, apply CSS stylesheets, and calculate layout geometry. When an engineer blocks paths such as /_next/, /dist/, /assets/, or /api/ in an attempt to protect proprietary resources, the crawler’s headless browser fails to construct the DOM tree. The search engine perceives the page as blank or broken, resulting in dropped search rankings, lost rich snippets, and degraded Core Web Vitals.
2. Conversational Answer Engine Disappearance: Many organizations fail to distinguish between training dataset harvesters and real-time retrieval agents. By treating all automated agents from OpenAI, Anthropic, or Perplexity as scrapers and disallowing user agents such as OAI-SearchBot, PerplexityBot, and ChatGPT-User, the website completely disappears from real-time answer engines. When a user asks an AI assistant for product recommendations, software comparisons, or technical documentation summaries, the assistant cannot retrieve the live URL and cites a competitor instead.
3. Accidental Directory Exposure Through Specificity Fall-Through: Due to the single-group matching mechanics defined in RFC 9309, defining a rule block for a specific user agent causes that crawler to ignore all directives in the generic wildcard group (User-agent: *). If an engineer allows /blog/ for a specific bot without re-specifying disallow rules for /admin/, /checkout/, or /internal-api/, those sensitive endpoints become fully accessible to that specific crawler.
4. Edge Network and WAF Blind Spots: Content Delivery Networks (CDNs) and Web Application Firewalls (WAFs) increasingly provide one-click managed rules to block automated AI scrapers. These security layers inspect incoming HTTP headers at the edge and reject requests with a 403 Forbidden or 429 Too Many Requests response before the traffic reaches the origin server. As a result, the crawler never reads the carefully designed robots.txt file, and engineering teams remain unaware that their search and citation discoverability is broken.

2. History

The Robots Exclusion Protocol originated in February 1994, created by Dutch software engineer Martijn Koster while managing the Nexor search engine. In the early days of the World Wide Web, automated crawlers (known as “web wanderers” or “spiders”) frequently overwhelmed web servers by opening dozens of concurrent connections, creating severe denial-of-service conditions. Koster proposed a simple, cooperative text file placed at the root of a domain (/robots.txt) to establish voluntary boundaries for automated crawlers.

For twenty-five years, the protocol existed purely as an informal consensus. Different search engine vendors developed proprietary parsing extensions and interpretations:

• In the early 2000s, Google introduced the Allow: directive and pattern-matching wildcards (* and $).
• Other search engines implemented varying interpretations of path precedence, whitespace handling, and character encoding.
• Crawlers adopted differing fallback behaviors when encountering HTTP redirect chains, 4xx client errors, or 5xx server errors during robots.txt retrieval.

In September 2019, Google initiated a project to standardize the protocol through the Internet Engineering Task Force (IETF). This effort culminated in September 2022 with the formal publication of RFC 9309, titled “Robots Exclusion Protocol.” RFC 9309 standardized syntax, character limits, record separation, longest-match precedence rules, and established a maximum file size limit of 512 KiB.

Between 2022 and 2026, the rapid growth of generative Large Language Models (LLMs) fundamentally disrupted web crawling dynamics. In August 2023, OpenAI introduced GPTBot, providing site owners with a dedicated token to control training data harvesting. Soon after, Anthropic launched ClaudeBot, Google released Google-Extended, ByteDance deployed Bytespider, and Common Crawl updated CCBot.

As AI systems evolved from static models into real-time retrieval-augmented generation (RAG) search interfaces, AI companies began splitting their crawler identities:

• OpenAI established OAI-SearchBot for real-time web search and link citations, while retaining GPTBot for offline training ingestion, and ChatGPT-User for live user-prompted browsing actions.
• Anthropic established Claude-Web for real-time browsing alongside ClaudeBot for model training.
• Apple introduced Applebot-Extended to govern Apple Intelligence model training without affecting standard Applebot search indexation.

Today, robots.txt is no longer merely a crawl-budget optimization tool; it serves as a critical policy boundary governing intellectual property, data licensing, and search visibility.

3. Definition

A robots.txt audit for AI crawlers is a comprehensive technical evaluation of an organization’s crawl policy infrastructure. The audit verifies three distinct outcomes:

1. Traditional search engine crawlers (such as Googlebot, Bingbot, and Applebot) retain unrestricted access to all public content, dynamic client assets, and rendering resources.
2. Real-time conversational search agents and RAG retrieval bots (such as OAI-SearchBot, PerplexityBot, and Claude-Web) are permitted or restricted according to the organization’s deliberate brand visibility strategy.
3. Automated model training scrapers and commercial bulk harvesters (such as GPTBot, ClaudeBot, Bytespider, and CCBot) are strictly governed without creating specificity fall-through vulnerabilities or breaking headless DOM rendering.

The audit encompasses the entire request and delivery pipeline, including DNS resolution, CDN and WAF edge security rules, HTTP header responses, character encoding, RFC 9309 directive precedence, client-side rendering dependencies, and extraction fidelity.

4. Architecture

A modern crawl governance architecture operates across five distinct functional layers. Each layer must be audited independently to ensure consistent policy enforcement.

Layer 1: DNS & Anycast Routing
      │
      ▼
Layer 2: Edge CDN & WAF (Cloudflare, Fastly, AWS WAF)
      │
      ▼
Layer 3: Origin HTTP Delivery (/robots.txt, text/plain, RFC 9309 < 512 KiB)
      │
      ▼
Layer 4: Crawler Policy Evaluation (Single Group Matching + Longest-Match)
      │
      ▼
Layer 5: Headless Browser Rendering & DOM Extraction (Chromium / V8 Engine)

• Layer 1: DNS and Network Routing: When a crawler initiates a visit, it resolves the domain’s A, AAAA, and CNAME records. Any misconfigured geo-DNS routing, regional IP blacklisting, or Anycast routing failures will terminate the connection before an HTTP handshake is established.
• Layer 2: CDN and Edge Web Application Firewall (WAF): Incoming HTTP requests reach edge infrastructure. The WAF evaluates User-Agent headers, TLS fingerprints (JA3/JA4), IP reputation, and managed rules like “Block AI Scrapers”. If triggered, the edge drops the connection with 403 or 429 before robots.txt is fetched.
• Layer 3: Origin HTTP Delivery: The origin web server serves /robots.txt with HTTP 200 OK, Content-Type: text/plain; charset=utf-8, and size under the 512 KiB RFC 9309 threshold.
• Layer 4: Crawler Policy Evaluation: The crawler executes Single Group Matching to find its specific token and evaluates target URLs using longest-match precedence.
• Layer 5: Headless Browser Rendering and Extraction: Permitted crawlers initialize headless Chromium to construct DOM trees, requesting JS bundles, CSS stylesheets, client fonts, and API hydration endpoints.

5. Internal Working

Understanding the internal parsing mechanics defined in RFC 9309 is essential for diagnosing unexpected crawl behaviors.

The Single Group Matching Algorithm

When an automated crawler reads a robots.txt file, it processes the text through a deterministic selection sequence:

1. The crawler normalizes its own user-agent product token to lowercase characters (for example, OAI-SearchBot becomes oai-searchbot).
2. It scans the robots.txt file line by line to identify all User-agent: declarations.
3. It checks for an exact, case-insensitive substring match between its product token and the declared user-agent lines.
4. The Critical Rule: If one or more groups specifically match its token, the parser binds exclusively to those matching groups. It completely ignores all other groups, including the global User-agent: * block.
5. If no specific match is found, the parser binds to the generic User-agent: * wildcard group.
6. If no generic wildcard group is present, the crawler assumes it has unrestricted access to crawl the entire domain.

Path Resolution and Longest-Match Precedence Logic

Once the active user-agent group is established, the parser tests the requested URL path against the Allow: and Disallow: directives in that group:

• The parser normalizes both the directive path and the target URL by decoding unreserved percent-encoded octets.
• It performs a prefix match starting from the root slash (/).
• If multiple directives in the active group match the target URL path, the parser calculates the length of each matching pattern in characters (octets).
• Longest Pattern Wins: The directive with the greatest number of matching characters determines the access policy.
• Tie-Breaking Rule: If an Allow: directive and a Disallow: directive match the exact same number of characters on a given URL path, the Allow: directive takes precedence.

Wildcard and End-of-String Mechanics

RFC 9309 standardizes two special pattern characters:

• The asterisk (*) represents zero or more occurrences of any valid URI character.
• The dollar sign ($) anchors the match strictly to the end of the URL path string.

Consider this rule set:

User-agent: *
Disallow: /private$
Allow: /private/

In this scenario, a request to https://example.com/private matches Disallow: /private$ (8 characters) and is blocked. A request to https://example.com/private/team matches Allow: /private/ (9 characters) and is permitted.

6. Components

A fully compliant robots.txt file is constructed from five core structural components:

1. User-Agent Declarations: Identifies the crawler identity to which a group applies. Consecutive lines stack together:

   User-agent: OAI-SearchBot
   User-agent: PerplexityBot
   Allow: /
   

2. Disallow Directives: Defines forbidden path prefixes. An empty disallow line (Disallow:) grants unrestricted access.
3. Allow Directives: Explicitly permits access to a specific path prefix or wildcard pattern within an otherwise restricted hierarchy.
4. Sitemap Declarations: Global directives providing fully qualified absolute URLs to XML sitemaps:

   Sitemap: https://www.example.com/sitemap.xml
   

5. Comments: Prefixed by #, ignored by compliant parsers. Use them to document the governance rationale.

7. Workflow

A comprehensive audit follows an eight-stage engineering workflow:

1. Route Discovery & Architecture Inventory: Compile domain variants, subdomains (docs.example.com, api.example.com), and path structures.
2. Crawler Identity & Policy Classification: Classify Tier 1 (Search), Tier 2 (AI Search / RAG), and Tier 3 (Training Scrapers).
3. Direct HTTP & Delivery Validation: Verify HTTP 200 OK, text/plain; charset=utf-8, and sub-512 KiB payload size.
4. WAF & Edge Firewall Inspection: Inspect edge security logs for unintended 403/429 status drops on legitimate AI search bots.
5. Syntax & Specificity Verification: Confirm specific bot blocks duplicate sensitive directory disallow rules.
6. Asset & Render Dependency Validation: Ensure CSS, JS, fonts, and client-facing API endpoints are accessible for headless rendering.
7. Real-Time Crawl Simulation: Simulate crawler execution using tools like Ollagraph to confirm DOM hydration and text extraction.
8. Production Deployment & Observability: Deploy via CI/CD and monitor crawler status codes in access logs.

8. Configuration

Strategy 1: AI Search Allowed, Foundation Training Blocked (Recommended Standard)

Permits traditional search engines and real-time AI answer engines to index and cite public content, while blocking bulk foundation model training scrapers:

# STRATEGY 1: AI SEARCH ALLOWED, TRAINING SCRAPERS BLOCKED

# Core Traditional Search Engines
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Bingbot
User-agent: Slurp
User-agent: DuckDuckBot
User-agent: Baiduspider
User-agent: YandexBot
Allow: /
Disallow: /admin/
Disallow: /checkout/
Disallow: /auth/
Disallow: /api/internal/
Disallow: /customer-portal/

# AI Real-Time Search & Retrieval Engines (Allowed for Citations)
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-Web
Allow: /
Disallow: /admin/
Disallow: /checkout/
Disallow: /auth/
Disallow: /api/internal/
Disallow: /customer-portal/

# AI Foundation Model Training Scrapers (Blocked)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: Amazonbot
User-agent: Diffbot
User-agent: Scrapy
Disallow: /

# Global Fallback for Unmatched Crawlers
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/
Disallow: /auth/
Disallow: /api/internal/
Disallow: /customer-portal/

Sitemap: https://www.example.com/sitemap.xml

Strategy 2: Maximum Discoverability (Full Search and Training Allowed)

Designed for open-source repositories and public documentation platforms:

# STRATEGY 2: MAXIMUM DISCOVERABILITY

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /auth/
Disallow: /api/internal/
Disallow: /private/
Disallow: /*?*sort=
Disallow: /*?*filter=

# Explicit asset access for headless rendering
Allow: /assets/
Allow: /static/
Allow: /_next/static/
Allow: /dist/

Sitemap: https://www.example.com/sitemap.xml

Strategy 3: Granular Subdirectory and Knowledge Shielding

Allows AI search engines access to docs and blogs while protecting proprietary customer forums and raw downloads:

# STRATEGY 3: GRANULAR KNOWLEDGE SHIELDING

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
Allow: /docs/
Allow: /blog/
Allow: /help/
Disallow: /community/threads/
Disallow: /whitepapers/raw-pdf/
Disallow: /customer-data/
Disallow: /

User-agent: *
Allow: /
Disallow: /community/threads/
Disallow: /whitepapers/raw-pdf/
Disallow: /customer-data/

Sitemap: https://www.example.com/sitemap.xml

9. Examples

Example 1: Testing Crawler Headers and Status Codes via cURL

# Check status code, redirects, and content type
curl -s -i -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://www.example.com/robots.txt | head -n 15

# Verify behavior across different AI user-agent headers
curl -s -o /dev/null -w "OAI-SearchBot Status: %{http_code}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://www.example.com/robots.txt

curl -s -o /dev/null -w "GPTBot Status: %{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://www.example.com/robots.txt

Example 2: Verifying Crawler IP Authenticity with Reverse DNS

# Step 1: Perform reverse DNS lookup on the crawler's IP
host 66.249.66.1
# Output: 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

# Step 2: Perform forward DNS lookup on the resulting hostname
host crawl-66-249-66-1.googlebot.com
# Output: crawl-66-249-66-1.googlebot.com has address 66.249.66.1

10. Performance

• Mitigating Origin Server Overload: High-frequency training scrapers like Bytespider and CCBot can generate significant request volumes. An explicit Disallow: / directive causes compliant scrapers to abort crawls immediately, saving CPU and database resources.
• Edge Caching of robots.txt: Because every compliant crawler requests /robots.txt before visiting pages, cache it at the edge with a 1-hour TTL:

  Cache-Control: public, max-age=3600, stale-while-revalidate=86400
  

• Rendering Overhead: If your server takes more than 1.5 seconds to return critical JavaScript bundles, headless browsers may time out, resulting in incomplete indexation and missing citations.

11. Security

A robots.txt file is a public document that provides instructions to cooperative software; it is not an access control system.

# SECURITY VULNERABILITY: Never list secret routes in robots.txt
User-agent: *
Disallow: /admin-secret-portal-v2/
Disallow: /internal-database-backups/

• Proper Access Control: Enforce authentication at the network or application layer (OAuth, SSO, API tokens), returning 401 Unauthorized or 403 Forbidden for unauthenticated requests.
• Preventing Content Poisoning: Ensure your robots.txt file cannot be modified dynamically by untrusted user inputs, file upload endpoints, or misconfigured reverse proxies.

12. Troubleshooting

Symptom	Underlying Cause	Resolution
Google Search Console: “Blocked by robots.txt”	A Disallow rule matches intended URLs (e.g. Disallow: /blog blocking /blog/my-post).	Add precise trailing slashes (Disallow: /blog/) or an explicit Allow: /blog/ directive.
Indexed by Google but without text snippets	Disallowing a page that has external inbound links prevents Googlebot from reading content or on-page noindex tags.	Remove Disallow rule and place <meta name="robots" content="noindex"> directly in the page HTML <head>.
ChatGPT / Perplexity do not cite your site	OAI-SearchBot, ChatGPT-User, or PerplexityBot are disallowed, or an edge WAF is blocking them with HTTP 403.	Allow these user agents in robots.txt and whitelist their bot tokens in CDN/WAF security settings.

13. Best Practices

• Keep the File Lean: Keep size well below 64 KiB at the domain root (/robots.txt).
• Explicitly Match Trailing Slashes: Append trailing slashes (/) when disallowing full directories.
• Separate Search from Training: Maintain distinct rule blocks for search engines, AI search engines, and model training scrapers.
• Duplicate Sensitive Rules: Repeat sensitive directory disallows across every custom user-agent block.
• Always Allow Rendering Assets: Ensure CSS, JS, fonts, and client APIs are accessible to permitted bots to maintain high Citation Readiness Scores.
• Declare XML Sitemaps: Place absolute sitemap URLs at the bottom of the file.
• Simulate Before Deploying: Test crawler rendering and extraction using OllaGraph.
• Monitor AI Search Impact: Track whether your crawl policies boost AI answer citations with the AI Search Visibility Score framework.
• Explore Topic Archives: Review our robots.txt category guides and AEO knowledge base for ongoing updates.

14. Common Mistakes

• Case-Insensitive Assumptions: RFC 9309 path matching is case-sensitive. Disallow: /admin/ does not block /Admin/ or /ADMIN/. Normalize URLs to lowercase on your server.
• Unsupported Directives: Crawl-delay: and Noindex: are not part of RFC 9309. Googlebot ignores both directives inside robots.txt.
• Blank Lines Inside Records: Blank lines indicate the termination of a record. Avoid blank lines between User-agent: lines and their associated Disallow: / Allow: rules.

15. Alternatives

• On-Page HTML Meta Robots Tags:
  • <meta name="robots" content="noindex, follow"> — Prevents indexing while following links.
  • <meta name="robots" content="nosnippet"> — Disables text snippet previews and AI Overviews.
  • <meta name="robots" content="noai, noimageai"> — Signals that content must not be used for AI training.
• HTTP Response Headers (X-Robots-Tag):

  HTTP/1.1 200 OK
  Content-Type: application/pdf
  X-Robots-Tag: noindex, noai
  

• The llms.txt Standard: Provides an /llms.txt index file containing curated Markdown documentation optimized for LLM context windows.

16. Comparison Tables

Dimension	Core Traditional Search	Real-Time AI Search Engines	Offline Model Training Scrapers
Key User-Agents	Googlebot, Bingbot, Applebot	OAI-SearchBot, PerplexityBot, Claude-Web	GPTBot, ClaudeBot, Bytespider, CCBot
Operational Purpose	Web indexing for SERP rankings	Real-time RAG retrieval & citations	Bulk token ingestion for foundation models
Headless Rendering	Full Chromium rendering	Dynamic DOM hydration & text extraction	Raw text / HTML scraping
Impact of Disallow	Complete de-indexing from search engines	Exclusion from AI answer cards & citations	Protects IP & prevents uncompensated training
Recommended Policy	Allow: /	Allow: / (with sensitive path restrictions)	Disallow: /

17. Enterprise Deployment

1. Infrastructure as Code (IaC): Store robots.txt in Git with mandatory pull request approvals from both SEO and DevOps leads.
2. Automated CI/CD Validation: Run pipeline tests verifying that Googlebot is never blocked and total file size is well under 512 KiB.
3. Simulation with OllaGraph: Use OllaGraph API infrastructure to simulate how AI search agents and headless crawlers parse staging routes before production releases.
4. End-to-End Governance: Combine robots.txt auditing with our Citation Readiness Scoring CI gate and AISVS analytics.

18. Cloud Deployment

AWS CloudFront Configuration

When serving Single Page Apps from S3 via CloudFront, create an explicit Cache Behavior for /robots.txt pointing to static storage with a minimum TTL of 3600 seconds.

Cloudflare Edge Worker Deployment

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);

    if (url.pathname === "/robots.txt") {
      const robotsContent = `User-agent: Googlebot
Allow: /
Disallow: /admin/

User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/
Allow: /static/

Sitemap: https://www.example.com/sitemap.xml`;

      return new Response(robotsContent, {
        headers: {
          "Content-Type": "text/plain; charset=utf-8",
          "Cache-Control": "public, max-age=3600, stale-while-revalidate=86400",
          "X-Robots-Tag": "noindex"
        }
      });
    }

    return fetch(request);
  }
};

19. Frequently Asked Questions (FAQs)

Q1: Will blocking GPTBot harm my website’s Google search rankings?

No. GPTBot is operated exclusively by OpenAI for foundation model training. Google’s ranking algorithms rely entirely on Googlebot.

Q2: What is the exact difference between Googlebot and Google-Extended?

Googlebot is the search crawler responsible for indexing pages for Google Search. Google-Extended is a governance token used to opt out of training Google’s Gemini models and Vertex AI APIs. Blocking Google-Extended has zero impact on Google Search rankings or AI Overviews.

Q3: Why does my site appear in search results even after adding a Disallow rule?

A Disallow rule prevents crawling, but if external websites link to that URL, search engines can still index the URL. To remove it completely, allow crawling and place <meta name="robots" content="noindex"> in the HTML <head>.

Q4: Does PerplexityBot train foundation models or provide live search citations?

PerplexityBot functions primarily as an indexer and real-time retrieval agent for Perplexity search. Allowing it enables your site to appear as a cited source in answer cards.

Q5: How do I know if an Edge WAF is blocking legitimate AI crawlers?

Inspect your CDN firewall event logs (such as Cloudflare Security Analytics or AWS WAF logs) for 403 Forbidden responses on user-agent strings like OAI-SearchBot or PerplexityBot.

20. References

• RFC 9309 (Robots Exclusion Protocol): Koster, M., Illyes, G., Zeller, H., and Sassman, M. (September 2022). Internet Engineering Task Force (IETF). RFC 9309 Specification
• Google Search Central Documentation: Official guide on robots.txt specifications and crawler directives. Google Search Central robots.txt Overview and Google-Extended Documentation.
• OpenAI Crawler Documentation: User-agent verification, IP ranges, and operational differentiation between GPTBot, OAI-SearchBot, and ChatGPT-User. OpenAI Bots Documentation.
• Anthropic Web Crawler Documentation: Operating policies and robots.txt directives for ClaudeBot and Claude-Web. Anthropic Crawler Guide.
• Perplexity Crawler Documentation: Verification and indexing guidelines for PerplexityBot. Perplexity Documentation.
• OllaGraph Web Intelligence & AEO Platform: Headless crawler simulation, live SERP aggregation, and structured content audit workflows. OllaGraph Documentation.

21. Conclusion

Managing automated crawl traffic is no longer a simple binary choice between complete open access and total exclusion. The rise of conversational answer engines and large-scale model training has made crawl policy design a critical component of technical architecture and brand strategy.

Achieving the right balance requires a disciplined, multi-layered approach:

1. Separate by Intent: Welcome traditional search indexers and conversational AI search agents to maximize your public visibility, while using targeted directives to govern bulk foundation model scrapers.
2. Adhere to RFC 9309: Avoid specificity fall-through vulnerabilities by ensuring custom user-agent blocks duplicate necessary security restrictions, and always allow full access to CSS, JS, and API assets required for headless rendering.
3. Validate with Active Testing: Use live simulation platforms like Ollagraph and monitor production logs to ensure your web infrastructure remains fully discoverable across global search engines while maintaining complete control over your proprietary data.