{"@context":"https://schema.org","@type":"BlogPosting","headline":"AI 爬蟲管理：robots.txt 設定指南｜GPTBot、ClaudeBot、PerplexityBot","description":"完整的 AI 爬蟲 robots.txt 設定指南。了解 25+ 種 AI 爬蟲的 User-Agent、如何允許或封鎖特定 AI 爬蟲，以及最佳實踐建議。","url":"https://aeo.codecity.com.tw/blog/ai-crawler-robots-txt","datePublished":"2026-04-01 15:23:57","dateModified":"2026-04-01 15:23:57","inLanguage":"zh-Hant","keywords":["robots.txt","AI爬蟲","GPTBot","ClaudeBot"],"author":{"@type":"Organization","name":"AEO Scanner"},"publisher":{"@type":"Organization","name":"AEO Scanner","url":"https://aeo.codecity.com.tw"},"articleBody":"## AI 爬蟲一覽表\n\n以下是 AEO Scanner 追蹤的 25+ 種 AI 爬蟲：\n\n| 爬蟲名稱 | 組織 | User-Agent |\n|---------|------|-----------|\n| GPTBot | OpenAI | GPTBot |\n| ChatGPT-User | OpenAI | ChatGPT-User |\n| OAI-SearchBot | OpenAI | OAI-SearchBot |\n| ClaudeBot | Anthropic | ClaudeBot |\n| AnthropicBot | Anthropic | AnthropicBot |\n| PerplexityBot | Perplexity | PerplexityBot |\n| Google-Extended | Google | Google-Extended |\n| Bingbot | Microsoft | bingbot |\n| Bytespider | ByteDance | Bytespider |\n| CohereBot | Cohere | cohere-ai |\n| YouBot | You.com | YouBot |\n| BraveSearch | Brave | BraveSearch |\n| Diffbot | Diffbot | Diffbot |\n| PetalBot | Huawei | PetalBot |\n| PhindBot | Phind | Phind |\n| KagiBot | Kagi | Kagi |\n\n## robots.txt 最佳實踐\n\n### 建議：允許所有 AI 爬蟲\n```\nUser-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nSitemap: https://example.com/sitemap.xml\n```\n\n### 如果需要封鎖特定 AI\n```\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n```\n\n## 為什麼應該允許 AI 爬蟲？\n\n1. **被 AI 引用 = 免費流量**\n2. **越早被索引，優勢越大**\n3. **封鎖 AI 不會阻止 AI 提到你**（它可能從其他來源獲得資訊，但不會是第一手資訊）\n\n## 監控 AI 爬蟲活動\n\nAEO Scanner 提供 AI 爬蟲追蹤功能。掃描你的網站後，可以在儀表板查看哪些 AI 爬蟲訪問了你的網站、訪問頻率和頁面。","encoding":[{"@type":"MediaObject","encodingFormat":"text/markdown","contentUrl":"https://aeo.codecity.com.tw/blog/ai-crawler-robots-txt.md"},{"@type":"MediaObject","encodingFormat":"application/json","contentUrl":"https://aeo.codecity.com.tw/blog/ai-crawler-robots-txt.json"}]}