{"@context":"https://schema.org","@type":"BlogPosting","headline":"2026-05-18: AI Crawler Activity Report","description":"Daily AEO Scanner report on AI crawler activity. We saw 574 visits and 9 bots active today. Includes actionable tips.","url":"https://aeo.codecity.com.tw/blog/ai-diary-2026-05-18-en","datePublished":"2026-05-18 00:28:39","dateModified":"2026-05-18 00:28:39","inLanguage":"en","keywords":["AI Crawlers","Daily","SEO","AEO Scanner"],"author":{"@type":"Organization","name":"AEO Scanner Auto"},"publisher":{"@type":"Organization","name":"AEO Scanner","url":"https://aeo.codecity.com.tw"},"articleBody":"Well, another day, another look into the fascinating world of AI crawlers on AEO Scanner. Today, we observed a total of 574 visits, with 9 distinct bots actively crawling our site. It's always a bit of a dance, seeing how these digital visitors interact with our content.\n\nHere’s a quick rundown of the top 5 bots that made the most noise today:\n\n| Bot Name        | Visits | \n|-----------------|--------|\n| Bingbot         | 541    | \n| ClaudeBot       | 17     | \n| YandexBot       | 4      | \n| ChatGPT-User    | 3      | \n| OAI-SearchBot   | 3      |\n\nBingbot is, as usual, a dominant force, accounting for the vast majority of bot traffic with 541 visits. ClaudeBot, while significantly less active, still managed 17 visits, suggesting ongoing interest from that particular AI. Interestingly, we saw a small but noticeable presence from YandexBot, ChatGPT-User, and OAI-SearchBot. It's always a good idea to keep an eye on these smaller players, as their behaviors can sometimes reveal emerging trends.\n\nOne interesting pattern we saw today was the consistent interest in the `/robots.txt` file. Across multiple bots – including Bingbot, ClaudeBot, YandexBot, OAI-SearchBot, Bytespider, and Googlebot – this file was accessed multiple times. It's clear that these bots are using the `robots.txt` file to understand which parts of our site they are allowed to crawl. Another noteworthy point is that both Bingbot and ClaudeBot showed a preference for sitemap files, suggesting they are using these resources to map the website's structure.\n\nFor a concrete actionable tip today, consider this: Given the heavy use of `/robots.txt` by all these bots, make sure it’s meticulously maintained and up-to-date. Review its contents to ensure you're not inadvertently blocking any important content from being indexed by these AI crawlers. Also, if you’re using sitemaps, double-check that they are properly formatted and accessible, as they are clearly a favored tool for these bots to navigate your site. Also, if you use a hosting LLM on your site, consider a sitemap just for them as Bingbot did show interest in `/sitemap-hosted-llms-3.xml`.\n\nRemember, understanding your AI traffic is crucial for staying ahead in the ever-evolving world of SEO and content strategy. Keep monitoring your site's AI crawler activity with AEO Scanner to stay informed. ","encoding":[{"@type":"MediaObject","encodingFormat":"text/markdown","contentUrl":"https://aeo.codecity.com.tw/blog/ai-diary-2026-05-18-en.md"},{"@type":"MediaObject","encodingFormat":"application/json","contentUrl":"https://aeo.codecity.com.tw/blog/ai-diary-2026-05-18-en.json"}]}