# 2026-05-27
Another day, another deep dive into the fascinating world of AI crawler activity! Today, we observed a total of 649 visits to our site, with 9 active bots diligently indexing our content. It's a busy digital landscape out there!
Here's a quick look at the top bots and their activity:
| Bot Name | Visits | Unique Paths |
| Bingbot | 514 | 504 |
| Bytespider | 69 | 57 |
| ClaudeBot | 31 | 6 |
| YandexBot | 12 | 7 |
| AhrefsBot | 10 | 10 |
Bingbot continues to be the workhorse, accounting for the lion's share of visits. Bytespider also had a significant presence, followed by ClaudeBot, YandexBot, and AhrefsBot. We see the usual suspects, but there's always something to learn.
Looking at the top paths, it's clear that sitemaps are still very important. /sitemap-news.xml saw 17 visits, followed by /robots.txt with 16 and /sitemap.xml with 14. These are crucial entry points for crawlers, so we must make sure they're always in tip-top shape. We should keep an eye on these sitemap files; are there new pages that need to be added? Are there any errors preventing a crawler from properly reading them?
Interestingly, we are still seeing several bots making direct calls to the "/robots.txt" file, with 7 different bots, including Bingbot, ClaudeBot, and YandexBot, requesting the file. Also, it’s worth noting that the errors we're seeing, all reported by Bingbot, are 503 and 404 errors related to LLMs. These are crucial to address quickly, since they are negatively affecting the crawler's experience.
For a concrete actionable tip today, let's look at those errors. We need to check those pages at "/aeo/shops/ghta-tw/llms.txt", "/aeo/shops/shi-com-tw/llms.txt", and "/aeo/shops/tasty-food-tw/llms.txt" to identify and fix the 503 and 404 errors. These are impacting Bingbot, which is one of our most important crawlers. Make sure those shops are online or update the robots.txt file to prevent crawling of these pages if those shops are no longer available. Check for any recent changes that might have triggered the errors.
That's all for today's AI crawler update. Remember to stay vigilant and monitor your own site's AI traffic with AEO Scanner! You'll be surprised at what you discover.