Well, another day, another deluge of AI bots crawling around AEO Scanner! Today, we saw a total of 1362 visits, with 7 active bots keeping things interesting. Let's dive into the details.
First, a quick overview of the top bots by visit count:
| Bot Name | Visits |
| YandexBot | 730 |
| Bytespider | 516 |
| AhrefsBot | 54 |
| ClaudeBot | 49 |
| Bingbot | 10 |
YandexBot continues to dominate the scene with a whopping 730 visits. Bytespider followed with 516 visits, consistently indexing a significant portion of our content. We also observed a few interesting patterns. The /robots.txt file continues to be a popular target, seeing a total of 18 visits across 7 bots. This indicates bots are actively checking our instructions, which is expected. The sitemap files (/sitemap.xml and /sitemap-news.xml) are also highly sought after, with ClaudeBot focusing on those quite a bit, suggesting it uses our sitemaps extensively for content discovery.
The most interesting thing is the 10 instances of 503 errors. Bytespider and AhrefsBot seem to be particularly struggling to access the /aeo/shops/ pages to get the llms.txt files. Specifically, /aeo/shops/sites-google-com/llms.txt, /aeo/shops/whsh-com-tw/llms.txt, /aeo/shops/ef2-com-tw/llms.txt, /aeo/shops/homepage-ntu-edu-tw/llms.txt, /aeo/shops/metr-com-tw/llms.txt, /aeo/shops/pcchh-com-tw/llms.txt, /aeo/shops/shop-unclesweet-com/llms.txt and /aeo/shops/tachungho-com/llms.txt all got the 503 error, all caused by Bytespider. We need to investigate why these specific pages are causing issues for these crawlers. This might indicate an issue with these external sites themselves that will eventually affect our AEO-related operations.
Here’s a practical AEO tip based on today’s observations: Since many bots are looking for LLM-related content, make sure your robots.txt file is set up correctly and your sitemaps are up-to-date, especially if you are using geo-specific content or shops. Double-check the shop pages Bytespider is struggling to access. Fixing those might lead to greater content discovery.
That's all for today's report. Remember to keep an eye on your own site's AI traffic with AEO Scanner to stay ahead of the curve!