Another day, another mountain of data from the AI crawler activity on AEO Scanner. We saw a total of 930 visits today, with 10 bots actively crawling our site. Let's dive into the details.
Here's a quick look at the top 5 bots by visit count:
| Bot Name | Visits | Unique Paths |
| AhrefsBot | 557 | 556 |
| Bingbot | 310 | 304 |
| ClaudeBot | 44 | 6 |
| Bytespider | 6 | 4 |
| YandexBot | 5 | 5 |
ClaudeBot continues to show interesting behavior. While its overall visit count (44) is significantly lower than AhrefsBot and Bingbot, it's focused on specific sitemap files. It made 19 visits to /sitemap-news.xml and 15 visits to /sitemap.xml. This targeted approach suggests a focus on news-related content, likely for indexing and analysis purposes. It also hit our robots.txt, 7 times.
Bytespider, with a smaller overall presence (6 visits), also warrants attention. It seems to be specifically interested in our feed data and our LLM access points, as reflected by its crawl of /aeo/shops/sites-google-com/llms.txt. Interestingly, a new bot, ExaBot, appeared today with 2 visits to /robots.txt. This highlights the ever-evolving landscape of AI crawlers and the need to stay vigilant.
Looking at the error logs, we find some specific issues. Bytespider reported a 503 error (2 times) while accessing /aeo/shops/sites-google-com/llms.txt. AhrefsBot got a 503 error (1 time) for /aeo/shops/petzoo-com-tw/llms.txt, and also a 404 for /aeo/shops/xn-7-bg0b367bti1c-tw/llms.txt. It seems we need to verify the availability and access permission to these paths.
For a concrete GEO/AEO tip, based on today’s data: Since both ClaudeBot and Bytespider are hitting our sitemap and LLM access points, review your sitemap.xml and robots.txt files. Make sure they accurately reflect the content you want indexed and the resources you want to be accessible to these specific crawlers. Ensure that your shop data paths are accessible and return the proper response codes. If you are serving different content to different bots, make sure the LLM access points return the correct status codes.
Remember to stay informed. Keep an eye on your site's AI traffic with AEO Scanner to identify trends and adapt your AEO strategies accordingly!