Another day, another deep dive into the fascinating world of AI crawler activity on AEO Scanner! Today, we observed a total of 644 visits, with 5 active bots diligently crawling our site. Let's break down the key observations from the past 24 hours.
Here's a quick look at the top 5 bots and their visit counts:
| Bot Name | Visits |
| Bingbot | 613 |
| ClaudeBot | 22 |
| OAI-SearchBot | 4 |
| YandexBot | 4 |
| Googlebot | 1 |
Bingbot continues to dominate the scene, accounting for the vast majority of our bot traffic with 613 visits. Its activity is focused on indexation, as we can see it hitting /robots.txt repeatedly with 3 visits, alongside other sitemap files: /sitemap.xml (2 visits), and /sitemap-static.xml (2 visits). ClaudeBot's presence is also notable, showing a more varied crawling pattern. /sitemap-news.xml (8 visits) and /sitemap.xml (7 visits) are clearly on their radar.
Looking at the top paths overall, the /robots.txt file stands out, visited 12 times by 4 bots. The sitemap files are also heavily accessed, indicating that bots are actively trying to understand the site structure. We also see evidence of previous day's diary being accessed: the path /en/blog/ai-diary-2026-05-10-en got 3 visits by 2 bots.
Interestingly, we spotted a 404 error from Bingbot: "/aeo/shops/xn-7-bg0b367bti1c-tw/llms.txt". This suggests a potential misconfiguration or a broken link related to a shop or LLM. This warrants further investigation to ensure we're not inadvertently blocking access or misleading the bots. Further, the hourly distribution indicates a relatively steady flow of traffic, with peaks around 05:00 (37 visits) and 22:00 (37 visits). It might be worth investigating why these hours are more active.
For an actionable GEO/AEO tip, based on the 404 error, we recommend reviewing the shop configurations and link structures on your site, particularly around the xn-7-bg0b367bti1c-tw shop and anything related to LLMs. Ensure that all relevant pages are accessible and properly linked to, and that the /robots.txt file doesn't inadvertently block essential content.
Remember, understanding your AI crawler traffic is key to optimizing your site for search and ensuring that your valuable content is being discovered. Keep an eye on your site's AI activity with AEO Scanner!