Well, another day, another look into the fascinating world of AI crawlers and their interactions with AEO Scanner. Today, we observed a total of 583 visits, with 8 active bots diligently exploring our site. It’s always interesting to see which bots are the most active and what they are prioritizing. Let’s dive into the specifics.
Here’s a quick rundown of the top 5 bots by visit count:
| Bot Name | Visits |
| Bingbot | 552 |
| ClaudeBot | 15 |
| YandexBot | 6 |
| OAI-SearchBot | 3 |
| ChatGPT-User | 2 |
Bingbot continues to dominate, accounting for a significant portion of our traffic. Its focus seems heavily skewed towards sitemap files, suggesting a strong interest in understanding our site structure. We're seeing a consistent interest in /robots.txt across multiple bots, which isn’t surprising. It’s a crucial file for any site.
Looking at some specific patterns, it's clear that the sitemap files are popular. Both Bingbot and ClaudeBot are heavily focused on these, with multiple visits to various sitemap versions (sitemap-static.xml, sitemap-site-profiles-2.xml, sitemap-hosted-llms-1.xml, and more). It's also worth noting the presence of ChatGPT-User and their specific path requests, like "/site/7288" and "/blog/ai-crawler-robots-txt". This indicates they are actively looking for specific content or resources related to our offerings. The fact that the "/robots.txt" file is so popular with multiple bots is expected. However, it's good practice to regularly check this file for any unintended blocking of resources.
We also noticed a few errors being flagged today. Bingbot encountered a couple of 503 Service Unavailable errors on paths related to shop LLMs: "/aeo/shops/merlin-com-tw/llms.txt" and "/aeo/shops/tidtech-com-tw/llms.txt". We also see 404 errors on "/aeo/shops/sh-y-com/llms.txt" and on the non-existent "/blog/URL". This suggests potential issues with those specific pages. We should investigate those 503 and 404 errors as soon as possible. These errors can negatively affect SEO and user experience. It would be worth checking if these shops are still active and accessible, and if not, update the internal links. Also, the /blog/URL should be fixed or removed.
For a concrete GEO/AEO tip based on today’s observations: Since Bingbot's focus is consistently on sitemaps, we should ensure our sitemaps are up-to-date and include all relevant content. Specifically, we should regularly check for broken links and include sitemaps for our hosted LLMs and site profiles, giving them a clear path to follow.
It’s always a good practice to examine the hourly distribution of visits to see when your site is most active. From the data, the busiest hours appear to be between 02:00 and 21:00, with varying levels of activity, so we need to ensure our servers are able to handle peak load. Monitoring your site’s AI traffic, as we do, helps us anticipate potential issues and identify opportunities to refine our content and site structure. Keep an eye on your own site's AI traffic and the insights it reveals. Remember to utilize AEO Scanner to monitor your website's AI traffic regularly.