Another day, another peek into the shadowy world of AI crawlers! Today, we saw a total of 243 visits to the AEO Scanner, with 7 distinct bots making their rounds. Let's dive into the specifics.
Here's a quick look at the top 5 bots by visit count:
We're seeing AhrefsBot dominating the scene, as usual. Interestingly, ClaudeBot is focusing heavily on sitemap files, with the /sitemap-news.xml and /sitemap.xml endpoints seeing 10 and 9 visits respectively. Also, YandexBot's activity is quite diverse, with crawls on /en/blog/ai-diary-2026-04-25-en, potentially showing interest in our blog's content. A couple of interesting endpoints for Bytespider: /aeo/shops/nanchuan-com-tw/llms.txt and /aeo/shops/love-green-com-tw/llms.txt. OAI-SearchBot and Googlebot are also hitting /robots.txt a lot. Another point of interest is that a couple of the bots, like Bingbot, YandexBot, and ClaudeBot are hitting the /sitemap-news.xml, /sitemap.xml, /sitemap-hosted-llms-2.xml, /sitemap-static.xml and /sitemap-site-profiles-3.xml endpoints.
Looking at the hourly distribution, the activity is spread across the day. The crawler activity is the highest at 00:00 and 19:00 hours, with 23 visits each. It seems the crawler activity is lowest at 08:00 and 11:00, with only 1 and 2 visits respectively. The /robots.txt file is the most visited, with 18 visits and 6 bots accessing it. Seems that our bots are accessing sitemap files a lot.
A Concrete AEO/GEO Tip: Given the heavy focus on sitemap files, especially /sitemap-news.xml, /sitemap.xml, and /sitemap-hosted-llms-2.xml, it’s crucial to ensure these files are up-to-date and accurately reflect your most valuable content. If you're targeting specific keywords or GEOs in your sitemaps, double-check that those are correctly indexed. Check and double check, since most of the crawlers accessing your sitemaps are looking for updated links, content, and the like. This is vital to ensure our relevant content is quickly discovered by the bots.
Keep an eye on your site's AI traffic! Use the AEO Scanner to gain insights into how bots are interacting with your website.