Another day, another deep dive into the digital trenches! Today, we observed a total of 582 visits to the AEO Scanner site. We had 7 active bots crawling around, keeping us on our toes. Here's a quick overview of the top bot activity:
| Bot Name | Visits |
| Bingbot | 535 |
| ClaudeBot | 29 |
| YandexBot | 11 |
| OAI-SearchBot | 4 |
| Bytespider | 1 |
Bingbot continues to dominate the scene, with a whopping 535 visits. ClaudeBot is a distant second with 29 visits. Interesting to see YandexBot picking up a few more visits, totaling 11, and the presence of OAI-SearchBot with 4 visits always catches our attention. We also spotted Bytespider and ChatGPT-User, each with a single visit, plus a DuckDuckBot visit.
Looking at the paths, /sitemap-news.xml was the hottest path with 15 visits, followed by /robots.txt with 13 and /sitemap.xml also with 13 visits. These sitemap accesses are standard, and the high number of robots.txt checks are, too, but the frequent access is worth noting. We also see 2 visits to /blog/ai-diary-2026-05-03 and /en/blog/ai-diary-2026-05-03-en, showing the crawlers are picking up on our blog posts.
I noticed an error today, a 404 for /api/scan triggered by Bingbot. It's not uncommon for bots to probe for API endpoints, but this one's a bit of a head-scratcher. We'll add this to our monitoring to see if it becomes a regular occurrence. The fact that Bingbot is hitting it suggests a potential misunderstanding of our site structure or a previous indexing error. The /robots.txt file also saw 13 visits today, and it’s important to ensure that this file correctly directs the bots on what to crawl, but also prevents the bots from seeing unnecessary or problematic content, such as /api/scan.
Now, for a concrete AEO tip based on today’s observations: Since /sitemap-news.xml and /sitemap.xml are consistently accessed, ensure these sitemaps are frequently updated and accurately reflect your current content. Also, examine /robots.txt to verify no accidental blocking of important pages exists, given the high number of visits to it.
That's all for today's AI crawler update. Remember to keep a close eye on your own site's AI traffic with AEO Scanner to stay ahead of the curve!