Another day, another fascinating dive into the world of AI crawler activity on AEO Scanner. Today, we observed a total of 851 visits, with 7 active bots diligently indexing our site. It's always a bit of a thrill to see how these automated visitors interact with our content.
Here's a quick rundown of the top bots by visit count:
| Bot Name | Visits |
| Bytespider | 477 |
| AhrefsBot | 324 |
| ClaudeBot | 41 |
| Bingbot | 5 |
| OAI-SearchBot | 2 |
Bytespider continues to dominate the landscape, accounting for a significant chunk of our traffic. They visited 477 times, with a remarkable 475 unique paths crawled. AhrefsBot followed, with 324 visits, and ClaudeBot had a more focused approach with 41 visits, but only 6 unique paths. It seems they are more targeted in their crawling.
One particularly interesting pattern today is the popularity of the sitemap files. Both /sitemap-news.xml and /sitemap.xml saw high traffic, with 16 and 15 visits respectively. This suggests that crawlers are actively using sitemaps to discover new content, which is great to see. Also, it’s worth noting the appearance of a new bot, DuckDuckBot, with 1 visit. We should keep an eye on this bot and how it engages with our content in the coming days.
Looking at the errors, we can see that a few shops on the site are returning 404 and 503 errors. Bytespider found a few of these. AhrefsBot also found a 404. These are potential areas for improvement. Since Bytespider is so active, it would be a good idea to ensure all the links on pages crawled by Bytespider are working, as a quick fix. Addressing the 503 errors should be a priority, to ensure we offer an optimal user experience.
As a concrete AEO tip, consider closely monitoring the crawl patterns of Bytespider. They are a significant source of traffic, so make sure they are able to access and index your site effectively. Check your site's structure to make sure that the links they are following are correct.
Remember to stay vigilant and keep an eye on your own site's AI traffic. Use AEO Scanner to gain these valuable insights and stay ahead of the curve!