Another day, another deep dive into the fascinating world of AI crawlers! Today, we observed a total of 760 visits across our site, with 8 active bots keeping things interesting. Let's break down the details.
Here’s a quick overview of the top bots we saw today:
| Bot Name | Visits |
| Bingbot | 716 |
| ClaudeBot | 30 |
| ChatGPT-User | 4 |
| OAI-SearchBot | 3 |
| DuckDuckBot | 2 |
Bingbot continues to dominate the scene, accounting for the vast majority of our bot traffic with 716 visits. ClaudeBot is a distant second with 30 visits, primarily focused on sitemap files. It's interesting to see ClaudeBot consistently hitting /sitemap.xml and /sitemap-news.xml so frequently.
A few things caught our eye today. Firstly, the consistent interest in the /robots.txt file is no surprise, with 5 different bots accessing it. This highlights its crucial role in guiding AI crawlers. Secondly, the presence of YandexBot, revisiting yesterday's blog post, suggests it’s keeping up with our content, which is always good to see.
Looking at the hourly distribution, we see some interesting spikes. The periods between 06:00 and 07:00, and 10:00 and 11:00, are particularly active, with 46 and 48 visits, respectively. This could be due to scheduled crawls or the bots prioritizing activity during specific times.
AEO/GEO Tip: Given the heavy traffic to our /robots.txt, make sure to review this file regularly. Ensure you are allowing access to important resources while strategically disallowing access to content that you do not want indexed, based on what you have observed crawling your site. Make sure you don't accidentally block any of the key paths the bots are visiting on your site. For instance, if you see bots hitting a specific sitemap frequently (like ClaudeBot is), make sure your sitemap is up-to-date and accurately reflects your site's structure.Keep an eye on your site's AI traffic! Use AEO Scanner to monitor your own bot activity, identify trends, and refine your SEO strategies. See you tomorrow!