Another day, another deep dive into the fascinating world of AI crawlers! Today, we observed a total of 519 visits to the AEO Scanner, with 6 active bots keeping us on our toes. It's always interesting to see how these digital visitors interact with our site and how we can refine our strategies accordingly.
Here’s a quick look at the top 5 bots by visit count:
| Bot Name | Visits |
| Bingbot | 500 |
| YandexBot | 11 |
| OAI-SearchBot | 4 |
| DuckDuckBot | 2 |
| Bytespider | 1 |
Bingbot continues to dominate the scene, accounting for the vast majority of our bot traffic with 500 visits. It's clear that the Microsoft crawler is actively engaged with our content. YandexBot trails far behind, but the activity is still worth noting. We also spotted a visit from ChatGPT-User, who checked out our blog post on geo-generative engine optimization. The presence of these bots helps us stay informed about how different search engines and AI tools are interpreting our content.
The most frequently accessed path, predictably, was /robots.txt, with a total of 13 visits from 4 different bots. This is a clear indicator that bots are checking our instructions first. /sitemap-news.xml saw 4 visits, highlighting its importance for informing search engines of our latest content. Interestingly, we also observed a 404 error for a bot trying to access a file in a particular shop directory (/aeo/shops/xn-hxt89po1jl6p-tw/llms.txt). Bingbot was the culprit here. This kind of error is a red flag. It tells us that either there is a typo in our internal linking, or a bot is trying to access resources that don't exist. It's a good time to double-check those URLs and ensure everything is correctly set up.
Now, for a concrete GEO/AEO tip. Given the 404 error we saw, it’s crucial to review all internal and external links pointing to our shops. Specifically, we should focus on the shop directories that the bots are accessing. Any broken links or incorrect URLs in the shop directories can affect bot's ability to crawl, index and understand the content, and it will hurt the ranking in search results. Ensure that the directory structure is logical, and the files are in the right place.
That's all for today's AI crawler update. Remember to keep a close eye on your own site's AI traffic with AEO Scanner! These insights can make a huge difference in optimizing your site for the future.