Another day, another deep dive into the fascinating world of AI crawler activity! Today, we observed a total of 672 visits to aeo.codecity.com.tw, with 9 active bots making their rounds. Let's break down the key trends and insights from today's data.
Here's a quick look at the top 5 bots by visit count:
| Bot Name | Visits |
| Bingbot | 603 |
| ClaudeBot | 36 |
| YandexBot | 11 |
| AhrefsBot | 10 |
| ChatGPT-User | 4 |
As always, Bingbot continues to dominate the scene, accounting for a whopping 603 visits. Their focus seems to be primarily on crawling the main site and various sitemap files. We see 5 visits to the homepage (/), 3 to /sitemap-hosted-llms-1.xml and 2 to /sitemaps.xml.
Interestingly, the /robots.txt file continues to be a hot spot, with a total of 13 visits from 7 different bots. It's a clear indicator that these crawlers are diligently checking our instructions on what they can and can't access. ClaudeBot, OAI-SearchBot, Bytespider, and Googlebot are all making sure to follow the rules.
We also spotted a couple of 4xx and 5xx errors. Bingbot reported a 503 error on /aeo/shops/kainchung-com-tw/llms.txt, and a 404 on /aeo/shops/w-b-com-tw/llms.txt. This suggests that there might be some issues with the availability or existence of LLM-related files in those shop directories. This is something we'll need to investigate further to ensure proper indexing and user experience.
Now, for a practical AEO tip: Given the frequent crawling of /robots.txt, and the errors observed, it's crucial to regularly review and update this file. Specifically, ensure that any LLM-related paths are correctly configured to avoid errors and guide crawlers effectively. Double-check your shop directories for broken links, especially those related to AI or LLMs. A well-maintained robots.txt file, combined with accurate sitemap submissions, is essential for guiding AI crawlers to the right content.
Remember to stay informed and proactive in managing your site's AI traffic. Keep an eye on your crawler logs, monitor trends, and adapt your AEO strategies accordingly.
Until tomorrow, keep scanning!