Another day, another mountain of bot activity! Today, we observed a total of 1383 visits, with 7 active bots crawling our site. Let's dive into the details.
Here’s a quick look at the top 5 bots, based on their visit counts:
| Bot Name | Visits |
| AhrefsBot | 773 |
| Bingbot | 550 |
| ClaudeBot | 44 |
| YandexBot | 9 |
| Bytespider | 3 |
Interestingly, AhrefsBot and Bingbot continue to dominate the scene, accounting for a significant portion of our traffic. We are also seeing a consistent presence from ClaudeBot, which is always interesting to monitor. AhrefsBot had 771 unique paths crawled. This level of activity tells us they are busy indexing our content.
Looking at the error logs, we find a few 503 errors and 404s. AhrefsBot reported 503 error on /aeo/shops/transmedic-com-tw/llms.txt and 404 errors on /aeo/shops/w-i-com-tw/llms.txt. Bingbot reported 404 errors on /aeo/shops/xn-54qr5qsul2q6b-tw/llms.txt. These errors could indicate potential issues with the LLM.txt files related to those specific shops or that the content is unavailable. It's always a good idea to check these links and ensure the content is correctly served, as this could impact the bot's ability to index our content effectively.
Today's actionable tip: Given the high frequency of /robots.txt requests (27 visits, across 7 bots), it's crucial to review and refine your robots.txt file regularly. This helps ensure that bots are correctly guided through your site and that important content is prioritized for indexing. Furthermore, since ClaudeBot is actively crawling our sitemap-news.xml and sitemap.xml, making sure these sitemaps are up-to-date is also very important for SEO.
Keep an eye on your own site's AI traffic with AEO Scanner! We'll keep you updated on the latest trends.