Got a warning for my blog going over 100GB in bandwidth this month… which sounded incredibly unusual. My blog is text and a couple images and I haven’t posted anything to it in ages… like how would that even be possible?
Turns out it’s possible when you have crawlers going apeshit on your server. Am I even reading this right? 12,181 with 181 zeros at the end for ‘Unknown robot’? This is actually bonkers.
Edit: As Thunraz points out below, there’s a footnote that reads “Numbers after + are successful hits on ‘robots.txt’ files” and not scientific notation.
the one with the quadrillion hits is this bad boy: https://www.babbar.tech/crawler
Babbar.tech is operating a crawler service named Barkrowler which fuels and update our graph representation of the world wide web. This database and all the metrics we compute with are used to provide a set of online marketing and referencing tools for the SEO community.
we?
It’s a quote from the website
It is common custom to indicate quotes, with either “quotes” or for a longer quote a
The latter can be done by prefixing the line with a
>here on lemmy (uses the common markdown syntax).Doing either of this help avoid ambiguity.