Surprising how quick and affordable it can be to crawl a billion webpages.

This would also explain why every website is under continuous load from spiders trying to capture content for training models.

https://andrewkchan.dev/posts/crawler.html