“
You can’t build a full web crawl in one day. You’d be blocked instantly. Crawlers have to be polite.
We spend a TON of time working with blog hosting companies to make sure our crawlers yield to their policies. For example, LiveJournal won’t allow us to crawl with more than 10 threads.
”
Saturday, January 5, 2008, 10:50