Googlebot Slows Crawlers in Effort to Support Blackout

As blackouts roll across the web this week, some sites opposed to SOPA/PIPA legislation have kept the lights on in order to maintain their search engine rankings. As one of many tech sites to openly protest the bills facing Congress that would censor the web, Google first offered advice to webmasters on how to close or limit access to their sites without medium-term effects. Now taking it a step further, Google has slowed their web crawlers in an effort to support sites eager to participate in the online demonstration.

Google’s Pierre Far posted instructions on Google+ detailing how to update your site to join the protest of these bills. One such way is using a 503 HTTP status, telling spiders that the site is only temporarily unavailable. By using this code, the content is not designated as “real,” and will not be indexed. Googlebot’s crawlers are automatically slowed when there is a significant spike in 503 statuses. Paired with the manual configuration on Google’s end to reduce the crawl rate, more sites will be protected.

Mr. Far later emphasizes the importance of not altering the robots.txt file on your site  to block any crawlers, as this could affect the recovery of the crawl rate post-blackout. By keeping it simple and avoiding any drastic changes in coding, websites can be involved without severely adverse effects.

Craigslist, Wikipedia, and Reddit are among the list of prominent sites behind the massive social media effort protesting the proposed censorship of Internet data and policing by the Department of Justice. While Bing and Yahoo have not expressed any involvement, search results on these sites may still be influenced.

What do you think of Google’s actions in protest of these bills?




Comments are closed.