Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You should look into robots.txt for telling Googlebot (and other crawlers) about where they should and shouldn't crawl on your sites.

Blocking by IP isn't the way to go, especially since Google's actual IP space in 66.249 and 66.102 is much smaller than the entire block (a /19 and /20, respectively) so you're causing a bunch of collateral damage to other networks in that address space.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: