Slight OT, but it took me 4 hours to realize that it's Google who's crawling the URL of pages (mostly cron jobs, or admin sites) that nobody should have never stumble upon.
I'm not sure how this works (the URL I paste and use in Chrome is sent to Google?), but it came back few hours later from an agent called "Google favicon". The IPs in 66.249. range strongly suggest it's a legitimate Google bot.
Why they do it - no idea, but just to be save I blocked all 66.249. and 66.102. traffic across my systems.
EDIT: yes there is robot txt that dissalows all traffic. Google ignores.
This is very internal site - no need to have everyone from 66.249 acessing it.
You should look into robots.txt for telling Googlebot (and other crawlers) about where they should and shouldn't crawl on your sites.
Blocking by IP isn't the way to go, especially since Google's actual IP space in 66.249 and 66.102 is much smaller than the entire block (a /19 and /20, respectively) so you're causing a bunch of collateral damage to other networks in that address space.
I'm not sure how this works (the URL I paste and use in Chrome is sent to Google?), but it came back few hours later from an agent called "Google favicon". The IPs in 66.249. range strongly suggest it's a legitimate Google bot.
Why they do it - no idea, but just to be save I blocked all 66.249. and 66.102. traffic across my systems.
EDIT: yes there is robot txt that dissalows all traffic. Google ignores.
This is very internal site - no need to have everyone from 66.249 acessing it.