Someone with access to your phone can also access your emails, text messages, contacts, photos and more... I guess to some people location data is more sensitive than those things, but not to me.