If an app asks for location information I usually deny it. I would do the same thing for my address book and photos if I had the ability.

Almost all of the apps I use have no reason to need my addressbook data so it would be nice to know that none of those are secretly stealing it.