I bring Stella off lead when we go visit my horses. Her recall is good, but not always 100%, so I make sure to watch her like a hawk. However, this place is enclosed and not open to the public... otherwise, I wouldn't let her loose.
I would never bring a dog off leash on a walk through a neighborhood, park, or any other place where you might come across strangers and other dogs. There's just too much risk. You're not being too overprotective at all- there's very good reason to be worried.