Could he have been abused by a man previously? Is he a rescue?
If so, he might just need time to adjust
also I heard, if you have a dog that's intimidated by you, you shouldn't look straight at him, but face him sideways and let them come to you.
Don't get discouraged!