Are you talking about the base of the ear (where it connects to the head) or the outer edge of the ear?
I don't think it would hurt anything to trim the hair. The only thing I can think of is if there is some hair at the base of the ear, it may be there to help protect from stuff (dirt, etc.) going down inside the ear since they stick up. But, it may not matter at all.