Ignore them. Or laugh at their insecurity, either way
My brother is the only person to ever tell me my slk was a girls car, only to find his beloved Z4 at the top of the list for cars most driven by women (forbesauto.) Not to mention he was just giving me a hard time and wasn't being serious.
Maybe it's because I'm too young to have a wife or because a convertable is perfect for the weather here that I haven't ever gotten any "girly car" comments. Just a lot of compliments, even from other car enthusiasts.