My husband is a good 6" taller than me...I'm 5'2" and he's 5'8" and I wouldn't have it any other way. I just can't hang with short dudes...I dunno why. I think that part of it is psychological; I have a 'big' personality so I need someone physically bigger than me to make me feel feminine and girly...and short guys just don't do that.
It's also my experience that shorter men have shorter willies. Yes, size does matter.