I agree with other that most people are not going to know the difference between the X5 and the M version. Biggest thing they are going notice is that you are driving a BMW period. It doesn't really matter what model or year it is to people who are not in the know.
Believe it or not, I have had people make comments like "I see businees is good" when I pull up in my beater e36 325i!!!
Do what makes you happy. You work hard and if you want to spend a larger portion of your income on a car than most of your co-workers...so be it.