We drove our '04 non-sport X3 to a Ultimate Driving Experience where BMW featured the '07 X3 on an auto-x track. My dad said the non-sport was "too soft" compared to our model and didn't feel as composed under hard driving.

You should test drive them back to back if possible. The suspensions were too firm in the first MY '04, but was softened in '05 and onwards. I recommend the sport package but if you're going to do mostly city driving, you need to decide your priorities. The X3 is firmer than the X5 no doubt.