I have a 2003 4.4 non sport with comfort seats and had the 18inch wheel setup. The ride was very smooth over bumps/potholes. However, there was a lot of float on agressive cornering.
Now I have put on the 20's replica wheels and wow what a difference! First off the car's look has changed dramtically. The car feels actually just as smooth as when it had 18's on them. The float is gone and only slightly more feedback from the road.
I had driven an x5 with sport package and it seemed a little too firm for us. My wife drives the x5 about 90% of the time with our 7month old baby in the rear car seat. To much bouncing around and firm sport ride was not an option. Hence, we got a great deal on a non sport model, but by swapping out the 18's to the 20's we still maintain a great/smooth ride, have an aggressive look and great handling! Oh, and the comfort seats are awesome with so many different positions available.

If you drive with family/kids, go for the non sport.
If the X is for yourself and you have no kids get the sport model!!

__________________
Current cars:
03 BMW X5 4.4i NAVI (Dinan software upgrades-tranny and Engine ECU, 4.6 body kit w/ 20's staggered wheels, aluminum running boards, M3 SMG steering wheel, eisenhaus race exhaust, predator angel eyes, 6000k HID fogs, hardwired V1 radar)
06 Honda Odyssey EX-L RES NAVI (4300k HID's)
00 Mercedes S430 SPORT NAVI
08 Honda Accord V6 NAVI sedan (4300k HID's)