When making the decision to buy my 05 4.4 I test drove a sport and non-sport back to back (to back). Included plenty of cornering, low-speed and high-speed lane changes. I found the sport package much more confidence inspiring with little or no compromise in ride comfort. Ended up taking one home.

fyi...a good friend of mine owned an 03 4.4 with sport package and complained about the ride constantly. We both agreed that my 05 rode MUCH better. There appears to have been some significant improvement to the sport suspension with the face lift in 04.

__________________
2005 X5 4.4 (Sterling on Truffle) w/sport everything
2011 Audi S4 (Prestige, DSG)
2012 Audi Q7 (Sport, S-Line)