It has been 3 months on mine and the difference is big. Once the Sport button is pressed, it makes the suspension stiffer and there is no body roll on turns and/or twist roads. Lots of fun!
However, I did not notice any difference in throttle response or speed between gear shifts. I used to have a 3-series (E46) with SMG transmission and the sport button on that car indeed shortned the throttle response and made the transmission slightly quickier. I believe the sport button in X5 only impacts the active suspension while on other models impacts the transmission and throttle.