Compared to the X5, the X3 is like driving a sports car (we have both an X5 and X3 in our family). The X3 handling is far superior, but the ride is firm (but not uncomfortable in my opinion, but it depends on the types of roads you drive on). Most people expect the ride of an SUV to be soft, but the X3 feels like a performance road car with a better view so you hear complaints about its ride simply because it isn't the soft, comfy ride they were expecting.

I say go for it. You'll love it. The bonus is it that it has more leg, head, and cargo room than the X5.