You should feel right at home in terms of performance with the 330xi. Naturally, you might feel boxed in inside...
However before I had the X5 and the 330i, I had an Audi A6 2.8 - that was a large car - more legroom front and rear than a 5-series. Going from that into a 330i was tight for the first 6 months. Plus I declared the trunk in the 330i as useless due to the hanging HK subs.
But you get used to it after a while, and with the 330i, you can move into small holes in the traffic easily. LOL I still perform such manuevers with the X5 - scares the s*it out of people; sometimes me too.
The 330xi is a VERY fine automobile, which would be my natural replacement for the X5 if I didn't need cargo space.