I would first look at the tires for unusual wear and then look at front end suspension parts. X5's (well, BMW's in general) are the tightest driving cars I've ever owned (I have 3 currently) and there should be no need to work the wheel during normal driving.

__________________
'03 X5 3.0, '98 M3 vert, '02 Audi TT, '99 323i