You need to check for any worn suspension parts, especially the inner bushings.

If there is nothing obviously broken, try replacing the 2 upper control arms of the rear suspension.

On both the 3.0 and 4.6 I had excess rear negative camber that could not be adjusted into spec on the alignment rack. Replacing the two top links we were able to get it back to spec.

The rubber bushings on the old arms were deflected inwards noticeably in both cases, resulting in extra negative camber.

__________________
04 X5 3.0i auto
03 X5 4.6is