Its going to be the lower control arm or tension strut as its called. The bushing goes bad and this is what is causing your issue. You can remove the arm (its a DIY), and get a new bushing pressed in.
I did mine a few weeks ago, but chose to go with new arms to save the hassle of pressing a new bushing it. It won't take you more than 2 hours to do both sides. If you chose to do just the bushings, you can find them here
BMW E70 X5 xDrive35d M57 3.0L Suspension Bushings - ECS Tuning
The entire tension strut with new bushings and ball joints is here
BMW E70 X5 xDrive35d M57 3.0L Suspension Control Arm - ECS Tuning