I just replaced the tension strut bushings and ball joints and it has dramatically reduced the problem. I think I'll also replace the lower control arms because the previous owner replaced only one side and I think the other is worn now. Best to replace both. Also my tires are worn a little uneven, so I'm sure this is contributing to the issue. I am running the OEM tire size and pressure is around 38 PSI which is the recommended pressure for these tires per the manufacturer for my X5. I'll update after the control arm replacement.