My passenger side mirror was doing the same thing. It was solved by installing a new mirror motor.

__________________
2006 X5 4.4i
Sport package, summer package, cold weather package, DSP + Nav
Bought 2/14 with 73k miles