They can be really hard to get out! It's been a long time but I am pretty sure I took the spindle off (or at least drive shaft out) and then once I was able to get on it properly with a drift and big hammer it wasn't that bad.

__________________
1988 325is (purchased new) sold
2004 X5 3.0 2005 X3 2.5
2008 X5 3.0 (new to me)