Assuming you're laying with your back/head leaning against the driver side front wheel, grab the alternator with both hands and pull it out /wiggle it back and forth towards you. Yes, it can't come out towards radiator because it sits in kind of a groove from which you gotta pull it out from first. Be careful as you do this so that you don't damage the oil lines that are nearby. Also be carefull as you take the alternator out not to damager the radiator. Put something like a card board on the radiator to protect it.

Now putting it back can also be a pain. I'd advice to put some grease so that the alternator slides into the grove easily.

You'll succeed in pulling it out- it just takes a bit of time and effort. Be patient. I've done this twice on my 2005 X5 4.4 sports. The alternator is the same as yours.

By the way, have you run diagnostics software on the car? If so, which one? INPA and/or DIS by any chance?

Good luck!

__________________
'05 E53 X5 4.4i, '97 E39 528, '07 E92 335i, '16 F86 X6M.