I had the same problem. You can remove it without moving the air conditioning compressor, besides I think its integral to the engine mount? You have to force it against its will a little bit. It is defiantly possible though. I think I turned it to the passenger side and had someone push up on the bottom while I bent it ever so slightly on top. Twice as fun putting it back in......

__________________
2003 X5 4.4 2002 530I