I looked on realoem, and yes the part I am speaking of is called the water valve:
RealOEM.com BMW E53 X5 4.4i Water valve/Water hose IHKA
It is #1 in the picture. Again, it's only a guess, but likely 1 of the 2 valves is stuck. Note that each valve controls one side, 1 drivers and 1 passengers. When mine was bad in my Merc it would intermittently blow hot air only, even if
AC was on. Sometimes it would be fine, then suddenly a rush of hot air would come out.
The water valve seems to be the likely culprit here IMO. Unlikely to have anything wrong with compressor, etc because you wouldn't be getting cold air on one side. It would be hot air all over. A stuck valve could easily overcome the AC system with hot air. I've felt it firsthand.