It's the compressor. Typical failure on the X5, it has a valve that regulates the refrigerant, rather than the clutch going on and off like 'normal' compressors. Mine does this too. I bet if you rev the engine, it gets cold quicker.

__________________
2000 e39 M5 Titanium on Black
2006 e53 M54 Titanium on Black
2007 e83 Black on Black Sport pkg
2004 Avalanche 2500 lifted 4.88 gears
1994 jacked up Jeep
1987 GMC lifted short bed on tons