I'm new to this forum. Don't know if you are still having this problem, but I had the exact same issue with my 2002 x5 4.4 last year in the heat of summer. It was, as Weasel mentioned, the evaporator freezing over. That's how is started out, then was only blowing cold from the drivers side vent and hot from the passenger side. I could leave the system on, let the air blow full-blast and just turn the A/C off for 5 or ten minutes and it would start blowing cold again for a few minutes. I thought I was in for a huge bill for the repair, but it only needed a recharge. Low charge was causing the freeze-over, and I was told that the system blocks out the passenger cold air as sort of a "limp-home" feature when it senses a lack of performance.

Anyway, hope this helps before you spend thousands trying to fix it....check the refrigerant level/pressures or have them checked.