I went through similar with an e46. Did you check the check valve?

It may seem to be "checking" but it may be leaking. I checked it by blowing against it into a container with water. It seemed fine with lots of resistance to my blowing but the bubbles showed otherwise so I replaced the valve along with the vac pump.

__________________
04 X5 3.0i auto
03 X5 4.6is