You have an vacuum leak! My suggestion - get a vacuum or a smoke test on the engine, I forgot the exact term but any Indy worth his/her salt will know what you are asking for - but you need to find where the leak is.
This is one of the most common issues with the E53/M54 engine especially if you have already ruled out the usual suspects. The culprit is generally the CCV (Crankcase-Ventilation-oil separator). Its a PITA DIY job but not impossible with some time and beer.
I have to attempt it soon on my Wife's X - not looking forward to it.
Get the test!
RealOEM.com * BMW E53 X5 3.0i Crankcase-Ventilation/oil separator