I have a 2005 3.0d but I've driven the 4.4 (both pre- and post-update) extensively. Also the 4.6is and the 4.8is. But for me it's not a question of the engines and I have no complaints about any of the Cayenne's various under-bonnet offerings. Here in Oz we get the V6, the atmo V8 in two states of tune and the twin-turbo V8 in two states of tune. We also get the Cayenne diesel.
In producing a vehicle that has genuine off-road ability, I feel that Porsche has compromised the Cayenne's on-road feel. Compared to the X it feels bigger, heavier and more cumbersome and the steering lacks the X's precision and feedback. All this varies of course with the respective tyre/wheel and suspension packages on offer (and the type of road surface). I'm sure that there are Cayenne fans/owners out there who would disagree, but that's just my two cents worth.
The other argument is that the Cayenne can perform off-road tricks that the X can only dream about so it's the better all-rounder. I just think that it's not as focussed as a road car.