My friend owns both. He has a 2001 4.4 sport and a 2004 Cayenne S (among other cars). Anyway, he says the Cayenne handles better at higher speeds (100+), but prefers the X5 in every other way. The brake calipers on his Cayenne look great, but he says they don't stop as well as the X5 and he says acceleration feels the same in both. He's had the Cayenne for about a year and the X5 since it was new. He's planning on turning both in on a 4.8 which goes to show you, well nothing really other than what he prefers, but at least it's a decent comparison.

__________________
2005 ///M3 Competition / Interlagos Blue-Black /6spd/SS Race Exhaust/Angel Eyes/Passport HW/Mich. Super Sports Soles (love em)
2008 ///M3 Melb Red w/ Beige Novillo, Tech, Prem, & 19" (Retired)
2005 X5 4.4i Estoril/Grey Sport, Nav, DSP, Prem, Pano, etc. (Wifey's)
2007 335i Silver/Black SP,PP,CWP, Comfort access, & Sirius (Retired)
2001 X5 3.0i (Retired) Silver/Gray Sport w/19"/Prem/DSP/CWP
2002 YZF600R (Retired)