I had a 2005 X3 before (my son was from 0 to 18 months), and now a 2002 X5 (my son is 2.5 years old). I changed car due to moving to another country, nothing to do with disliking the X3, I loved it a lot.

- Handling - X3 was significantly better, much more fun (xDrive), and felt a lot more like driving a sedan / coupe instead of SUV, than the X5 (weight); My X5 has sport suspension and lowered, my wife and son have no issues with them :-p

- Cargo space - reports / specs claim X3 actually has bigger cargo space than the X5, but they felt pretty much the same to me. I seat my 3 x Golden Retrievers in both cars without putting down the rear seat without any worry.
- Passenger space X3 - is definitely smaller, but both my wife and I are short so not much of a problem for us in terms of head and log rooms. I had no problem putting the infant seat (back-facing) in the rear center seat, but the remaining left and right seat was a little small, especially for a larger size adult on a long trip (it was never a problem for us because it's usually just me, wife and baby).

- Passenger space X5 - Significantly better, difference is exactly like a 3 series vs 5 series. My son is now 2.5 and has a much bigger front-facing baby seat which is put in the rear center seat as well, and the left and right seats are still comfortable. I can't imagine if it was the X3, unless it is put on the left / right rear seat instead.

Hope this helps

__________________

Here's the Photo Album of my (sold) X3; Album of my (newish) X5 is on the way!
(WARNING: read this before you buy anything from Umnitza)