There isn't a big size difference between an X3 and an X5. In most interior dimensions they are within about 1-2" of each other. I've compared, my brother has a 2014 X3 sdrive 28i. and I have a 2008 E70 X5 4.8i.

Overall because of the better aerodynamics of an X3 vs e70 X5, better transmission lower weight I'd expect real world the x3 x35 vs an x5 35d fuel economy would be awash. The only real advantage of the D would be at the moment lower diesel prices, but that savings will be more than offset by the cost of emissions repairs and higher maintenance.. The M57's have well documented reliability problems with emissions components and suffer from accelerated Carbon Build Up. Which is why ABC deletes are so popular (I know I have a 2011 335D which is the same engine in the E70 except for low pressure EGR).

My opinion would be, the D because of its lower reliability, more expensive engine oil, DEF refils, etc. would more than offset any cost savings from fuel and not save you any money over the 35i. If you get 35D tuned that changes the equation some, but you'll have to spend $3000+ to do an emissions delete which has its downsides (smoke, smell). $3000 buys you a lot of gasoline.

The reason to buy a diesel is if you really like the engine and plan to do a lot of towing. Better yet would be to buy a N57 in the 2014 diesels which maybe more reliable. But don't buy a diesel one thinking it will save you money, its unlikely due to reliability and higher maintenance costs. Most D owners will tell you your happiest days are when you buy a D and when you sell it :-) I do like mine however.