If you want an offroader that performs as well as the X5 on road, get a Cayenne. It has a number of key features that are sorely lacking in the X5:

--Available air suspension can lift the car MUCH more than the system in the X5
--Comes with a low range and lockup feature in the transfer case
--Available with an electric locking rear diff, (quite rare though)
--Double sealed doors make it waterproof up to about the door handles

On road, a V8 "S" model is faster than a 4.6 or 4.8 X5, and considerably better riding. If that's not quick enough, you can get a Turbo. A Range Rover would be another superior option, and if you're looking for a used one, you can get one with many shared components as an E53 X5, including the same engine.

Face it, if you want to go off road, the X5 is the wrong choice. If all you want is to handle a gravel road, a set of all-terrain tires on the X5 should do it. I recommend the BFG Rugged Terrains. Comfortable on pavement, good in gravel and dirt, and decent in snow.

__________________
2014 BMW 328i Xdrive
2011 BMW 335i M-Sport
2008 BMW X5 4.8i Sport
2000 BMW 528i 5sp