Lightness and low inertia really matters in rally to get quick changes in direction. That's why the Mitsubishis and Subarus do so well because there's nothing left when you strip out the interior. Also important to have low unsprung mass (wheel/tire/brakes) so the wheels can be controlled through the rough patches. Both of those are a bit lacking in an X5, but it does seem to be durable. May need some beefed up versions of front and rear links.
Center of gravity could be an issue, so would need to lower it some too. Travel is needed, but not necessarily the ground clearance.

__________________
2001 X5 3.0i 5-speed manual (sport, tow, cold)
2018 Chevrolet Bolt EV
1981 VW Scirocco mid-engine VR6
1983 VW Vanagon Westfalia
2006 GSX-R750