the wheel bearings are attached with a large c clip to the main steering knuckle, then the hub is friction fit (ALOT of friction!!) to the wheel bearing inner race, typically it requires tremendous lateral force to remove. If you are 'rolling' the X5 without starting the engine, I suspect it will be fine. however if you start it up, and move it, your transmission/transfer case may not be happy about it! (who needs power steering and power brakes anyhow eh!!)
good luck!

__________________
2003 X5 3.0i (02/2003) sport package, Sapphire black metallic / Black Dakota Sport seats. compass mirror retrofit, trailer hitch retrofit, Prodigy P2 trailer brake controller installed, PDC retrofit, Lumbar support retrofit. heated rear seats retrofitted. Running boards retrofitted and subwoofer install in progress. 4.8is Brakes installed front and back. Bluetooth TCU installed, Rear Fog lights installed. Retrofitted reclining rear seat backs.