Looks like the lock ring on the shaft seal popped off; bent the dust cover.
Edit: just noticed you said it was the driveshaft. But it looks like an axle shaft in the pic? Looks like they discontinued a shaft seal after the e53 was out of production.. Maybe a faulty part?
You might not have to drop the diff; but it depends on whether or not the drive shaft did any damage. Probably wont be able to tell until you start pulling off parts.
This is the 4.8is diff:
not sure if they're the same or not..
RealOEM.com * BMW E53 X5 4.8is DIFFERENTIAL-DRIVE/OUTPUT