If there's no evidence of fluid leakage from the differential, then the seal is probably okay, as would be the bearings. So the movement would be the result of excessive clearance between the end of the half-axle and the "socket" it fits into on the differential. Best case scenario is you only need a replacement of the half-axle. Worst case is both the axle and differential. Either way, you're probably going to have to pull out that axle for a complete inspection.

2002 X5 3.0 238,000 miles
2004 325i 106,000 miles