That is exactly the concern. The ball joint fortunately doesn't move much and has 100s of times more surface area (than the wheel bearing) so can handle much more impact and why it's relatively safe to use impact to remove the rear hub.

I've been working on a lug bolt push method for the rear to avoid impact but it's just so convenient to use a slide hammer.

I won't use slide hammer on the front unless I'm already planning to replace ball joints in the front.

__________________
2011 E70 • N55 (me)
2012 E70 • N63 (wife)