Yeap, its the ball joint that is seated in the Knuckle alright. It's poor design if you ask me. I love it when e39'ers post the 'Oh, use the ball joint removal tool!" God Damn it, I wish I could. Those ball joints are stuck in their faster than an Alabama Tick.
I think this is about right
Its a painful job without an air chisel, as I said, I am good with a grinder and did mine with out damaging the knuckle, but you need time and patients. After you weaken the ball joint housing, then the BFH comes into play. Smash it, bang it, and go to town on it.... it will come out... but it makes you work for the victory.
Good luck.