I feel your frustration. This is poor design and unacceptable part fit and quality
That being said: I fixed mine with a small self taping counter sunk screw. Pre drill a small whole in the handle stem. Install the handle using the vertical/horizontal insert twist procedure. From the back insert the screw and be careful not to screw down to tight. Just snug enough to secure the stem in place. Mine has worked great. Next time I'll have to buy another handle, but for now it perfect.Good Luck. Yes, you do have to remove the door panel.
J

__________________
X5 3.0 AUTO -2001 - 238,000 miles