It is very easy to remove that piece. You don't even need to remove the door panel. You do need to remove the black plug in the side of the door. Once that is removed, you will have access to the bolt that holds on that piece. Once you remove that, you will have access to the handle. Hopefully that is all it takes to fix your issue. If you need to go further, you will need to remove the door panel. See here for the plug and hex bolt:
http://www.xoutpost.com/articles/x5/b...tml#post550399