my 2007 X5 had this problem. My new 2011 Has a different piece that is just a nice matte plastic with no rubber coating to peel off.
They replaced trim on 2007 under warranty.