I did this a few months ago to replace a burnt out bulb. If I remember correctly there are two metal tabs holding it in place facing the back of the X, they are located between the main interior light button, and the individual map light buttons so you have to start prying from those locations. What are you using to pry it up? i highly suggest using a plastic panel removal tool if you have one handy, the two clips do a good job of holding it in place. Once you get past those two clips you just rotate it down out of the housing and the front tabs will unseat themselves.

just go slow...you'll get it...

Edit: Like this video shows (i still can't get the embed youtube function to work...)

__________________
| E30 M3 | E34 M5 | E91 328xi | E53 X5 4.6is | E38 740i M-Sport |