I don't know how long the glue takes to set but I would try to remove it if it hasn't set yet.

Reason being... the plastic emblem itself can easily be put back with new gromits; however, when the "paint" comes out of the plastic emblem you will need to remove it again, and if it is hard set by then I forsee some paint damage removing it.

__________________
2006 X5 Sport 4.4i