The dust collar is press fit. Once you have the hub off you can as you are thinking, tap it off with hammer and chisel. I usually replace them when servicing the bearing so I just hit it with a hammer no chisel needed. If I used a slide hammer to remove the hub I don't remove the dust collar. After putting hub back on, I will fine tune the shape of the collar to make sure it's not scraping the hub using chisel and hammer.

__________________
2011 E70 • N55 (me)
2012 E70 • N63 (wife)