Your 2006 X5 should not need the adapter and neither should the 2004. However BMW seemed to drop the ball in that some do and some don't. Based on the build dates of your 04 and 06 it is not needed
If it turns out you do check out our videos and you can watch us wire up the DSP