For 2004 X5 MY your car was fitted with BMW Assist Only. No Bluetooth, despite the fact you have a BT passcode in your user guide/documentation. BMW included the passcode in error. Originally they planned to rollout the Assist with Bluetooth for the 2004 MY, but ran into problems and delays.

It was finally rolled out in 2005 MY.

You can "upgrade" your system to the Assist with Bluetooth at your own expense, or dump the Assist system, and upgrade to a Bluetooth only system. The choice is yours .....