I think I found something on online WDS. On cars up to 2001_09, with this mod, you'll see separate signals from stereo amp and telephone amp, 4 wires in all, as in mine.
From 2002_03 phone signals goes through stereo amp and from there the normal 2 wires to the speaker. Thats why you didn't noticed any changes on your X5.

Connecting the wires as you suggest is not a good solution, you'll probably end up with a bursted telephone amp.
Maybe the only solution is to fit an extra speaker somewhere hidden away or is there some easy way to separate these to circuits?