I had to unpair from X5 side; unpairing from phone side did not help. And to unpair from X5 I had to pair 4 other phones as the E53 X5 can pair up to 4 phones and they stay in memory till the next phone is paired...First-in-first-out(FIFO).

My thread with this issue.