Yeah I have play there too.

One way to confirm is to slide off the boot from the rack housing so the rack is exposed. Then move the steering wheel left to right. You may see the rack move up and down like I did.

Looking into it there are plastic bushings in rebuild kits but I've never seen a diy on them. Most people just buy a rebuilt.

__________________
04 X5 3.0i auto
03 X5 4.6is