Similar problem with my 02 several years ago. Turned out to be the u joint, thankfully, as I didn’t want to deal with the rack. I ended up replacing my u joint (“double joint” in RealOEM) to fix it. Several good threads around here though that discuss tightening the screw connecting the u joint to the lower assembly, which has fixed this problem for some folks.

Hopefully it’s not your rack.

__________________
Current
Original owner 2002 E53 X5 4.4i to 4.6i swap
2026 G80 M3 6 spd
2025 G06 X5 50e
Former
1972 Audi Fox
1986 Saab 900S
1996 BMW Z3
1998 BMW E36 M3 Sedan
2004 BMW E46 M3
2006 Audi A3 Quatro
1993 Mopar 318 Jeep Grand Cherokee
2015 V6 Jeep Grand Cherokee