Mine had a similar issue. Hot or cold. Sometimes it would take 2 cranks. Super rare for 3 cranks. I did a lot of work on mine though. I think a starter replacement helped the most. A new starter spins up the 3.0 fast. Check the condition of the battery too.

__________________
'06 X5 3.0i - bought @143,123 miles (12/26/20)