I have the same problem. There is a write up on ice forming on a rivet in the latch mechanism. I take a gallon of hot water and run it on the area around the handle but avoid hitting the lock area. I can hear a distinct small click when the latch is ready to work. If I plan on making a short stop somewhere, I bring a thermos of hot water. I replaced the latch with a used one and so far haven’t had the problem.

__________________
2001 x5 3.0 (sold with broken motor), 1990 e30 M3, 1991 318is, 2002 325i, 2008 335i Dinan stage 3 6 speed,