What does OP mean by "a few minutes"??
My idle drops down after 5-10 secs. If car has been driven up to 3 hrs earlier it still has a lot of heat in it (depending on your climate obviously).

Anyway most of the time I don't even wait for the idle to drop. Just put it in gear and go. I rarely get over 2k rpms anyway with ALL THIS TORQUE!