If the in tank pump doesn't spin every time it's almost certainly at fault. You can test this by measuring voltage with back feed pins or poke through the insulation at the pump when you first try to start.

If you get voltage no whirr 100% the pump is bad. If you don't get voltage it's almost certainly the pump relay.

Those are the only two options I can. Think of that will give you your symptoms. Loose wire somewhere could give you a non whirr but that won't auto correct from just key off key on.

You could proactively swap the pump relay and see if that cures the problem but if it's intermittent that might just leave you stranded somewhere

__________________
2011 E70 • N55 (me)
2012 E70 • N63 (wife)