I've removed the panel so many times and with the confusion that followed. I started using Velcro to hold the push clips into place.
Now I just tear the door off the panel when I need to get back there...reinstalling carries much less confusion.