I think the plastic part stays where it is . maybe the part on door panel slides over it .
I removed my panels 2 years ago to stop leaks and gave up at the same stage where you had trouble until I got it figured out .
once I knew the procedure the panel removed easy .