Interior door handles melting

[X5Cat]

Just received the new part and installed this afternoon. Decided to clean the dirt & grunge from the vinyl/leathery panel located & normally hidden just below the base of the handle part being replaced. When I did, the color surface started coming off the panel, and it seemed to be melting, too. I was only using a damp cloth. The handle hides the area well enough, so I just stopped with the cleaning. Now, I really might like to try something to arrest the process, like possibly PotsyDriver's approach. Just worried that it might be too obvious where the glue is applied, and I don't really want to coat the entire panel. Maybe I will practice on the piece I just replaced, with different compounds. I'm open to ideas on different things to try....