I had a loan car from Lexus last week and it had the same issue. They had stuck velcro around the plastic section of the buckle (the soft part of velcro) and it was both un-offensive and effective. It will not stop the exposed metal hitting your B pillar but it may help.

I agree that it just shouldn't be an issue in the first place.