Unfortunately, TIS is correct, you have to remove the entire center console to properly refit the surrounding trim
Did this myself a while back, no way around it, you have to remove the console, otherwise you will never clip the part towards the front.