Turns out it was a pretty decent boost leak from two things.

1 - When I deleted the EGR, the plate that goes on the hole near the turbo was on backwards so it was allowing boost gasses to pass by slightly under normal use and a lot when under a lot of pressure. Took it off, flipped it around, solved

2 - The caps used for the swirl flap delete had shoddy seals. We replaced them with new caps that had better o-rings.....solved.

All is well now. Brad Tourt did a majority of the work and I am very appreciative he lives somewhat near by. He had a great DIY tool that connected to the inlet of the small turbo that he hooked to the compressor and would pressurize the engine to 40lbs. Made these boost leaks VERY obvious. I'm going to making one in the future because damn was it helpful.