my apologies, did not know you had a diesel.
I bet the difference is, as you say, just operating in a higher rev range. The motor needs to be pushing through air in order for the turbo to have a good effect. Which is hard to do at low revs, and small throttle openings, which is kinda what non sport mode is doing, to keep emissions low.