Yep, I know what you mean.
It's very annoying to me but only when I see/hear a group of people speaking loudly in Chinese in public. But unlike you I think I'm over sensitive because I DO understand what they are saying.
It happens at my daughter's school where a group of Chinese mom's always gather in the hallway and yak loudly. I always walk by and pretent I don't see or hear them. I guess I'm put off because it sends the message that people shouldn't even say "Good Morning" to them unless it's in Chinese. That and the fact that they're so LOUD.