I assume you have a 2003 4.6.

Well, it looks like a vacuum line goes to that intake hose (to the larger of the two connections). I don't know how much of a performance hit you will experience. If you're not experiencing anything much different, I wouldn't worry too much about it.

Looks like you may just need to R&R the smaller hose. Do you mean its hard to get to the other end of the smaller hose to replace it? Are you sure another hose connects to the smaller connection? I ask only because at realoem nothing is shown connected to it. I don't really know as I have a 4.8

Hopefully someone with a 4.6 can chime in and say definitively.

RealOEM.com * BMW E53 X5 4.6is Vacuum control - engine