An audio clip would certainly help.

Maybe the secondary air pump that starts during cold starts to help warm the catalysts faster... or something related. There is a valve and a couple of vacuum hoses associated with that pump as well.

As for the exhaust manifold... my experience is on E46's, but the nuts are really fun to get to with the engine in situ. Also, at least half of the studs will pull out with the nuts. If it ends up being a gasket further down the line... they are also fun to get at and remove since they are usually fused due to corrosion. Order new ones before starting the job once you locate the leak (if it turns out to be an exhaust leak).

__________________
2005 X5 4.4i Build 04/05 Maintenance/Build Log
Nav, Pano, Sport (Purchased 06/14 w/ 109,000 miles) (Sold 8/15 w/121,000 miles)

2006 X5 4.8is Build 11/05 Maintenance/Build Log
Nav, DSP, Pano, Running Boards, OEM Tow Hitch, Cold Weather Pckg (Purchased 08/15 w/ 90,500 miles)

2010 X5 35d Build 02/10
Nav, HiFi, 6 DVD, Sports Pckg, Cold Weather Pckg, HUD, CAS, Running Boards, Leather Dash, PDC, Pano (Purchased 03/17 w/ 136,120 miles)