You can also help us help you better if you would "identify" WHICH 2005 E53 X5 you have.
You could have one of 4 engines BMW used that year...there may be something specific to the N62 V8 engines (4.4i/4.8is) versus the M57TU diesel engine (3.0d), versus the M54 I6 engine (3.0i) that we may be able to point you directly to (as well as pulling the OBD II codes)...but without any model info...we could suggest incorrect information.
You can add this detail to your signature by clicking on the USER CP link at the top left of the page and then look for EDIT SIGNATURE on the left side of the page.
Off the top of my head and definitely just an assumption...it sounds like you may be describing the
N62's intermediate lever issue....but this of course only applies if you have one of the N62 engines.