I am getting rid of this issues just by letting the key in #2 position and adjusting everything the way I like hitting the M (memory) button on the driver's seat followed by #1 button which holds my presets.
It won't freak until my wife uses the X.
Well to my luck she is not driving that often.
All this started when I got the spare master and Valet from the dealer and they had them programmed for my X...

I still don't understand why my drivers side mirror seems to move in the first place. The steering wheel moves every time I get in and out, so that makes sense, but still.

well,steering wheel let's you get in with less discomfort possible,
every time you unlock the X it receives key's memory presets and adjusts all the moving components : mirrors-->seat,A/C temperature-->st.wheel
try to do what I described in the previous post,it should stop it,if you adjust everything when you already started an engine it won't write the memory.

I moved the seat and mirrors around a little to have ir be different, reprogrammed all 3 settings, so now I've got my fingers crossed that when I leave the casino's tonight that it'll remember my settings. Otherwise, I'm still on the hunt for the module.