This is happening to me as well. It was raining the night I parked the car in the garage. Rolled the windows up and the next day when I got in the vehicle the Front left window did not roll down anymore. When pressing the switch up I hear the clicking and the lights dim so that means the motor is working, but pressing to go down does nothing. Checking from a working window you hear a click going up and going down no matter what. I too am thinking it is the switch. A wiring diagram should be online, there is a BMW X5 WDS online
WDS BMW Wiring Diagram System - X5 E53 that you can access. Now I have to roll my sleeves up and figure this out.