There is a reset procedure for the auto-up/auto-down on the e38/e39 models...but the issue you presented with the shop that tried to install a new motor sounds like something totally different than normal reset procedures following the replacement of a window regulator.
There's a "normalization" process that should have occurred after replacing a properly working window motor/regulator where the "mechanical end stops" (up & down) are learned upon opening & closing the window for the first time. In your scenario...the motor kept going, when it should have stopped upon reaching that "mechanical end".
Once normalization has been learned (learnt)...then the auto-up/auto-down should be initialized. The procedure below is written for the "rocker/toggle" window switches before BWM switched to the "flip-up/push-down) type installed in the e53...just extrapolate the open & close position of the old switches to the new type for reinitializing the e53's window switch: