Yeah,

I'm not exactly sure what the fault is in my sloppy mirror. I'm pretty sure it's the housing, but when digging around the Jeep forums, many people also said the gear set that go between the motor and housing ring also may fail. I'll probably end up ordering both the housing and gear kit.

For everyone mentioning tightening the torx screws, if they are loose, that will give you a Z axis slop (up and down), but if you have X to Y slop (rotational), the Torx are not it, and more than likely the housing or gear assembly. with the links you posted being the fix parts.