Nothing ages a man (or woman some day, for that matter) like being POTUS. They all come in youthful, and leave looking much older than a normal 8 years should put on a person.

Imagine all of that they know, that the rest of us don't. It's no wonder they bond with other ex's when they leave, even if they were political rivals at one point.