That's where you don't get it. Tom Brady is not most men or like most NFL players. You don't win like that, accomplish all those feats, and become the greatest of all time by doing the same things as your peers. That's delusional.
Kobe used to talk about how he'd work out 4-5 times a day for hours starting at 4am... in the off-season. You better believe Tom at 45 is doing the same shiit he's been doing his whole career. Part of what makes athletes great is a huge amount of study. Then add in business ventures that require travel, endorsements/commercials that require travel and being on set.
So yeah, the average athlete isn't doing all of that. But the ones who want to reach unreal levels of greatest do so knowing their personal life will suffer. They are willing to make that sacrifice most aren't.