I would've too.
I've never agreed with the part of football culture that 'allows' refs to put their hands on players as they often do, and as this ref did.
Not only did he put his hands on the player, he was forcibly pushing the player away..and not just away from the point of the initial conflict, but even after the player had reassembled with his own team the ref was still in his face pushing and pulling and tugging on him.
Yep, I would've clocked the livin sh!t outta him, same way. Let my coaches and teammates hemm me up like that, not you.