Who are we? We are friends, brothers, cousins, co-workers, etc. of these guys. They may not listen, but even presenting a challenge shows that there is a consequence to acting on a sexist/patriarchal attitude.
I was using the white privilege/supremacy example to make a point that you can relate to as a man of Color. The same logic applies for why we should challenge other men on their sexism and misogyny.
That being said, I do think that women should strap up and be prepared to defend themselves. All oppressed/marginalized people who are aware of their social position should embrace self-defense
I can't give you a guide for when to intervene or not except if you see or hear something that is fukked up, challenge it if you can do so safely, or if you are comfortable with the possible danger.