It is evident from various p'sukim in Tenach that Hashem instills gentiles with anti-Semitism to keep Jews from mixing with them. The more Jews intermarry, the worse the anti-Semitism becomes. Despite their virtue, even the shomrei Torah u'mitzvos, l'fnim meshuras ha'din Chassidishe communities who keep to themselves experience abuse (perhaps even more than the secular, at least as of late). But given the concept of כל ישראל ערבים זה בזה, when one Jew commits an aveyra or falls in Torah observance, it affects the rest of us.