It's not guilt that's stopping them.
It's the laws put in place that are stopping them, prime example is how civil rights laws pretty much made KKK groups dead/irrelevant and faded into the background. They still exist but are powerless compared to how they were before the laws were enacted.
If Jim Crow and many of the other segregated pre-1960s laws stayed in place heading into the 70s, 80s, 90s, etc to now then the state of Black people might be much different than it is today because white people would have much more leeway, and heart to take more chances and act out extreme measures knowing the laws would benefit them.