He's not really wrong, but like it was indicated, he's playing semantics.
Will expects every "racist" to be some frothing at the mouth psychopath with a Winchester and a noose. That's not the case.
You get this behavior when whites et al let a few token blacks or other races in to be actors, athletes, or musicians, etc.
When the people who allowed you in lead you to believe you are special and you got in off your merits alone, you believe everything is equal.
The entire system is rigged, and you're being let in on a technicality or to fit an agenda. Not because everything is equal.
That's not to say everyone in power is racist. You like, support and promote what is similar to you. Most white people don't even have to make these considerations; it's inherent.
So perhaps it's less racism, and moreso privilege.