I think this is a reality you need to get used to. Its not just being supposedly "cultured" whatever you mean by that term but you're going to get funny looks for being a black person that does ANYTHING outside of the low expectations and views other races and even other black people put on you as a black man. This is only another manifestation of that overall view. If you're not playing ball, being a menace to society, or fitting into what others expect of you, they'll look at you as odd and are probably just waiting for you to do one of those things.
Another way of saying it is, they don't trust you. They wonder when the ****** is going to act like a ****** basically.