I agree. That was what my step-grandfather was essentially arguing when he told me different stories demonstrating that fact. During college I was all into the Pan-Africanism thing, and he was trying to open my eyes to the truth.
One thing about my step-grandfather was he was so light and had straight hair, that everyone who saw him thought he was Greek or something. So he was able to hear a lot of BS different groups said about us from immigrants, police, and of course White people; all because they thought he was a White man. Even one time he took me and my grandmother to a driving range, where a White man whom he use to work with at the Steel Mill saw us and was shocked he had a Black Wife. When the White man asked him about it, my step-grandfather told him he was Black too. That White man never talked to him again.