Before 1860 -- it really didn't matter what they were listed -- cause if you looked like you had any Black -- you were enslaved -- or subject to be enslaved - if you were in a free state and didn't have papers.
But, if someone was classified as Mulatto or Colored before then -- and on the 1850 Census and they were Free/Manumitted.
They were usually in Free States -- or if they were in the South -- they were more than likely -- classified as
gens de couleur libres or FPOC and living in N.O. With another classification -- Not Black -- Not White.
Regardless of their ethnicity, freedmen had been able to own plantations and often owned large numbers of slaves themselves. The slaves were generally not friendly with the freedmen, who sometimes portrayed themselves as bulwarks against a slave uprising. As property owners, freedmen tended to support distinct lines set between their own class and that of slaves. Often working as artisans, shopkeepers or landowners, the gens de couleur frequently became quite prosperous, and many prided themselves on their European culture and descent. They were often well-educated in the French language, and they tended to scorn the Haitian Creole language used by slaves. Most gens de couleur were Roman Catholic, and many denounced the Vodoun religion originating in Africa.
So, no they were not looked at as Black men -- nor did most want to be. And the those in the South were usually FPOC who were women. They were also enslavers.
BLACK MEN WITH WHITE FATHERS WHO SIGNED THE CERT WERE CLASSIFIED AS BLACK ON CENSUS FORMS IN THE MAJORITY OF SOUTHERN STATES BECAUSE THEY COULDN'T READ OR WRITE. THEIR FORMS WERE FILLED OUT BY CENSUS TAKERS. YOU HAD TO SHOW BIRTH RECORDS IN ORDER TO BE CATEGORIZED ANY OTHER WAY.
It was MILLIONS of Black men and women with White Fathers -- and they weren't signing birth certificates because they sexually violated their mothers -- and didn't want anything to do with them. The majority of Black people knew why they were mixed and it wasn't out of love. They weren't leaving them land or anything -- they wanted extra enslaved people to make them money.
Black people weren't on the Census until 1860 -- UNLESS they were free -- which wasn't many. Less than 500K total in the U.S. -- free and slave states combined -- and most were not classified as Black/Negro -- but gens de couleur libres or FPOC. The highest amount was always in N.O. or Maryland ( a border state to a free state)