He helped finance the publication of David Walker's Appeal and Henry Highland's "Call to Rebellion" speech. He gave land to fugitive slaves. He and his wife agreed to raise a black youth as one of their own. He also participated in the Underground Railroad and, in 1851, helped establish the League of Gileadites, an organization that worked to protect escaped slaves from slave catchers.
In 1847 Frederick Douglass met Brown for the first time in Springfield, Massachusetts. Of the meeting Douglass stated that, "though a white gentleman, [Brown] is in sympathy a black man, and as deeply interested in our cause, as though his own soul had been pierced with the iron of slavery." It was at this meeting that Brown first outlined his plan to Douglass to lead a war to free slaves.
Brown moved to the black community of North Elba, New York, in 1849. The community had been established thanks to the philanthropy of Gerrit Smith, who donated tracts of at least 50 acres to black families willing to clear and farm the land. Brown, knowing that many of the families were finding life in this isolated area difficult, offered to establish his own farm there as well, in order to lead the blacks by his example and to act as a "kind father to them."