There was a black community in lower Manhattan where Wall St. is currently located. In fact, it’s in the site of an old slave market. Trinity Church Wall Street was built by slaves. Blacks however could not worship at Trinity, I believe, so they built their own church, St. Phillips, up in Harlem.