Robert Smith, originally of Baltimore, Maryland, moved to New Orleans when he was sixteen years old and by the time he was in his early 20s, had set up a grocery business with a partner and bought a house. Smith successfully purchased and emancipated his enslaved wife Annette and their four children Peter, Randolph, Frances Cora, and Henry Clayton in 1836. Due to manumission laws, they were required to leave Louisiana, so the family relocated to Natchez in 1837.

The Smith family were officially licensed to stay in Natchez as of August 1843 and by 1851 had finished construction of a new brick home, the Smith-Bontura-Evans House, on the National Register of Historic Places. Smith built a successful business as a “hackman” in Natchez, which allowed his family and him to thrive for several decades. As a visible, entrepreneurial member of the free Black community and a sexton in the First Presbyterian Church, his presence in town was considerable. At his death in 1858, his obituary noted that “every citizen knew him” and that a great number of people showed up to mourn him, including fifty members of the Temperance Society, at least twenty-five carriages, and fifty people on horses. His wife sold the house in 1860 and the family moved away from Natchez due to tightening restrictions on free Black people leading up to the Civil War.