Robert had a difficult life in Elizabethtown. After his father’s death, he lived with (and likely helped to support) his mother while also raising his own family. He eventually found work as a day laborer and married for the first time in his early twenties. He and his wife, known only as Mrs. Robert Van Horne, had three children together between 1844 and 1847, all of whom died very young and were buried alongside their grandfather, Peter. Not long after the death of their last child, Robert’s wife also died of unknown causes.
By 1852, Robert had married once again, this time to a woman named Sarah Johnson, who had come to New Jersey from Maryland. They had several children together, Mary Etta, Sarah, Frances, and Josephine, but only the younger two, Frances and Josephine, would survive to adulthood.
Robert’s mother, Sarah Van Horne did not live to meet her grandchildren from his second marriage. She died of dropsy on February 10, 1852 at the age of 57, nearly twelve years after the death of her husband. Robert was widowed for a second time in 1863 and by the time he died in 1871, at approximately age 48, his surviving teenage daughters were already working as domestic servants in white households.