In 1858 Collins began living with Caroline Graves and her daughter Harriet. Caroline came from a humble family, having married young, had a child, and been widowed. Collins lived close to the small shop kept by Caroline, and the two may have met in the neighborhood in the mid-1850s. He treated Harriet, whom he called Carrie, as his own daughter, and helped to provide for her education. Excepting one short separation, they lived together for the rest of Collins's life. Collins disliked the institution of marriage, but remained dedicated to Caroline and Harriet, considering them to be his family.[19] Caroline had wanted to marry Collins. She left him while he wrote The Moonstone and was suffering an attack of acute gout. She then married a younger man named Joseph Clow, but returned to Collins after two years.[4]
In 1868, Collins met Martha Rudd in Winterton-on-Sea in Norfolk, and the two began a liaison. She was 19 years old and from a large, poor family. She moved to London to be closer to him a few years later. Their daughter Marian was born in 1869, their second daughter, Harriet Constance, in 1871 and their son, William Charles, in 1874. When he was with Martha he assumed the name William Dawson, and she and their children used the last name of Dawson themselves.
For the last 20 years of his life Collins divided his time between Caroline, who lived with him at his home in Gloucester Place, and Martha who was nearby.