Fulton County constructed the buildings now known as Chastain Arts Center and Galloway School between 1901 and 1911. The buildings were originally almshouses (or poorhouses) for the elderly, disabled, and unemployable. Thomas Henry Morgan, known as the 'Dean of Atlanta's architects', designed the two racially segregated Neo-Classical buildings.
Both buildings were made of handmade brick; the White almshouse designed to accommodate a maximum of 145 residents and the Black almshouse designed for a maximum of 85 residents. During the depression, the facilities were operating at more than capacity. There were no other nursing homes, and as many as 200 individuals were housed in the White almshouse during the depression. Within the homes, residents were separated by gender in both the living and dining areas.

Female African-Americans from the Fulton County prison system cooked and cleaned for both almshouses. The smaller building of the Black almshouse, presently City Gallery at Chastain, housed a caretaker and his family as well as the female prisoners. Male Fulton County prisoners, who were kept in a prison building located on the current site of the baseball fields, raised livestock and vegetables to provide food for the almshouse residents. Known as the Fulton County Farm, these gardens were located on the site of what is now Chastain Park Golf Course.

In 1936, the Fulton County commission changed the name of the facilities to Haven Home. They kept the 1,000-acre land parcel intact until the early 1940's, when they sold out all but 268 acres for residential development. The remaining land was developed into North Fulton County Park, featuring premiere recreational facilities. Walter Hagan, with Bobby Jones as a consultant, designed the golf course. The Works Progress Administration built the eighteen-hole golf course, amphitheater, and swimming pool. A manmade pond originally fed the Olympic-size pool. The woods were landscaped with azaleas, rhododendron, dogwoods, and goldfish ponds. On June 20, 1944, the amphitheater held its debut performance. After the death of Troy D. Chastain in 1945, a man who loved the park and lived nearby until his death, the Fulton County Park was renamed Chastain Park in his memory. In 1952, the City of Atlanta annexed the park.

The White almshouse remained occupied until 1962 and the Black almshouse continued to operate until 1968. It was sold to the City of Atlanta, under the Department of Parks and Recreation, and opened as an arts center in late 1968. In 1975, building expansion destroyed the original office, infirmary and chapel. Another renovation effort in the late 1980's created a professional gallery space in the old prisoner/caretaker quarters.