Goingback Chiltoskey was born into a Cherokee-speaking family in the Piney Grove community of the Qualla Boundary. Given the name James at birth, he was renamed Goingback while still a young child. Like most children born at the start of the 20th century, Goingback's parents made many of the things they needed at home. He grew up watching his father carve wood implements used by his family. Even before he went to school, Goingback was making his own toys, assembling toy waterwheels from cornstalks and setting them up in the rushing water of nearby streams.