Allan Ferguson Brumby was born in 1905 in the far north of South Australia where his parents, Harry Brumby and Grace (nee Ferguson), owned a pastoral station. As a young man Allan worked on pastoral stations in the far north and also engaged in dogging, collecting dingo scalps for the bounty then offered by the South Australian Government. As a dogger he evidently travelled widely throughout the Central Australian Aboriginal Reserve. According to Michael Terry, an explorer/prospector, Brumby, with another dogger, TH ('Ginger') Brown, climbed Ayers Rock in 1928 or 1929. Allan Brumby had an Aboriginal wife, Munyi, who bore him four daughters. After Allan's uncle, Stan Ferguson, established Ernabella Station in the early 1930s, Allan worked there until Stan sold the station to the Presbyterian Church in about 1936, which later established a mission there.

In December 1932 - January 1933 Brumby was engaged by HH Finlayson, the Honorary Curator of Mammals at the South Australian Museum, as camel-man and guide on Finlayson's collecting expedition to the Musgrave Ranges. Later that same year Brumby was hired by NB Tindale as guide and camel-man for Tindale's expedition through the Musgrave and Mann Ranges.

In 1935 Brumby left his Aboriginal wife and children and married a white woman, Beatrice Lennon. Brumby's part-Aboriginal children were later admitted to Colebrook Home at Quorn. In the 1940s Brumby and his wife worked on pastoral stations in the far north and from about 1947 to 1955 he was the manager of Sundown and Mt Cavenagh Stations, which were owned by Rex Lowe. In the late 1950s Brumby was the publican at the Finke Hotel.

Allan Brumby provided some notes on the Pitjantjatjara of north-western South Australia to NB Tindale in 1933 - see AA 338/2/25. There are references to Brumby in Tindale's 1933 Mann Ranges Journal (AA 338/1/9) and in Tindale's journal for his 1957 trip to North-West South Australia (AA 338/1/22/1).