Australia ended the third day of the third Test on 102 for two, still trailing South Africa by 340 runs with eight wickets remaining in Cape Town on Saturday.
Australia, who clinched the series by winning the first two Tests, were bowled out for 209 before South Africa made 651, their highest ever score against the touring side.
Captain Ricky Ponting's poor match continued as he followed his first-innings duck with just 12, edging an expansive drive at a delivery outside off stump from Dale Steyn to be caught behind.
Opening batsman Phillip Hughes blazed his way to 32 before a hesitant prod at spinner Paul Harris's third delivery was edged to skipper Jacques Kallis at slip.
Fellow opener Simon Katich and Michael Hussey then survived an anxious last hour with Katich ending on 44 and Hussey on 13.
Steyn, who made the batsmen jump in a fiery second spell, finished with one for 32 in nine overs.
Earlier, AB de Villiers' innings of 163 led the way as South Africa amassed a huge score.
South resumed after lunch on 508 for six and De Villiers hit four consecutive sixes off medium-pacer Andrew McDonald, becoming the third batsman after Indian Kapil Dev and Pakistan's Shahid Afridi to achieve the feat in Tests.
Albie Morkel was bowled for 58 but De Villiers then received staunch support from Paul Harris (27) and reached his 150 in 285 minutes off 181 balls.
Ponting brought on occasional spinner Katich in the 150th over and he dismissed De Villiers in his second over, the batsman sending a pull shot to deep mid-wicket.
Katich had Steyn caught at slip for a duck in the same over and South Africa's record-breaking innings, beating their previous best against Australia of 622 for nine declared in Durban in 1969-70, ended when Harris was caught behind off Mitchell Johnson.
Ashwell Prince also made 150 and Jacques Kallis 102 for South Africa. Australia conceded 62 extras, their highest ever tally in a Test innings.
