South Africa reached 122 for five in the 17th over in pursuit of the 153 required for victory against India, sending Pakistan into the last four on better net run rate along with Australia.
Sri Lanka will play Pakistan and Australia face West Indies in the semi-finals.
Earlier on Tuesday, Pakistan handed a shock 32-run loss to Australia, who qualified regardless on net run-rate.
Pakistan made 149-6 after being sent in to bat in the group two Super Eights match at the Premadasa stadium, before a five-man spin attack restricted Australia to 117-7.Prolific spinner Saeed Ajmal claimed 3-17, but it was 20-year-old Raza Hasan who was declared man of the match for conceding just 14 runs and taking two wickets in four overs of steady left-arm spin.
Both teams ended the Super Eights on four points each, which knocked winless South Africa out of the tournament.
“I am grateful to see the effort put up by the boys today,” said Pakistan captain Mohammad Hafeez.
“It was a big game for us and we needed to be at our best against a side like Australia.
“Once we got a good total, we knew we will do well because there was going to be help for the bowlers. Raza Hasan was brilliant, a great find for us.”
Australia advanced to the semi-finals despite the defeat due to having the best net run-rate in the group. The result left India needing an emphatic win over South Africa later on Tuesday to stop Pakistan's progress.
Sri Lanka and the West Indies have already qualified from group one.
Australia, who have never won the World Twenty20 despite taking four World Cup titles in the 50-over game, had breezed through four consecutive matches in the tournament before being halted by Pakistan.
“Pakistan outplayed us,” Australian captain George Bailey admitted.
“The ball turned a lot more than before and they used spin against us very well.
“The challenge for us now will be to get the winning form back in the semis.”
Veteran Mike Hussey was Australia's mainstay with an unbeaten 54 off 47 balls, as none of the other batsmen managed to cross 15 against the rampaging spinners.
Australia never looked comfortable after their in-form openers, Shane Watson and David Warner, were dismissed by the fifth over with only 19 runs on the board.
Australia, who needed to make 112 or more to qualify, achieved the target when Hussey cut seamer Umar Gul to the fence off the first ball of the final over.
Gul's two overs at the end were the only ones by a seamer as the spinners shared 18 overs between themselves.
Pakistan's innings revolved around left-hander Nasir Jamshed's 55 off 46 balls, which contained four boundaries and two sixes.
After the openers were removed by the sixth over, Jamshed put on 79 off 55 balls for the third wicket with Kamran Akmal, who made 32 before both batsmen were dismissed in the space of three deliveries.
Abdul Razzaq marked his return to the side in place of Yasir Arafat to make 22 from 17 balls. Left-arm fast bowler Mitchell Starc claimed 3-20 for Australia.