Riding on Duminy's blistering knock, South Africa scored 70 runs in the last five overs to post a challenging 170 for six after being invited to bat in the Group 1 match.
The left-hander single-handedly took his side to a formidable total, hammering 10 fours and three sixes in his 43-ball knock.
The Kiwis were cruising towards the target, with Ross Taylor (62) and Kane Williamson (51) batting superbly. However, leg-spinner Imran Tahir (2-27) and Steyn (4-17) dealt crucial blows at key moments to turn the tide South Africa's way.
New Zealand needed seven runs off the last over, in which Steyn scalped two batsmen, effected a run-out and conceded just a boundary as the Kiwis ended up with 168 for eight.
The Kiws found Steyn too hot to handle as he dismissed Luke Ronchi off the first ball. He next bowled two dot balls, conceded a four on the fourth and dismissed Nathan McCullum with the fifth.
Taylor was on strike with his side needing three to win off the last ball but he could just connect the ball and ended up running himself out.
South Africa had lost their first match to Sri Lanka while New Zealand had defeated England.
The Kiwis had a steady start with openers Williamson and Martin Guptill (22) raising a 57-run stand in seven overs.
Williamson was the chief aggressor as he broke the shackles by punishing pacer Morne Morkel. Williamson smashed Morkel for a six and two fours to collect 16 runs from the fifth over.
Guptill then creamed off three boundaries in the next over, bowled by Lonwabo Tsotsobe.
With nothing going South Africa's way, skipper Faf du Plessis handed the ball to Albie Morkel and medium pacer responded to the SOS call with a wicket in his very first ball.
Guptill miscued a pull, giving a simple catch to wicket-keeper Quinton de Kock.
Tahir had Brendon McCullum stumped but Taylor was batting superbly from the other end. He hit three consecutive sixes off Morne Morkel.
Soon after completing his half-century Williamson became Steyn's first victim but New Zealand were comfortably placed at 140 for three in 16 overs at that stage.
Tahir bowled a sensational 17th over, in which he conceded just two runs and dismissed Colin Munro.
However, Taylor and Ronchi reeled off 14 runs off Morne Morkel in the
19th over to yet again swing the match their way, before Steyn produced an exceptional final over and kept his side in the hunt for the semi-finals.
Earlier, South Africa's top-order struggled in the face some disciplined bowling by the Kiwi bowlers. Only opener Hashim Amla resisted the attack with aplomb, hitting a 40-ball 41 with two boundaries.
New Zealand’s fast bowlers -- Corey Anderson, Tim Southee and Kyle Mills -- were terrific but Duminy changed the complexion of the match with his fearless batting.
Image: Dale Steyn of South Africa celebrates running out Ross Taylor of New Zealand off the last ball of the match.
Photograph: Gareth Copley/Getty Images
Broad fined for public criticism of match officials
10 vital World T20 stats you must know!
'Shami is the best fast bowler that India has at the moment'
Virender Sehwag starts new innings as an entrepreneur
WT20 PHOTOS: Sri Lanka edge past South Africa