The tourists now lead the series 2-0 with the fourth and final Test starting in Antigua on Friday.
Fast bowler Nel took a career best six for 32, his third five-wicket Test haul, as South Africa won by an innings and 86 runs.
Nel also scooped 10 for 88 in the match, his first 10-wicket Test haul.
The West Indies, who needed 252 to make South Africa bat again, were dismissed for 166 in their second innings after Graeme Smith's side declared their first innings on 548 for nine early in the morning session in reply to West Indies' 296.
The result gave South Africa their fourth consecutive series win over West Indies, and their second in the Caribbean after their 2001 conquest.
Nel struck in successive overs after lunch to remove the key pair of Brian Lara and Shivnarine Chanderpaul, extinguishing any home hopes the match could be saved.
Lara, who scored 13, was dismissed in the sixth over after lunch when he played back to a Nel inswinger.
The ball missed Lara's inside edge and struck him on the back leg and on line with his off stump, making umpire Billy Bowden raise a crooked index finger.
Nel has now dismissed Lara eight times in 12 innings.
In his next over Nel pitched a ball on the left-handed Chanderpaul's off stump.
The delivery straightened and appeared to hit Chanderpaul's front pad marginally outside the line of his off stump as he played back.
Bowden thought differently, and the captain was dismissed for 31 to reduce the West Indies to 63 for five.
They dwindled to 71 for six when Dwayne Bravo slashed at a wide delivery from fast bowler Jacques Kallis and was caught behind by wicketkeeper Mark Boucher for six.
Thirteen balls before tea Ryan Hinds was out for 15 when he top-edged a sweep off left-arm spinner Nicky Boje to Kallis at backward square leg.
Daren Powell survived until the eighth ball after tea before Nel trapped him in front for five.
Fidel Edwards then fell for two with a limp drive to Makhaya Ntini at mid off and Nel celebrated his fifth wicket.
The match ended when Courtney Browne pulled a ball from Nel to Boeta Dippenaar at deep square leg.
Browne stood unbowed amid the carnage with an aggressive 68, his maiden Test half-century, which he scored off 75 balls with five fours and four sixes.
South Africa seized control of the match before lunch when Nel and Ntini ripped through the top order, taking three wickets in the space of four deliveries for no runs.
The visitors had resumed day four on 521 for eight and batted on for just 5.5 overs in the first session to push their lead beyond 250 runs.
Scoreboard
West Indies 1st innings 296 (B. Lara 176, S. Chanderpaul 53; M. Zondeki 4-50, A. Nel 4-56)
South Africa 1st innings (Overnight: 521-8)
G. Smith c W. Hinds b Gayle 104
A. Villiers c Browne b King 178
B. Dippenaar run out 71
J. Kallis b W. Hinds 78
H. Gibbs c Bravo b Gayle 8
A. Prince c Bravo b Gayle 23
M. Boucher b Powell 28
N. Boje not out 20
M. Zondeki c Chanderpaul b Powell 2
M. Ntini c Bravo b Powell 16
A. Nel not out 4
Extras (b-1 lb-2 nb-13) 16
Total (for 9 wickets declared, 177.5 overs) 548
Fall of wickets: 1-191, 2-334, 3-392, 4-410, 5-450, 6-496, 7-502, 8-504, 9-542
Bowling:
Edwards 32.5 - 3 - 112 - 0 (nb-5) King 30 - 5 - 80 - 1 (nb-6) Powell 31 - 2 - 103 - 3 (nb-1) Bravo 29 - 3 - 73 - 0 (nb-1) R. Hinds 18 - 1 - 67 - 0 Gayle 27 - 3 - 85 - 3 W. Hinds 9 - 2 - 18 - 1 Sarwan 1 - 0 - 7 - 0
West Indies 2nd innings
C. Gayle c Smith b Ntini 5
W. Hinds c Gibbs b Nel 11
R. Sarwan c Gibbs b Ntini 0
B. Lara lbw b Nel 13
S. Chanderpaul lbw b Nel 31
D. Bravo c Boucher b Kallis 6
R. Hinds c Kallis b Boje 15
C. Browne c Dippenaar b Nel 68
D. Powell lbw b Nel 5
F. Edwards c Ntini b Nel 2
R. King not out 3
Extras (b-1 lb-2 nb-4) 7
Total (all out, 54.2 overs) 166
Fall of wickets: 1-17, 2-17, 3-17, 4-54, 5-63, 6-71, 7-107, 8-130, 9-143, 10-166
Bowling:
Nel 16.2 - 3 - 32 - 6 Ntini 11 - 2 - 40 - 2 (nb-1) Zondeki 8 - 3 - 43 - 0 (nb-3) Boje 11 - 1 - 33 - 1 Kallis 6 - 3 - 7 - 1 Smith 2 - 0 - 8 - 0
Result: South Africa won by an innings and 86 runs
