Umpire Ian Gould signalled five runs to Pakistan after talking to South African captain Graeme Smith during Pakistan's second innings.
The decision meant the umpires had decided the ball's condition had been changed contrary to the rules of the game.
Pakistan were struggling at 62 for three with Dale Steyn about to start the 31st over when the umpires took notice of the ball and took the decision.
Under International Cricket Council rule 42.1 on tampering, if an umpire spots a change in the condition of the ball but doesn't know who is the culprit, he will issue a first and final warning to the captain and the ball will be changed.
In case the ball is tampered with again, the bowling team will face a five-run penalty and umpires will not only change the ball again but the captain will be held responsible and reported.
But in this case the umpires saw Du Plessis rubbing the ball in an apparent move to tamper with the ball and they called Smith and docked five penalty runs.
An ICC spokesman confirmed the incident.
"As per 42.1 of the ICC playing conditions, the umpires replaced the ball and fined South Africa five penalty runs for ball tampering," said an ICC spokesman.
Pakistan is the only other team to have been penalised with five penalty runs, during the infamous Oval Test against England in 2006.
In a sharp reaction to Australian umpire Darrell Hair's decision, Pakistan captain Inzamam-ul Haq walked off the field.
Hair then ruled that Pakistan had forfeited the match -- the only such instance in Test cricket history.
Graeme Smith double hundred in SA run-feast against Pak
Smith becomes second South African to cross 9000 runs in Tests
SA in command after Tahir sends Pakistan crashing for 99
India is one of the 5 fragile economies: US official
South Africa could miss Steyn, Amla for 2nd Test vs Pak