The world champions beat the hosts 4-2 in the seven-match series earlier this month, which saw heated exchanges between players of both teams and racial taunts directed towards Australia's Andrew Symonds by spectators in Baroda and Mumbai.
"While there is no question that the blame rests with the management of both teams for letting it descend to such levels, the ICC match referee and the umpires are no less culpable," Gavaskar wrote in his syndicated column on Sunday.
"This protocol of the match referee acting only if the umpires make a report is just not on, for why then have a match referee?" the former India captain added.
"He is there not just to protect the umpires from the players but also to see that the game goes on without any untoward incidents."
Englishman Chris Broad was the match referee for the series and the one-off Twenty20 international on Saturday which the hosts won by seven wickets.
"One is not privy to the report that the match referee may have sent to ICC but the fact that not one player from both sides has been reported and reprimanded shows that the referee and the umpires did not do the job assigned to them."
Gavaskar said there had seldom been a series as ill-tempered as the recently-completed one.
"It would be sad if the ICC turns a blind eye to what happened during the series," he added.
"While accepting that the game has changed and become far more aggressive than before, what was seen on the cricket field did not do any good to the image of the game."