Smith, at 22 South Africa's youngest ever captain, was quoted on the front page of the Cape Times newspaper on Tuesday following a speech he delivered at a business breakfast the previous day.
Last week Klusener was left out of the South African one-day and Test squads for the tour of England which begins next month.
A poor attitude in the dressing room was widely quoted in South African media as the principle reason for his omission.
"We decided Lance as a team man can only cause hassles and we want to move forward in SA cricket," Smith said at the breakfast.
"To be honest Lance, as fantastic as he is...can sometimes ruin a team.
"His ability as a cricketer is very good, but his ability as a team man is not very good and he kind of can infect a team and bring down the youth," Smith said.
Klusener was the player of the tournament at the 1999
Smith said he was confident his team would perform well in England without Klusener as South Africa look to win a series there for the first time since readmission to the international game in 1991.
South Africa drew 1-1 in 1994 and were controversially beaten 2-1 in 1998 after a series of umpiring decisions went against them in the deciding Test at Headingley.
"We are determined to go there and win the series -- a daunting task when looking at our two previous tours there," Smith said.
Prior to Smith's comments, Klusener said after hearing he had been dropped: "I am the way that I am. If personality gets in the way of statistics and performance, then so be it."
He added: "If the powers-that-be want to make decisions like that then so be it. It's their loss."
Klusener added that he had received offers from Australia and was also interested in playing county cricket in England.