Lee injured his left ankle during Australia's practice match last week and missed the first Test which started on Monday. It was the same ankle from which he had bone spurs removed last year.
He started bowling in the nets on Tuesday but complained again of soreness and team physiotherapist Alex Kontouri decided to send Lee home for treatment by his foot and ankle consultant.
"Brett [Lee] is suffering from inflammation at the back of his ankle joint and as a result we felt it was prudent for him to see doctors in Australia," Kontouri said in a statement.
Lee was disappointed but hopeful he would return as soon as possible.
"Initially I felt pretty good after the training session but once the pain returned I knew I was in trouble," Lee said in a statement.
"I am really disappointed because I thought I was on the improve. I guess it's back to square one now to rectify the problem."
A replacement will be named once the selectors have met, team official said. Glenn McGrath may be called up having returned to first-class cricket from injury last weekend.
Kasprowicz sprained his shoulder after colliding with a ball boy towards the end of the second day's play in Galle. He will not bowl again in Sri Lanka's innings.
His condition will be monitored and Australia are hopeful that he will be able to bowl in the second innings.
Sri Lanka were bowled out for 381 in their first innings, in reply to Australia's 220, on the third day of first Test on Wednesday.
Scores:
Australia 220 ( D.Lehmann 63; M.Muralitharan 6-59);
Sri Lanka 381 ( M.Jayawardene 68, T.Dilshan 104; S.Warne 5-116).
