The South African captain fractured his right middle finger when he took a brilliant catch to dismiss Delhi Daredevils opener Virender Sehwag, in Ahmedabad, on Monday..
England's Mascarenhas injured his ankle when he attempted a quick second run but was run out.
Rajasthan captain Shane Warne confirmed both players would be returning home after being ruled out of the tournament.
"Unfortunately [Graeme] Smith and [Mascarenhas] Dimma [are] going home due [to] injury. Big setback for us as Smith and Dimma are very good players for Royals and their experience will be hugely missed," Warne said on his Twitter page.
Smith also revealed that he was disappointed to be ruled out of the mega event after playing just two games.
"I have a double fracture in my right middle finger, so will be missing the IPL. I hurt it taking the catch off Sehwag last night when I landed my finger got hooked in the ground," Smith wrote on his Twitter page.
The left-hander has suffered a series of finger injuries in the past few years and recently he was forced to miss the three-match One-Day International series against India because of a similar injury.
It comes as a major blow for Rajasthan Royals, who have lost their first two matches of the tournament, against Mumbai Indians and Delhi Daredevils.
They are already missing Ravindra Jadeja after he was banned from this year's tournament, while Shane Watson is busy with international duty in New Zealand.
Former Australia batsman Damien Martyn is expected to open the batting in place of Smith.
Photograph: Getty Images