The composer, who has already won a Golden Globe and two Academy awards, registered wins in both the categories he was nominated for.
He was the earliest winner in the Best Compilation Soundtrack for a motion picture category and picked up another trophy for his song Jai Ho in the Best Motion Picture song category moments later.
"This is insane, God great again," the 44-year-old composer said while accepting the golden gramophones.
Rahman beat Steve Jordan for Cadillac Records, Quentin Tarantino for Inglourious Basterds, the producers of Twilight and True Blood to bag the award in the best compilation soundtrack for a motion picture category.
In the Best Song category, Rahman's rivals included Bruce Springsteen for his song The Wrestler, from the Oscar-nominated movie of the same name.
Rahman's Grammy win was similar to his triumph at the Oscars last year where he picked up the Best Original Score statuette before winning the Oscar for Jai Ho, which has become an international anthem.
This is Rahman's first Grammy.
In the picture: Rahman poses with singers Tanvi Shah and Vijay Prakash at the 52nd annual Grammy Awards in Los Angeles. Photograph: Mario Anzuoni/ Reuters
Video: Rahman thanks his audience
Want to listen to the songs? Click here to listen to Jai Ho and O Saya.
If you want to become a fan of Rahman, just click here!
Here's your chance to congratulate the legend on his own Fan Page. Wish him!
Congratulate Rahman on his Grammy wins!
Mallika Sherawat in Los Angeles
Vinnaithaandi Varuvaaya: Vintage Rahman
Pix: A R Rahman captivates Australia
Spotted: Shikha Uberoi in Los Angeles