Rai, 68, was suffering from brain cancer for the past one year and was admitted at Manipal Hospital on Old Airport Road, where he died at 2.30 am, the sources said. Rai is survived by two sons.
Born into a Tulu-speaking Bunt family in Dakshina Kannada's Puttur town, Rai entered the crime world at a very young age.
Karnataka police issued arrest warrants against Rai in eight cases, including murder and conspiracy.
In 2002, Rai was deported to India from the United Arab Emirates. On arrival, he was questioned by various investigation agencies such as the Central Bureau of Investigation, Research and Analysis Wing and Intelligence Bureau, and by the Karnataka police. He was later acquitted due to the lack of evidences.
In his reformed years, Rai founded charitable organisation 'Jaya Karnataka'.
Rai has appeared in Tulu film 'Kanchilda Baale' in 2011 and Kannada film 'Katari Veera Surasundarangi' in 2012.
Bollywood director Ram Gopal Varma wanted to make a movie based on Rai's life. The film roped in actor Vivek Oberoi for the lead role and was shot in Bengaluru, Mangaluru, Mumbai, Dubai and London. However, it has not been released due to production delays.
Rai was enthusiastic about the film and even celebrated his birthday with Varma and Oberoi.
After his cancer diagnosis, Rai withdrew from public life and resigned from Jaya Karnataka.
His last rites are likely to be performed at Bidadi on Friday, family sources said.
Media's fatal obsession with the underworld
3 sectors where corporate-underworld nexus flourishes
How underworld funds Indian Mujahideen's deadly plans
Mumbai's underworld is back in business
Tales from the underworld