In the hastily-arranged late night party leadership ballot, 57-year-old Abbott, who had been plagued by poor opinion polls, received 44 votes against Turnbull's 54.
Turnbull's victory is reminiscent of the coup former prime minister Julia Gillard staged against Kevin Rudd in 2010 and makes the former communications minister Australia's fifth prime minister in just over five years.
Liberal MPs also voted for Julie Bishop to remain deputy leader of the party.
Turnbull 60, is expected to be sworn in after Abbott writes to the Governor General and resigns.
Turnbull, who will be the 29th prime minister of the country, resigned from cabinet this afternoon at the question time and told Abbott he would challenge for the leadership.
He said the prime minister was incapable of "providing the economic leadership the country needs". Turnbull said he had been under sustained pressure to put his name forward.
"Now this is not a decision that anyone could take lightly. I have consulted with many, many colleagues, many Australians, many of our supporters in every walk of life," Turnbull said adding "This course of action has been urged on me by many people over a long period of time".
"It is clear enough that the government is not successful in providing the economic leadership that we need. It is not the fault of individual ministers. Ultimately, (Tony Abbott) has not been capable of providing the economic leadership our nation needs. He has not been capable of providing the economic confidence that business needs," Turnbull said.
"The prime ministership of this country is not a prize or a plaything to be demanded," Abbott said at Parliament house on Monday after Turnbull advised him of the challenge and his decision to quit cabinet.
"I am dismayed by the destabilisation that's been taking place now for many, many months and I do say to my fellow Liberals that the destabilisation just has to stop," he said.
"I firmly believe that our party is better than this, that our government is better than this and, by God, that our country is so much better than this," Abbott said.
Earlier on Monday, at a press conference, Turnbull said if Abbott remained as leader, the coalition government would lose the next election.
He said he had not taken the decision lightly, but that it was "clear enough that the government is not successful in providing the economic leadership that we need" and that Australia needed a new style of leadership.
The man who stole India's past
New MH17 video shows militia shot down plane by mistake
They did not deserve to be shot out of the sky: Australia mourns MH17 crash
What's the latest on Europe's migrant crisis
Adani's mine project: India conveys disappointment to Oz