Andrew McDonald would act as interim head coach
Langer's departure came after he held contract discussions with Cricket Australia on Friday, with his current deal set to expire in June.
Australia also won last year's Twenty20 World Cup for the first time under Langer but there have been reports of player discontent over his coaching style.
Langer was reported to have reacted angrily during a recent meeting with Cricket Australia over his contract and had been asked to reapply for the role.
His management company, Dynamic Sports & Entertainment Group (DSEG), said his resignation was effective immediately.
"DSEG confirms that our client Justin Langer has this morning tendered his resignation as coach of the Australian men's cricket team," the company tweeted.
Cricket Australia said in a statement Langer was offered the extension on Friday.
"Justin was offered a short-term extension to his current contract, which sadly he has opted not to accept," it said, adding that Andrew McDonald would act as interim head coach with a Test tour of Pakistan just weeks away.
"The contract extension offered to Justin was the result of a thorough review process that evaluated many factors including future requirements of the team and the upcoming extensive schedule of fixtures."
"It included the opportunity to defend the T20 World Cup title in Australia at the end of this year. Justin informed CA this morning he was not accepting the offer and would resign with immediate effect."
Langer took over as head coach in the wake of the ball-tampering scandal in South Africa in 2018 and Cricket Australia CEO Nick Hockley said he had been 'outstanding' in the role.
"He has restored the trust in the team and his legacy is assured," he said.
Despite Australia's success, there had been speculation in recent months that Langer did not enjoy the full support of the dressing room.
Senior players and Cricket Australia managers met last year amid reports Langer's intense coaching style had rubbed players the wrong way.
Australia test captain Pat Cummins recently praised Langer for doing a 'fantastic' job but said it was only fair for Cricket Australia to conduct an evaluation process before making a decision on renewing his contract.
Former captain Ricky Ponting was scathing in the way the governing body had handled the matter.
"It's a really sad day as far as Australian cricket is concerned," he said on ABC Radio on Saturday.
"Reading the tea leaves, it seems, as he would say, a very small group of the playing group ... haven't loved the way he's gone about it."
"And that's been enough to force a man that's put his life, heart and soul into Australian cricket and done what I believe is a sensational job in turning around the culture and the way the Australian cricket team has been looked at over the last three or four years ... out of his dream job."
Langer, who earned 105 Test caps, said last month he was proud of how the team had been performing.
"Whatever happens from now on, we can all be incredibly proud of this little period," he said on Australian SEN Radio.
"We had two missions: to win the (Twenty20) World Cup and to win the Ashes."
Langer's resignation comes a day after his England counterpart Chris Silverwood stepped down from his position.
PIX: India train in right earnest ahead of WI ODIs
I am doing my job as BCCI president: Ganguly
Ganguly's special message for India women ahead of WC
Australia confirm first tour of Pakistan in 24 years
The one big challenge for Rohit as India captain...