"The simple truth is that I cannot continue to serve as foreign minister if I don't have Prime Minister Gillard's support," 54-year-old Rudd said at a media conference in Washington DC, which was broadcast live in Australia.
"I therefore believe the only honourable course of action is for me to resign," he said.
A "disappointed" Gillard, who ousted Rudd a prime minister in June 2010, said he did not inform her in advance that he would resign.
"I am disappointed that the concerns Mr Rudd has publicly expressed ... were never personally raised with me, nor did he contact me to discuss his resignation prior to his decision," 50-year-old Gillard, who was in Adelaide, said in a short statement.
Rudd's sudden resignation came amid signs that he was preparing for a showdown by challenging Gillard for leadership in the ruling Labor party's upcoming caucus this week. It also followed claims by supporters of the prime minister that she was prepared to sack the foreign minister for disloyalty.
"It's time for some plain speaking on this," Rudd said at the press conference at 2am local time. "The truth is I can only serve as foreign minister if I have the confidence of Prime Minister Gillard and her senior ministers."
"I feel very uncomfortable doing this from Washington rather than in Australia but I don't feel I have a choice," Rudd said.
Rudd said the reign of "faceless men" and their iron grip on the control of the Labor leadership must end.
Calling the leadership tussle a "soap opera", he said he would not have "anything to do with it".
Rudd, who would fly back to Australia on Thursday, said he would consult his family and electorate before deciding about his political future.
In his resignation speech, Rudd was highly critical of Labor Party's move to oust him as prime minister in 2010.
"I can promise you this; there is no way that I will ever be a party to a stealth attack on a sitting prime minister elected by the people," he said. "We all know that what happened then was wrong and it must never happen again."
Australia's Labor government has been torn by speculation about whether Rudd would mount a bid to return to the top job.
Rudd said that in recent days sections of the Labor party had publicly attacked his integrity and therefore his fitness to serve as a minister, a reference to the 'quit notice' served on him by former Labor leader Simon Crean.
"In recent days Mr (Simon) Crean and a number of other faceless men have publicly attacked my integrity and therefore my fitness to serve as a minister in the government. When challenged today on these attacks Prime Minister Gillard chose not to repudiate them. I can only reluctantly conclude that she therefore shares these views."
The globe-trotting diplomat said he would make a full statement on his future before Parliament resumes on Monday.
Image: Former Australian PM Kevin Rudd
'MNS is not against any community'
HC asks police to prosecute Mumbai Cong chief for graft
PIX: Bar Refaeli plays tennis in her underwear!
5 foreign players: Sahara needs consent of franchises
What are the Oscar odds: The Best Actress nominees