Germany drew their first two group matches, including a 0-0 result with debutants Latvia, before going down to the Czech Republic 2-1 on Wednesday in Group D. The Czechs were playing their second string team.
The 44-year-old Voeller said the national side needed a new start before the 2006 World Cup in Germany.
"I have the feeling that only someone who is untarnished and has a certain credibility -- like I had four years ago -- can do the job over the next two years," he told a news conference.
Voeller told team officials overnight and they had asked him not to make a hasty decision.
"He told us it was not a spur-of-the-moment decision taken just after the game. We regret this decision very much but we have to accept it," said German federation president Gerhard Mayer-Vorfelder.
Voeller had a contract until the World Cup in 2006. His predecessor, Erich Ribbeck, resigned after Germany were knocked out of the 2000 European Championship in the first round.
Ottmar Hitzfeld, recently sacked by Bayern Munich, is favourite to replace Voeller. Greece manager Otto Rehhagel and Christoph Daum of Turkey's Fenerbahce are also in the frame.
A World Cup winner as a player in 1990, Voeller restored Germany's pride with a surprise run to the 2002
POOR RUN-UP
Germany struggled to qualify for Euro 2004 and suffered a poor run-up to the tournament, including a 5-1 thrashing by Romania.
Voeller was jeered by German fans after the Czech defeat and was close to tears in the post-match news conference.
"There was too little yesterday," Voeller said, referring to the match. "I had it in my head already that I would probably not go on.
"It would be fatal to hang on to the job and carry on regardless. "I told the players last night 'Life goes on'."
Voeller is the first coach to fall on his sword at the European Championship. However, the coaches of two other traditional powers who were knocked out in the first round, Spain and Italy, could lose their jobs in the next 36 hours.
Inaki Saez, the coach of Spain, has tendered his resignation, according to media. Italy coach Giovanni Trapattoni could have his contract ended when his country's soccer federation meets on Friday.
The quarter-finals of Euro 2004 start on Thursday with the game between hosts Portugal and England.