The 25-year-old Adler will undergo rib surgery and Rolfes, his Bayer Leverkusen team mate, has failed to recover from a knee injury which has kept him out of action for three months.
Adler became Germany's first-choice keeper after Robert Enke committed suicide last year and he has won nine caps.
"This was the most difficult decision of my life," Adler said in a statement. "But it would have been irresponsible for me, my club and the national team, to take part in the World Cup."
"Medically it was possible but the pain is so strong that I would not be able to be in top shape for training or the matches."
Werder Bremen's Tim Wiese and Schalke 04's Manuel Neuer, who have made two international appearances each, should be in the preliminary squad named on Thursday.
Coach Joachim Loew will also have to choose another keeper for the tournament in South Africa which starts next month.
BUTT RETURN?
Borussia Dortmund's Roman Weidenfeller is among the candidates along with experienced Bayern Munich player Hans-Joerg Butt.
The 35-year-old Butt, who played in the 2002 Champions League final with Bayer Leverkusen, has had a superb season with his club and was in the Germany squad at the 2002 World Cup and Euro 2000.
"We were informed of (Alder's planned operation) this morning and we are naturally very sad that Rene has to undergo surgery and will miss the World Cup," Loew said.
"We will now sit down with the coaching staff to discuss which keeper will travel to South Africa and will announce it on Thursday."
Rolfes, the 28-year-old Leverkusen captain, has been struggling for fitness since undergoing surgery in January.
"I am very satisfied with the healing process but it is clear from the schedule that it would make no sense to take part in the World Cup," said Rolfes, who has 21 international caps.
"My plan is now to be fully fit for the start of the new Bundesliga season. The decision to pull out did not come easy. Now there is nothing left but to wish the team success in South Africa."
Loew said Rolfes has made the right decision.
"We are sad that he had to pull out but wish him a speedy recovery," Loew said.
Germany have been drawn in World Cup Group D alongside Serbia, Australia and Ghana. Their first match is against Australia on June 13.