Ballmer said in a statement that he would have timed his retirement in the middle of Microsoft's announced transformation to a devices and services company. But he said: "We need a CEO who will be here longer term for this new direction."
The software company said its board had appointed a special committee to direct the process of appointing a new CEO.
The committee is chaired by John Thompson, the board's lead independent director, and includes Microsoft founder and Chairman Bill Gates, as well as other board members Chuck Noski and Steve Luczo.
It will consider both external and internal candidates and is working with executive recruiting firm Heidrick & Struggles International Inc, according to the company.
Microsoft shares rose 8.9 percent to $35.27 in premarket trading.
(Reporting by Sinead Carew; Editing by Jeffrey Benkoe and Lisa Von Ahn)
The faces behind Microsoft's reorganisation
Giving by force
Tycoons who won't leave their fortunes to their kids
With PC sales in a free fall, where can Microsoft go?
Microsoft revamp to keep up with technological changes