Axis Bank's current chairman and CEO P J Nayak did not agree to the board's decision and told the directors he would leave the bank forthwith.
The appointment of Sharma, who is MD of ICICI Prudential Life Insurance, would be for a period of five years, Axis Bank said in a statement.
The appointment is subject to the approval of the Reserve Bank of India and shareholders of Axis Bank.