Former England wicketkeeper Downton, 58, took over the role in February, 2014 following the team's 5-0 defeat in the Ashes series in Australia.
He made the decision to fire Alastair Cook as one-day captain in December and the team endured a dismal World Cup in Australia and New Zealand this year, failing to get through the group stage.
"The England Cricket Department needs to deliver performance at the highest level and our structure needs to be accountable for reaching the standards we aspire to," ECB chief executive Tom Harrison said in a statement.
A new leadership position focused on performance will be created.
"The new role we are putting in place will deliver an environment where world-class performance is at the heart of everything we do," Harrison said before paying tribute to Downton's achievements.
“Paul is a man of great integrity who has worked extremely hard to make a difference at the ECB," Harrison added.
"He joined at a very difficult time, but under his leadership the Test team have made significant strides. We thank him for his hard work, drive and determination and wish him every success for the future."
England start the first of three Tests against West Indies in Antigua on Monday before a home series against New Zealand and Australia.
Image: Paul Downton along with coach Peter Moores
Photograph: Gareth Copley/Getty Images
Sangakkara, Lanning named Wisden Cricketer of the World
We gave away that game in the end: Van Ass
Put England coach Moores in charge of kids: Vaughan
Pietersen not in our focus, says England coach ahead of WI tour
World Cup Blog: England the laughing stock