"Mullah Fazlullah, Taliban's top commander in the Swat valley, has ordered his fighters to withdraw from Buner district," Taliban spokesman Muslim Khan was quoted by Dawn TV as saying.
The decision by Taliban to withdraw came after meetings between Taliban leaders Qari Muhammad Khan and Muslim Khan and Malakand Divisional Commissioner Syed Muhammad Javed in the presence of Maulvi Sufi Mohammed, who acted as a mediator, TV channels reported.
Following the deal, Taliban militants started pulling out of Buner, which is just 100 kms away from the federal capital. The meeting with militant commanders came as Pakistan President Asif Ali Zardari assured the United States that Pakistan government will not allow anybody to challenge its writ or run a parallel administration in any part of the country.
Reacting to international concerns over reports of Taliban advancing towards the Pakistani capital, Prime Minister Yousuf Raza Gilani told Parliament that country's defence was in safe hands and its nuclear arsenal was secure.
Sufi, the chief of the banned Tehrik-e-Nifaz-e-Sharia Muhammadi, who negotiated the Swat deal had offered to persuade militants to leave the two districts of Buner and Shangla, which they had occupied in the past few days.
He was accompanied by senior clerics and Taliban commanders, reports said.
The move to get the Taliban vacate the two towns has assumed urgency as the US, which is highly critical of the Swat deal is putting on pressure on authorities in Islamabad to confront the Taliban.
Pakistan has already moved units of paramilitary, frontier constabulary into the region, who are now manning police stations and government buildings.
A confrontation appeared brewing as heavily armed Taliban militants continued to patrol Buner's main streets. Armed militants also moved into Shangla district, which is merely 60 kms from the federal capital.
Pakistani authorities have moved eight platoons of paramilitary frontier constabulary to the areas who are now taken over policing duties to instil confidence as Taliban presence had made provincial politicians and officials to leave the town.
In recent days, hundreds of Taliban militants had poured into Buner and Shangla from adjoining Swat valley, setting up check-posts, occupying mosques and warning people not to engage in un-Islamic activities and barring women from schools and streets.
200 Sikhs flee from Swat
Pakistan Parliament approves Shariah in Swat
'Swat deal shows how weak Pakistan has become'
Pakistan's Swat truce worrisome for India: Antony
Pakistan must take action against Taliban: US