VHP president Ashok Singhal warned UP government of "adverse consequences" if it used force to enforce the ban.
Singhal said Yadav was receptive to them when he met him on August 17 to seek his party's support for the 20-day yatra.
"We were shocked when we came to know of the government's decision. The leader representing Muslims in his party had showed his displeasure over the meeting and Yadav apparently succumbed under pressure," Singhal said in an apparent reference to Khan, who is a minister in UP government.
Khan had expressed disapproval over the meeting and said it would send wrong signals to Muslim community.
The UP government on Monday banned the '84-kosi padyatra' citing security reasons.
It had suspected political reasons behind the VHP's decision as elections are due early next year.
Singhal claimed that there was no politics behind it, as their agitation for Ram temple in Ayodhya had been continuing for over six decades.
He said saints across the country have been informed of the yatra, which seeks to pass through six eastern UP districts in an attempt to garner support for the temple, and it could not be cancelled.
UP government bans VHP's Ram Temple yatra from Ayodhya
SC to hear Durga suspension issue on Aug 12
Samajwadi Party to back Food Bill, to move amendments
SP demand for special status poll gimmick, says Mayawati
Congress to push for Sonia's pet Food Bill in Lok Sabha today