"These governors should have themselves quit. All of these people are appointed because they sat in Sonia Gandhi's durbar and propositioned her. There is no question of any of them coming in on merit," said Swamy.
B L Joshi resigned as Uttar Pradesh governor earlier in the day.
According to television reports, Prime Minister Narendra Modi is reported to be mulling over the idea of removing at least six Congress loyalist governors and replacing them with BJP appointees.
Among the governors likely to be told to go are Sheila Dikshit in Kerala, K Rosiah in Tamil Nadu, M K Narayan in West Bengal, Shivraj Patil in Punjab and Rajasthan, K Sankaranarayanan in Maharashtra.
-Agencies