"These governors should have themselves quit. All of these people are appointed because they sat in Sonia Gandhi's durbar and propositioned her. There is no question of any of them coming in on merit," said Swamy.
B L Joshi resigned as Uttar Pradesh governor earlier in the day.
According to television reports, Prime Minister Narendra Modi is reported to be mulling over the idea of removing at least six Congress loyalist governors and replacing them with BJP appointees.
Among the governors likely to be told to go are Sheila Dikshit in Kerala, K Rosiah in Tamil Nadu, M K Narayan in West Bengal, Shivraj Patil in Punjab and Rajasthan, K Sankaranarayanan in Maharashtra.
-Agencies
Punjab CM raids offices of ministers, finds 8 of them 'absent'
India vows action, warns hoarders after May inflation spike
Does your state have enough toilets?
5 rapes in 36 hours, but UP cops are in denial mode
India confident of rescuing abducted aid worker in Afghanistan