Justice Ganguly, who is now heading the WBHRC, has been indicted by a Supreme Court panel for "unwelcome behaviour" and "conduct of sexual nature" against a law intern who alleged that he had made advances towards her in a hotel room in Delhi in December last year.
Raising the issue n the Lok Sabha, Leader of the Opposition Sushma Swaraj said the relation between "guardian and ward" is of trust and faith and any breach cannot be pardoned.
She said since Ganguly was holding a high position, he should not be allowed to go scot free.
"On ethical grounds he should have resigned by now. But he is stubbornly sticking to his post. He should immediately resign or else he should be removed from his post," she said.
Referring to the December 16, 2012 incident of gang rape of a girl in Delhi, Swaraj said at that time the whole country was outraged over the incident and a stricter law was passed with the hope that such a crime would not be repeated.
Trinamool Congress leader Sudip Bandopadhya endorsed Swaraj's view saying Ganguly should resign immediately.
He said such crimes cannot be tolerated and such persons should not be allowed to occupy a high posts.
In Kolkata, Justice Ganguly, who has made it clear that he will not step down, today refused to comment on demands in Parliament for his removal from the post of chairman of WBHRC.
"I will not say anything about this," Justice Ganguly said when asked for his reaction on the uproar in Parliament.
"I have denied the allegations, I have nothing more to say," Justice Ganguly told PTI.
Apart from the BJP, Trinamool Congress has also been very vocal demanding his removal.
West Bengal Chief Minister Mamata Banerjee, who had chosen Justice Ganguly as the chairman of WBHRC after coming to power in 2011, has written twice to President Pranab Mukherjee seeking strong action against the former judge.
Justice Ganguly hits out at media: My resignation none of your business
Not unlike foe BJP, Cong govt had wanted gay sex to remain illegal
Amended Lokpal Bill tabled in Rajya Sabha
BJP is eyeing Vijayakanth and Ramadoss as TN allies
Justice Ganguly rules out resignation from WBHRC