Speaking against the motion at a session entitled 'Should Freedom of Speech be Absolute?' Kher alleged that a perception of intolerance was being created at festivals like JLF, adding that there should not be an impression that people in the country are living in fear.
"There is a perception of intolerance being created at festivals like these. Freedom of expression comes with a sense of responsibility in every citizen of the country. The rules which you follow at home (freedom of speech), you should follow in the country as well," he said.
The actor, who was selected for the Padma Bhushan, said India is one of the countries with an "expansive freedom of expression".
"Let's not create this impression that we're living in a country, where we have to live in fear and that there is no freedom of expression here and that people are going to come to your home to arrest you and then put you in jail," he said.
Kher said that it was only in this country that one could call the prime minister a coward or a psychopath and get away with it whereas when he went abroad he was "subjected to a lot of checking, where I have to take off my shoes and sometimes even my pants".
Hitting out at Kher for apparently targeting Delhi Chief Minister Arvind Kejriwal, Aam Aadmi Party minister Mishra said that prime minister was not the only person who could speak his 'Mann Ki Baat'.
"Can only one man do his 'Mann Ki Baat' in the country? Everybody can do it. Politicians must not tell me what I write on Twitter or Facebook. Those very people who've assumed the position of the country's culture and religion have no knowledge of our 'dharma'," he said.
Mishra said everyone has the right to chant Modi's name without being afraid of being jailed, to which the crowd responded by chanting the prime minister's name.
Vigorously leading the chants was Kher, who was described by fellow panelist Salil Tripathi as a "bad music conductor".
Janata Dal-United MP Pavan K Varma posited that reasonable restrictions were required on freedom of expression, adding that if everybody exercised their right to speak without restraint, democracy would collapse.
Madhu Trehan and Suhel Seth also spoke against the motion stating a danger of fringe elements hijacking the discourse, and cited examples of figures like US politician Donald Trump.
Dalit writer P Sivakami brought forth view that freedom of press is essential to bring forth the narratives of those who have been subjugated by the upper echelons of the society.
JLF matters
'A tradition which prevents women at temples needs to change'
1984 riots: SC asks Centre to file status report on SIT probe
Tuki vows to 'seek justice' as Arunachal comes under Prez's rule
Odd-even scheme kicks off, CM says overwhelmed by response