"We have not done anything wrong and why should we worry," a source quoted the prime minister as saying during a meeting with some ministers.
The sources said the prime minister suggested that the ministers should communicate with the people directly instead.
The ministers were sitting with the prime minister to discuss the current political situation with Rajya Sabha not functioning over the row on conversions.
The prime minister's comments came amid demands from the opposition in the Rajya Sabha that he should come to the House and reply to the debate on rising incidents of communal violence in the country and proclaiming that till this happens
Parliamentary Affairs Minister M Venkaiah Naidu denied so when asked if there is any proposal of the prime minister making a statement in Parliament to end the current impasse.
"There is no agenda for the prime minister to make any statement. There is no such notice to the prime minister or the prime minister accepting any notice. There is no proposal at all. The prime minister is coming to Rajya Sabha during Question Hour," Naidu told reporters.
Asked about the fate of the Insurance Bill, which was listed for the day in the Upper House but could not be taken up after repeated disruptions by the opposition, Naidu exuded confidence of getting it passed along with other reform legislations.
"I am confident. At the end of the day, they will realize that people's sympathy is paramount than political differences. They will cooperate with the government and allow the House to function in the people's interest. As parliamentary affairs minister, I am still confident," he said.
Communal row washes out Rajya Sabha proceedings
Singhal crosses 'Lakshman rekha'; says reconversion process to continue
TMC members allege misuse of CBI; stage walkout in LS
Indian-American Vivek Murthy becomes youngest surgeon general
Rahul Gandhi sidelines senior Congress leaders, allege partymen