Sen also hit out at the Modi government over its "interference" in academia, saying it has never happened on this scale now and rued the "cuts" in the country's health and education budget. "The academic interference that we see now has never really happened in India on this scale."
81-year old Sen replied in the affirmative when asked whether he was still skeptical about Modi's secular credentials after his articulation of secularism as "India First" and "Sabka saath, sabka vikas" (harmony with all, development for all).
"Yes, the short answer is yes," he told India Today TV channel
During the 2014 Lok Sabha poll campaign, Sen had said he would not like to see the Gujarat Chief Minister as the country's prime minister.
In controversial comments, Sen was then quoted as saying, "As an Indian citizen I don't want Modi as my PM, he has not done enough to make the minorities feel safe."
Sen on Tuesday said he stood by his reservations to Modi donning the mantle of prime ministership.
"There is no particular reason to change it because the fear that I had that some of the worst aspects of the previous government, neglect of basic education, neglect of healthcare, neglect of nourishment would continue and intensify which is actually happening, in the budget as cut in funds...
And on top of that the problem of secularism would become more intense under the 'hindutva rule'. And particularly given the Gujarat riots controversy, my confidence in his (Modi's) ability to be a great defender of secular values wasn't there," Sen said.
Stating that he would like to see more action in terms of reaching out to the minorities, Sen said it should be a situation where there is an end to episodes like "ghar wapsi", church attacks and giving preference to "Hindus point of view" in the work of Indian Council of Historical research..."
"Then I think that would be an appropriate claim (on upholding secularism," he said.
Modi said it is the duty of the state to protect every religion not to allow things to happen like what happened in 2002 Gujarat riots.
Sen said there is a totality of a picture of "academic interference" as witnessed in developments involving institutions like Indian Council of Cultural Relations, Indian Council of Historical Research, National Book Trust, Tata Institute of Fundamental Research, a number of the IITs as well as the controversial bill relating to Indian Institutes of Management.
Sen, who in the past has minced no words in his criticism of the Bharatiya Janata Party-led National Democratic Alliance government, also spoke about what he called his "ouster" from the Chancellorship of Nalanda University but declined to comment whether it was linked to his controversial comments against Modi during the Lok Sabhapoll campaign.
He is slated to step down as Chancellor of Nalanda on July 17, when he will be replaced by George Yeo, former foreign minister of Singapore.
"The Board (of the University) wanted me to continue unanimously, the government did not and it became clear to me that if I succeed in staying on which many of the board members wanted me to go on fighting, I'll be making a mistake because the important cause is Nalanda university and I could not be an effective leader of Nalanda with government hostility," he said.
Sen is reported to have written about his exit from NalandaUniversity in a 4,000 word candid essay about to be published in the August issue of the New York Review of Books.
Sen's forthcoming book 'The Country of First Boys', a collection of essays, also dwells on his experiences at Nalanda.
Referring to the much touted Gujarat model of development, Sen said the physical infrastructure in the western state may be good but it had not done enough in social sectors like education, health care and gender parity.