The prominent economist also criticised Modi's model of governance, saying he did not approve of it.
"Yes, I don't want him," Sen told CNN-IBN in reply to a question on whether he wanted him as his prime minister.
"As an Indian citizen, I don't want Modi as my PM. He has not done enough to make minorities feel safe," he said.
Sen elaborated on his comment by stating, "He could have, first of all, been more secular and he could have made the minority community feel more secure."
"No, I don't approve of it. I don't think the record is very good. I think I don't have to be a member of the minority in order to feel insecure. We Indians don't want a situation where the minority feel insecure and could legitimately think that there was an organised violence against them in 2002. I think that is a terrible record and I don't think the Indian prime minister as an Indian citizen, who has that kind of record. No, I do not."
Sen said the physical infrastructure in Gujarat may be good but Modi has not done enough for minorities or for the majority population. He also said that the Gujarat model needs to do much more on the health and education sectors.
He said Modi could have made the majority community feel that they are not maltreating the minority and going against the Indian tradition of being tolerant.
"He could have also taken both of the facts -- that Gujarat’s record in education and healthcare is pretty bad and he has to concentrate on that -- as much as he is concentrating quietly on the physical infrastructure," the Nobel laureate said.
BJP recently anointed Modi as the chairman of its election campaign committee. He is widely believed to be the party’s choice for the prime ministerial candidate in the 2014 Lok Sabha polls.
Modi draws blood, triggers fresh war of words
Rediff Ballot: 76 pc pick Modi as next PM, 5 pc for Rahul
Readers' agenda for Modi: 'Don't speak of Ram Mandir'
Readers' agenda for Modi: 'Prove to Muslims you are secular'
Why it is wrong to compare Rahul Gandhi with Modi