"If I have hurt anyone, I express my regret," he said after Speaker Sumitra Mahajan told him that he should express regret though she cannot force him to do so.
He contended that his remarks made at a public rally in Kolkata last week was "in reaction to the BJP general secretary saying 'Bhaag Mamata Bhaag' insulting our leader."
He, however, added, "It was not my intention to insult the prime minister or show disrespect to Lal Bahadur Shastriji. I respect both of them."
Prior to this, BJP members created uproar over the remarks made by the Trinamool Congress deputy leader.
They rushed into the aisle and the front rows to demand apology.
Kirit Somaiya (BJP) said there should be "unconditional apology" from him for his remarks against Modi and former Prime Minister Lal Bahadur Shastri.
The issue was raised in the House yesterday too by S S Ahluwalia (BJP), after which Minister of State for Parliamentary Affairs Rajiv Pratap Rudy had warned of bringing a censure motion against the TMC leader if he remained "adamant" and failed to apologise.
After the protests during the Zero Hour on Wednesday, Banerjee said some members had objected to his statements "made outside the House", adding that it was a "political speech" which should not be discussed inside.
Somaiya immediately got up to say, "We want him to clearly apologise and express regret."
The Speaker, after hearing Parliamentary Affairs Minister M Venkaiah Naidu and Congress leader Mallikarjun Kharge, said "No one becomes small by seeking an apology."
She told Banerjee
Somaiya had given a notice for a censure motion against Banerjee, which the Speaker said was under her consideration.
Making his submission on the issue, the parliamentary affairs minister said "our minister (Sadhvi Niranjan Jyoti) had also apologised" for her statement outside Parliament but making a statement against the Prime Minister was a "very serious matter".
Demanding that the TMC member should tender unconditional apology, Naidu said if Banerjee wanted to contest the demand for an apology, the Speaker should get all the records of the "slogans raised inside the House, besides the videos of his speech made outside and take a decision as per the rules."
Maintaining that he had gone through books relating to parliamentary practices and precedents, Naidu said, "We cannot accept if someone says outside the House that the prime minister will be slapped."
Congress leader Mallikarjun Kharge said after the Speaker had "resolved the issue yesterday, I can't understand why Naiduji is again raking it up. Various comments are made during election time but all these are not discussed in the House."
"Discussing all statements made outside Parliament will set a very bad precedent and we should not set a bad precedent," he said.
SP chief Mulayam Singh Yadav agreed with BJP members that everyone should used decent language but expressed himself against the ruling party becoming "so angry."
Suggesting that it was but natural for opposition parties to become agitated, he said if ruling party becomes angry then it can inflict "a lot of damage on us as it is powerful".
Soon after Zero Hour, agitated BJP members were in the aisle shouting demanding an apology from Banerjee.
BJP cries foul over TMC MP's remark on Shastri, Modi
Despite PM's remarks, stalemate over Sadhvi's remarks persists
Sri Lanka president offers prayers at Tirumala temple
PM disapproves of minister's speech; SACK her, says oppn
Climate deal in Lima: India talks tough