The activist has been on an indefinite hunger strike for the passage of Jan Lokpal Bill in Parliament.
"Why are you interrupting him? If you want to create a ruckus, you can go out," Hazare told Rai.
"It is not right to interrupt when someone is speaking," he added.
Hazare publicly rebuked Rai, asking him to leave the venue if he wanted to disrupt the former army chief's speech.
Singh, who was sharing the dais with Hazare, said, "We should not split and form different groups in our fight against corruption."
The AAP leader raised some objections, which led to a heated argument between him and General Singh, following which the septuagenarian asked Rai to immediately leave the village.
"We have not asked you to fast. If you want to create noise, you can leave Ralegan Siddhi," Hazare told the AAP member.
The anti-graft crusader, whose fast entered the fourth day today, has parked himself near the Yadav Baba temple at Ralegan Siddhi village in Maharashtra's Ahmednagar district.
The 76-year-old Gandhian has said he will not end his fast till the Jan Lokpal Bill is passed in Parliament.
Later, speaking to the media, Rai said some people inimical to the Jan Lokpal movement were misleading Hazare.
"The agenda is Jan Lokpal and people are talking about betrayal. That too, a person who is making rounds of Modi for a ticket to contest elections. Many people are unable to digest our unity and they are poisoning Anna's ears," Rai said.
Asked about growing distance between Hazare and the AAP, Rai said, "There is no distance. As long as Jan Lokpal is our common goal, there cannot be distance. We would probably not have come here if Anna were not fasting for Jan Lokpal.
"Anna said don't stay in Ralegan Siddhi but this is the fight of the entire country. I am leaving Anna's village but will not leave him. I will continue my fast in Delhi," he said.
Replying to a query about whether Hazare was taking wrong decisions under the influence of a coterie, Rai said, "We hope Anna will take the right decision at the right time."
Reflecting a thaw in their relations with Anna Hazare, a team of the AAP had on Thursday called on the Gandhian and pledged to lend their support to his fight for passage of the Jan Lokpal Bill, pending in the Rajya Sabha.
AAP leader Arvind Kejriwal, who could not come due to indisposition, had also spoken to Hazare, who was claimed to have given him his blessings.
Relations between the anti-graft crusader and Kejriwal had come under severe strain after a video showing Hazare expressing disappointment over alleged misuse of funds collected during the India Against Corruption's campaign for Jan Lokpal surfaced just ahead of the Delhi polls.
Kejriwal had said he was "very, very sad" over the video and demanded an open probe into Hazare's allegation. Hazare had later clarified he was mainly concerned about "misuse" of his name in the poll campaign and said the two are "not enemies".
Kejriwal used Hazare as a ladder, which he kicked later
Arvind Kejriwal: The David who slayed Goliath
One day, Kejriwal will become the CM: Anna Hazare
Kejriwal replies to Anna on Aam Aadmi Party funding
Kejriwal used Hazare as a ladder, which he kicked later