"For three years you have seen the progress, it's because we have kept away not only people like Brijesh but also people associated with him. I think that was extremely critical. We need good people to contribute and to do this you don't need to be in power," he said in Bangalore.
Kumble alleged that Patel tried to prevent him from contesting KSCA presidential election on a frivolous basis.
"He was against my contesting elections as he asked me to withdraw my candidature saying the registrar of society had rejected his membership as the association did not intimate him about an amendment of a law pertaining to granting of membership. The amendment was done to help cricketers like me who have played 50 Tests to get membership, without waiting for two years as it was before," he said.
Kumble said that Patel, who was a part of Sports Frontier earlier, was the biggest beneficiary of Karnataka Premier League in 2008 and 2009.
"The biggest beneficiary was Sporting Frontiers which was hired to conduct KPL. In two years, close to Rs 1.6 to 1.7 crore were received by them as fee at the cost of franchisees ... There are records that in 2000-2001 Brijesh was part of that agency," said Kumble who is not contesting next month's KSCA elections.
He also supported former India fast bowler and KSCA secretary Javagal Srinath's challenge to hold a public debate with Patel over his intentions of fighting an election after 12
"I back Srinath's comments of having a public debate with Brijesh as to what his intentions are coming back to power or to fight an election after 12 years when we fully know within two years of becoming the secretary in 1998, the previous secretary had to face hell. His membership was cancelled," Kumble said.
"People who want to be in power and do not contribute -- that is what hurts me as a cricketer because to me I am today because of cricket," he said.
Kumble said Sadananda Maiya and A V Jayaprakash will contest for president and secretary's posts respectively in the upcoming KSCA elections.
"It should not just be Kumble and Srinath. It should be these people (Maiya and Jayaprakash) also contributing and that is the change we want to see. The fresh ideas that Jayaprakash will bring in will certainly help the association to move forward. I want to see KSCA as number one," he said.
Kumble said he and Srinath entered KSCA with an aim to develop cricket in the state.
"The reason we came in was because for the last 12 years there was absolutely no development," he said.
Kumble said he and Srinath would be available for assistance in cricket matters even though they are not contesting the elections.
"My vision is not about cricket in India but sports in India. How can we become a better sporting nation ... To take this forward I don't think I and Srinath need to be in office," he said.
Kumble said to achieve this goal, his team would support and guide good people, but not those who have vested interests.