The tribunal said Reliance's application is not maintainable because the new consent mechanism norms are applicable with retrospective effect.
A full bench of the SAT, headed by presiding officer JP Devdhar, said the consent mechanism norms in the Sebi Act were amended with retrospective effect from April 2010.
Reliance has been contesting a show-cause notice from the Securities & Exchange Board of India in the case since December 2010 and from last year, the regulator's exclusion of the company from the new consent mechanism.
The case dates back to 2008, when Sebi alleged violation of norms against RIL when it merges unit Reliance Petroleum with itself in 2007.
The Mukesh Ambani-led company challenged the Sebi charges at the SAT in December 2010.
It also involves RIL challenging the maintainability of Sebi's decision to exclude the company from the consent mechanism under new norms issued in January 2013.
Under the mechanism, a company can settle an issue with the regulator after paying a penalty, without accepting or rejecting the charges against it.
On Friday, when the SAT concluded the hearings, senior Sebi counsel Darius Khambatta had argued for maintaining the regulator's rejection of the RIL application, citing the January 2013 consent provisions.
He also said the SAT cannot maintain the RIL appeal as the revised consent mechanism norms apply to the RIL case.
He said Sebi rejected the RIL appeal for consent because the case involved serious violations of norms, apart from conducting fraudulent and unfair trading practices.
RIL senior counsel Janakdwaraka Das had argued the settlement proceedings should be conducted as per regulations made under the Sebi Ordinance and that it cannot arbitrarily exclude someone from a redressal mechanism.
Sebi notified the new consent norms on January 9, 2013, after issuing the draft regulations in May 2012.
Under the new norms, Sebi excluded cases involving larger amounts or fines from the settlement process.
It said the provisions would apply with retrospective effect from April 2010.
RIL, prior to the merger of RPL with itself, allegedly short-sold a 4.1 per cent stake in RPL valued at Rs 4,023 crore (Rs 40.23 billion) to prevent a slump in the stock.
The RPL shares were first sold in the futures market and later in the spot market, covering the sales in the futures market, it was alleged.
In 2008, Sebi initiated a probe into the matter and in 2010 began quasi-judicial proceedings and said it had found that RIL had booked a profit of Rs 513 crore (Rs 5.13 billion) in the futures segment through this deal worth Rs 4,023 crore.
Sebi sent a show-cause notice to RIL, saying the company was aware of the sale of shares and sold futures ahead of that.
RIL challenged the show-cause notice in December 2010.
Following this, Sebi ordered a probe and found that RIL had violated norms on prohibition of fraudulent and unfair trade practices relating to the securities market.
Though RIL moved Sebi for a consent settlement, the regulator did not entertain the application and RIL then moved the SAT.
Image: RIL chief Mukesh Ambani
Please click here for the Complete Coverage of Budget 2014 -15
Insider-trading case: SAT reserves order on Reliance plea
Insider-trading case: SAT reserves order on Reliance plea
SAT adjourns RIL insider-trading case
RIL, BP to invest Rs 800 cr in Tamil Nadu
Real estate firms may raise funds via IPO route