He made the attack on Jung while joining a protest by BJP MP Maheish Girri, who demanded an apology from Chief Minister Arvind Kejriwal for linking him to the murder of an NDMC official.
Separately, Swamy, in a letter to President Pranab Mukherjee, said he has information to suggest that the Jung was playing a "mischievous role" in "abetting" Kejriwal at the behest of Congress to embarrass the BJP.
"He (Jung) has no work other than making rounds of Ahmed Patel's place. He (Jung) should have spoken out for MPs like Girri and he should have ordered Kejriwal to provide documents or apologise," Swamy told reporters.
He also said Jung should have categorically told Kejriwal that if he does not substantiate his allegations against Girri or apologise, then he would recommend to the Centre dismissal of the Aam Aadmi Party government.
The BJP MP said he would go after Kejriwal like the way he took on RBI Governor Raghuram Rajan. "I was after Rajan in the recent past. He has left now. Girri is a religious person who has dedicated his life to public service...You are the CM, give written complaint. And what this LG is doing."
In his letter to President Pranab Mukherjee, Swamy said, "I request you to intervene, particularly since I have information to suggest that the LG is playing a mischievous role in abetting Kejriwal at the behest of the Congress party to embarrass the BJP and, at the same time, appearing to oppose Kejriwal in public for sake of credibility."
Swamy said Delhi government should be dismissed if Kejriwal refuses to apologise to the party MP over his allegations.
He also accused Kejriwal of breaching constitutional norms in running the Delhi government and sought the President's intervention, saying there appears to be a complete breakdown of governance.
Swamy requested the President to ask the Union home ministry to issue a directive as per provision of the Constitution and seek a report asking on what basis Kejriwal and his colleagues have made allegations of murder against Girri.
"There appears to be a complete breakdown of any semblance of governance in National Capital Territory of Delhi. The AAP is functioning in a highly arbitrary, unreasonable and malafide manner. Since the NCT is where the Union Government is also located, it has become crucial to restore some semblance of governance immediately," said Swamy in the letter.
Girri has been holding a protest outside Kejriwal's residence in Civil Lines demanding proof of his allegations. Swamy had joined Girri briefly in the protest.