"The terrorist camps operating from Pakistan was the main impediment to improve India-Pakistan relations," he said.
Noting that Pakistan was still pushing trained terrorists into Jammu and Kashmir, he described it as a matter of "serious concern" and that necessary steps were being taken to prevent it.
As a policy, he said, India was not for any confrontation with its neighbours. "We want to avoid strained relationships with our neighbours and to improve ties."
To a query on avoiding middlemen in defence deals, Antony said the government had been successful in its attempt so far "to a great extent".
"Whenever there was a complaint, the government had acted swiftly and ordered a probe," he said.
On the recent hijack of two ships with Indians aboard by Somalian pirates, Antony said it was a matter of serious concern.
"Even after the world community and marine forces of nations, including US, Russia, China and UK... These things are happening. Indian Navy is also actively involved in the anti-piracy operations on the Somalia coast," he said.
He added that since the pirates keep changing their strategies, new strategies would have to be devised to tackle them.
Pakistan continues to push in trained terrorists from a large number of terror camps across the border into India, Antony said.
This was a matter of serious concern and steps are being taken to check this, he told reporters here.
"Even now, not only the terrorist camps are operating there, (but) there is continuous efforts to push a large number of trained terrorists into Jammu and Kashmir and India.
"It is a serious matter, it is a matter of concern to us. So to prevent that, whatever steps (are) needed we are taking," Antony said.
He said it would be difficult to improve ties with Pakistan unless it made "serious" and "sincere" efforts to dismantle the large number of terror camps on its soil.
The terrorist camps operating from Pakistan was the main impediment to improve India-Pakistan relations, Antony said.
As a policy, he said India was not for any confrontation with its neighbours. "We want to avoid strained relationships with our neighbours and to improve ties."
Image: Defence Minister A K Antony
'Pak terror groups played serious role in attacks in India'
Pak should stop using terror as state policy: PM
Pak not doing enough on fighting terror: US lawmakers
Composite talks can't resume till Pak acts against terror, says PM
Pak terror camps a threat to India: Antony