VFS Centre, housed on Bangalore's busy Richmond Road, had taken the building on lease for the last three years. Apart from the UK's main visa processing centre on Cunningham Road, this is the only other entity to process visas for Britain in the city.
With the news of the brothers' alleged involvement in the terror plot, VFS has sought termination of the lease.
The multi-storeyed building, opposite the city's landmark Baldwin girls school, was constructed by the Ahmeds around 10 years ago. The building is jointly owned by Kafeel and Sabeel, and was leased out to the VSF Centre around three years ago.
Family sources confirmed they have now received a legal notice from the entity seeking to terminate the annual lease. Both the company and the family were tightlipped about the lease amount.
Sources in the entity said they decided to move out of the building immediately and have informed the UK authorities of their decision.
"We felt it would not be right to sit in a building owned by them and process visas for a country which they are alleged to have targeted," the sources said.
Sajan Poovaya, counsel for the company, refused to comment on the issue and said it was a confidential matter.
Kafeel and Sabeel's parents too refused to speak on the issue.
Dr Kafeel Ahmed, an engineer, is alleged to have driven a flaming jeep into Glasgow airport on June 30 and then set himself afire; his brother Sabeel, a doctor, was detained in Liverpool for interrogation in the same case. Their parents are doctors and residents of Banashankari II stage in Bangalore.
Cops question Kafeel's associates
Bangalore's IT firms told to be on alert
Karnataka to set up Mumbai-style ATS
Kafeel unlikely to survive: doctors
Experts scrutinise Kafeel's hard disk for clues