Intermittent explosions were heard and flames threatened to engulf nearby buildings as rescuers fought thick plumes of smoke to reach those trapped.
Shocked people huddled in groups and some carried those with severe burns to ambulances, which were rushed to the spot soon after the incident at Omshakti Fireworks.
A thick pall of smoke engulfed the area and was visible from a distance of over 1.5 km.
A fire brigade official said 20 of the 40 rooms in the factory spread across 10 acres had been completely gutted and the debris was spread over a two kilometre radius.
He said they were unable to collect any data about the unit or workers as the people running it had fled the spot.
Though it was not clear how the fire had broken out, it quickly spread to other rooms, he said, adding that rescue operations were hampered as there was no proper road connectivity.
In an ironic twist to the tragedy, some people who tried saving those in the factory were also trapped in the thick smoke.
Tamizhmuni, a worker, said he came to unload stocks but fainted due to the heavy smoke.
Sivakasi is the largest hub for manufacturing matches and firecrackers, accounting for 90 per cent of the country's total fireworks production.
Sivakasi is home to roughly around 450 factories employing 40,000 people and over one lakh indirectly, at an estimated annual turnover of between Rs 800 and Rs 1000 crore.