The African side trailed to a stunning Keisuke Honda strike in the first half, but the introduction of striker Didier Drogba off the bench proved the catalyst to turn the match on its head after the restart as they scored twice in three minutes.
The victory at the Pernambuco arena moved the Ivorians level on three points at the top of Group C with Colombia, who beat Greece 3-0 earlier on Saturday.
- PHOTOS: Balotelli takes Italy past England
- PHOTOS: Costa Rica shock Uruguay, Colombia ease past Greece
With coaches so often judged on their ability to make brave decisions, Ivory Coast's Sabri Lamouchi put his neck on the line by leaving out Drogba, for so long a talismanic presence in the lineup, in favour of the in-form Bony.
It looked a poor call as Japan had the better of the first half.
They were sharp and nimble in possession, working clever angles as they picked off their opponents' frequently misplaced passes and took the lead with a goal of genuine quality from Honda after 16 minutes.
Collecting a pass across the box, he took one touch to shift it out of his feet and then drove it powerfully into the top corner with his left foot, before galloping away in celebration as blue hordes of Japanese fans celebrated deliriously.
The Elephants had their moments, with midfielder Yaya Toure showing occasional glimpses of his destructive powers, shifting through the gears with trademark bursts of raw energy.
Yet there was little killer instinct as half chances came and went, with Drogba prowling on the sidelines before entering the fray after 62 minutes.
The change had the desired impact on the Ivory Coast who levelled two minutes later when Bony leapt to glance a fizzing Serge Aurier cross into the far corner.
Two minutes later and another drilled Aurier centre was met this time by Gervinho who got in front of his marker to direct a header on goal that squirmed past Japan goalkeeper Eiji Kawashima, who should have done better.
Image: Ivory Coast's Gervinho (left) and Didier Drogba celebrate their goal against Japan
Photograph: Ruben Sprich/Reuters
World Cup chit-chat: Messi goalless; Van Persie injured
World Cup: Falcao-less Colombia look set for last 16
Points Table: Football World Cup 2014
10 players to watch out for at the World Cup in Brazil
Colombia thrash Greece in joyous World Cup return