The man was photographed walking past a group of scouts outside the houses of parliament as he wore a large ISIS flag on his back. He was carrying a small child on his shoulders, who was waving a smaller flag.
Police were criticised for failing to arrest the man who was seen wandering near the parliament and the Big Ben draped in the Islamic State flag on Saturday.
However, metropolitan police chief Bernard Hogan-Howe said that the force will investigate further.
"It may not have been the flag. But I would like to get to the bottom of it and we will investigate it," he told BBC.
"The law says clearly that if you're a member of a proscribed organisation which ISIS is, or appear to be, then of course the police can take action. Just having a flag itself, you've got to make a few more inquiries," he said.
The incident sparked outrage on social media with people questioning why there were no legal consequences.
"I don't want to overreact because it may not have been the flag. But I would like to get to the bottom of it," Hogan-Howe said.
Scotland Yard had earlier said that its officers spoke to the man and considered his actions to be within the Public Order Act 1986.
A metropolitan police spokesperson said, "This man was spoken to by officers with consideration given to relevant legislation and a decision was taken by officers at the time that the man was acting within the law. He was not arrested."
The image caught by a tourist on his camera shows the man and the little girl with the black and white banner associated with ISIS terrorists walking past the Big Ben and Palace of Westminster in central London over the weekend.
Images of the man were reportedly posted on a Korean forum after a tourist spotted him in central London.
The incident comes just days before the capital marks the anniversary of those killed in the 7/7 terror attacks in 2005, and a week after 30 British tourists were killed in the Tunisia beach massacre.
Image: The incident has fueled the people's anger on the social media. Photograph: Mete Sohtaoğlu/ Twitter
The Weekend Agriculturist
Black money: Disclosures to enjoy immunity under FEMA
Masood, Younis put Pakistan in sight of rare win in Sri Lanka
Birthday Special: The mad, MAD face of Ranveer Singh!