Flight MH148 departed Melbourne Airport at 2:16 pm (0416 GMT) to head towards the Malaysian capital Kuala Lumpur but had to perform an emergency landing. All passengers were safe.
"After departing from Melbourne, the aircraft's systems indicated there was an engine fire, so it returned and landed," a spokesman for the Australian Transport Safety Bureau was quoted as saying by the ABC news.
However, an inspection of the aircraft after it landed found no evidence of a fire on board, the spokesman said.
Malaysia Airlines also tweeted that early inspections found no evidence of fire.
"More than 300 passengers were onboard the Malaysia Airlines flight from Melbourne to Kuala Lumpur," a spokesman of Metropolitan Fire Brigade (MFB) was quoted as saying.
Emergency crews were called to the airport around 2:20pm (local time).
The spokesman said the plane dumped fuel before making the emergency landing about 3:00pm.
According to online flight trackers, the flight took several circles above Melbourne before landing.
The cash-strapped state-owned airline lost two airliners in disasters last year.
A Beijing-bound Malaysia Airlines flight MH370 -- carrying 239 people, including five Indians, an Indo-Canadian and 154 Chinese nationals -- had mysteriously vanished on March 8, 2014 after taking off from Kuala Lumpur.
Another Malaysia Airlines plane MH17 was shot down over Ukraine in July last year, killing all 298 on board.