The opposition parties voted 156-145 in favour of a Liberal motion, citing Harper's minority Tories for contempt of Parliament and expressing non-confidence in the government.
The contempt charge marks a first for a national government anywhere in the Commonwealth.
Harper emerged from the House of Commons after the vote and announced he will visit the Governor General's residence on Saturday to dissolve the 40th Parliament and sound the starting gun on an election campaign.
It will be the fourth election in seven years. The election is expected to take place May 2.
Image: Canada's PM Harper pauses while speaking following a non-confidence vote in the House of Commons on Parliament Hill in Ottawa
Photoghraph: Blair Gable/Reuters
Close to signing nuke deal with India: Canadian PM
Tata Steel may buy coal mines in western Canada
Images: Lee, Watson take the fight out of Canada
Singapore beefs up security for APEC meet
India holds mini-Pravasi Bharatiya Divas in Canada