"After 20 months of instability and inconsistent results, the federation have agreed with Zaki to end his contract by mutual consent," the federation said in a statement, adding that a new coach would be appointed in the coming days.
Zaki first managed the side for three years from 2002, losing the Africa Cup of Nations final to Tunisia in 2004.
Herve Renard, who was sacked by French club Lille earlier this season, is rumoured to be lined up to take over.
French sports daily L'Equipe reported that sources close to the FRMF said last month that Renard was "a priority to take over the Atlas Lions".
Frenchman Renard, 47, has won the Africa Cup of Nations twice as a manager, first with Zambia in 2012, and then with the Ivory Coast last year.