A former interior minister was sworn in as Malaysia's premier on Sunday, marking the return of a scandal-mired party to power after a reformist government's collapse but ex-leader Mahathir Mohamad, 94, slammed the move as illegal.

The Southeast Asian nation was plunged into turmoil after Mahathir's reformist "Pact of Hope" alliance, which stormed to a historic victory in 2018, collapsed amid bitter infighting.

Mahathir — who was the world's oldest leader — initially quit as premier but then sought to return. He lost out however to little-known Muhyiddin Yassin, who heads a coalition dominated by the country's Muslim majority.

The king's decision on Saturday to pick Muhyiddin as premier was greeted with shock as Mahathir's allies claimed to have enough support, and sparked widespread anger that the democratically elected government had been ejected.

Just before Muhyiddin's inauguration, Mahathir accused him of betrayal and said he would seek a parliament vote challenging the new premier's support — signalling the political crisis is far from over.

"This is a very strange thing [...] losers will form the government, the winners will be in the opposition," he said.

"The rule of law no longer applies."

Muhyiddin's coalition includes the United Malays National Organisation (UMNO), the party of disgraced ex-leader Najib Razak, as well as a hardline group that wants tougher Islamic laws.

UMNO was the corruption-riddled lynchpin of a coalition thrown out at the 2018 elections amid allegations Najib and his cronies looted billions of dollars from state fund 1MDB. Najib is now on trial for corruption.

Despite a last-minute bid by Mahathir to prove that he had enough support to return as premier, Muhyiddin's inauguration went ahead Sunday morning at the national palace.

His alliance boycotted the ceremony.

'Betrayal'

To become premier, a candidate must prove to the king, who appoints the prime minister, that he has the backing of at least 112 MPs.

Meanwhile, public anger was growing at the ejection of the reformist alliance, with the hashtag "NotMyPM" trending on Twitter and more than 100,000 people signing a petition that said the move was a "betrayal" of voters' choice at the 2018 poll.

Mahathir — who served a first stint as premier from 1981 to 2003 before staging a comeback two years ago— also raised concerns that ongoing court cases against Najib related to the 1MDB scandal could be affected by the change of government.

Mahathir came out of retirement aged 92 to take on his one-time protege Najib over the controversy, and has pledged to bring him to justice.

Analysts said Muhyiddin's government was in a weak position as it was not clear whether he had a parliamentary majority, and was a member of a tiny party without a strong support base.

He was a senior UMNO figure for decades before being sacked from the party by Najib in 2015 after criticising the handling of the 1MDB scandal, and then allied with Mahathir to oust them at the last elections.

In a volte-face, he joined hands last week with UMNO in his quest for the premiership — although he is not a member.

In addition, he lacks "the domestic legitimacy or the international legitimacy the previous government had", Bridget Welsh, an analyst from the University of Nottingham, told AFP.

He is a Muslim nationalist — he once controversially described himself as "Malay first" and Malaysian second — and there are concerns his leadership could worsen already tense race and religious relations in the multi-ethnic country.

About 60 per cent of Malaysia's population are ethnic Malay Muslims and the country is also home to substantial ethnic Chinese and ethnic Indian minorities.

The political crisis began a week ago when a group of ruling coalition lawmakers joined forces with opposition parties in a bid to form a new government and push out Mahathir's designated successor Anwar Ibrahim.

In a week of fast-moving twists and turns, Mahathir and Anwar — who have a long and stormy history — briefly locked horns in a fight for the leadership.

They then joined forces again in an 11th-hour bid to stop Muhyiddin, but it proved too little, too late.