Najib's comments on the issue of who succeed him added to uncertainty after a spokesman for the king's palace said Mahathir would not be sworn in as prime minister on Thursday.
Najib, 64, is under pressure to improve upon his performance in the 2013 election, when his Barisan Nasional coalition squeaked out a win while losing the popular vote for the first time.
"We are not seeking revenge", Mahathir said.
The election commission said Mahathir's opposition alliance had won 115 seats, over the threshold of 112 seats needed to form a government. Citizens were also unhappy with rising costs.
Mahathir has succeeded in luring away Malays from Najib, chipping away at a support base that his government had nurtured with an official affirmative action programme and racially-charged rhetoric.
He also spent years squabbling with his old rival and another towering figure in Asian politics, the late Singapore leader Lee Kuan Yew.
Najib's BN coalition won 79 seats, a collapse from the 133 it won in the 2013 election, which was itself the coalition's worst poll performance ever at the time.
The U.S. Justice Department says $4.5 billion was looted from state investment fund 1MBD by associates of Najib between 2009 and 2014, including $700 million that landed in Najib's bank account and a $23 million pink diamond necklace bought for Najib's wife. "It's going to be a headache", Mr. Mohamad told reporters.
The conservative Malaysian Islamic Party (PAS) also gained ground with 18 federal seats up from its existing 14.
Mahathir joined Pakatan Harapan after quitting the ruling bloc he once helmed in 2016 citing a loss of confidence in his protégé over the 1MDB scandal.
The scandal is being investigated by at least six countries, although Malaysia's attorney general cleared Mr. Razak of any wrongdoing.
Pakatan Harapan chairman Tun Dr Mahathir Mohamad will seek an audience with the Agong today.
In an even more unlikely change of heart, Mahathir buried a feud with Anwar, 70, previous year and the two agreed to join forces to topple Najib.
"Both the system and the society will be shaken up", head of political studies at the Penang Institute Wong Chin Huat said when asked what can be expected from a PH victory.
The government recently passed a law redrawing election boundaries, leading to accusations that it had gerrymandered constituencies to ensure they were filled by Malay Muslims, who are traditionally BN supporters.
Mahathir was a medical doctor before becoming Malaysia's fourth prime minister in 1981 and kicking off a mission of modernization.
Then the states started to fall. However, he was stopped in his tracks by charges of sodomy and graft, which he denied, but for which he was jailed.
Defeat could just be the beginning of Najib's troubles. Najib's party had long staked its legitimacy on providing preferential treatment for the Bumiputera, or "sons of the soil", which include ethnic Malays and indigenous groups.
That saw the opposition make strong gains in long-time BN strongholds in the polls, including Sarawak state on Borneo island, and the state of Johor, the birthplace of UMNO.
- Jubeir: Saudi Arabia will seek nuclear weapon if Iran does
- Can Manchester United win the Premier League this season?
- Polls open in North Carolina primary voting
- Justice Chelameswar refuses to attend farewell
- Wet wipes will face ban under new United Kingdom plastic rules
- White House Correspondents' Association rebukes Trump over tweet on press credentials
- Passengers evacuated from plane after smoke in cabin
- Annual rankings report: Murray High 8th in state
- ZTE has ceased 'major operating activities' due to United States supply ban
- Gareth Bale scores brilliant goal as Real Madrid draw with Barcelona