The 28-year-old Finn, who started from pole position, led from the lights to the flag apart from a spell following his first pit stop when the four-time champion Briton was in control.
It was Bottas third career victory, all of them recorded since he joined Mercedes to partner the 32-year-old Hamilton in January following the unexpected exit of 2016 champion German Nico Rosberg.
“This is a really important win for me after having a pretty difficult start to the second half of the year,” said Bottas.
“We Finns don t show much emotion but it doesn’t mean we don t have any. I am so happy. There has been so much support. I cant show it too much but I feel good.”
Hamilton came home four seconds behind Bottas, having eased off in the closing laps.
The Briton scored points in every race of the season, underlining his dominance this year.
Four-time former champion Sebastian Vettel finished a lonely third for Ferrari, as the two Mercedes men delivered celebratory doughnuts for the crowd.
It was Vettel s 99th career podium and his 13th of the season.
Kimi Raikkonen was fourth in the second Ferrari ahead of Max Verstappen of Red Bull and Nico Hulkenberg for Renault, whose sixth-place finish assured Renault of sixth in the constructors championship ahead of Toro Rosso.
Sergio Perez came home seventh ahead of his Force India team-mate Esteban Ocon, Fernando Alonso of McLaren Honda and Felipe Massa, in his last F1 race before retirement, for Williams.
Australian Daniel Ricciardo, who was fourth on the grid, failed to finish his Red Bull for the third time in four outings.
Hamilton was vocal in his dislike of the narrow track.
“It’s impossible to pass here man!” he said. “I was like — where are all the back-markers… They ve got to change this track.”
He added: “Big congratulations to Valtteri, he did an amazing job to hold me off.”
As the sun went down at the warm Yas Marina Circuit, Bottas avoided repeating his poor start in Brazil two weeks ago and pulled clear from pole into the first turn ahead of Hamilton.
Behind them, Vettel stayed third ahead of Ricciardo and Raikkonen while, at the back of the field, Dane Kevin Magnussen spun and recovered in his Haas.
The two Mercedes men were producing a duel at the front and were 2.5 seconds ahead of Vettel.
The two Silver Arrows continued to pull clear before the pit stops began, Bottas arriving for new tyres after 21 laps. Hamilton inherited the lead and pushed before pitting himself after 24 laps, re-joining second again.
By then Ricciardo s race was over after he had retired with a hydraulics problem, his third early shower in four races.
At the front, Bottas appeared to be struggling on his new super-soft tyres with Hamilton applying pressure and reducing the gap to 0.9 seconds.
His superior speed, however, did not allow him to find a way to pass his team-mate in the turbulent air behind him.
The pressure told on Hamilton when he ran over a kerb at turn 17 and then, at the same place, went off the track completely. Mercedes boss Toto Wolff could afford to smile, knowing that both championships were won already.
This pushed Hamilton back to 1.8 seconds behind while behind him, Carlos Sainz had to retire when his front left wheel was not properly attached to his Renault in a pit stop.
Hamilton continued to deliver fast laps and cut the gap to 1.1 seconds, but struggled to gain in the third sector through the marina and hotel complex and was clearly frustrated by the lack of overtaking.
The Briton continued to push and with six laps remaining cut the gap to 0.5 seconds, but Bottas stayed cool as the champion loomed in his mirrors and hoped for a mistake by his team-mate.
By this stage, Vettel was more than 20 seconds adrift, but 19 seconds clear of fourth-placed Ferrari team-mate Raikkonen.
Hamilton scored 363 to finish top in the final drivers standings ahead of Vettel on 317, Bottas on 305 and Raikkonen on 205.
In the constructors , Mercedes led the way with 668 ahead of Ferrari on 522, Red Bull on 368 and Force India on 177.
The opening race of 2018 in Melbourne is a 17-week European winter away on March 28, and will herald a new era with halo cockpit protection devices for the first time. Bottas, as he said, plans to be ready.
