The away side took the lead in the first minute. A long ball was played forward and not properly cleared and Jack Harrison chipped an inviting cross into the box and Vitaliy Mykolenko headed in from close range.
Palace were level four minutes later when Eberechi Eze went on a surging run into the Everton penalty area before being brought down by Jarrad Branthwaite. The England midfielder picked himself up and slotted the penalty confidently past Jordan Pickford.
Everton retook the lead in the 49th minute. Mykolenko hit a volley from the edge of the area that struck Sam Johnstone’s post and Abdoulaye Doucoure was on hand to tap into an empty net from close range.
Palace thought they had done enough to draw the match when Odsonne Edouard took advantage of a mix-up between James Tarkowski and Pickford, but that was before substitute Gueye scored late on to take all three points.
The result means Everton are unbeaten in four games in all competitions and eight points clear of the relegation zone.
The match between these two sides was played at a breathtaking pace and was not short of incidents.
Everton caused Palace a lot of problems with their crosses into the box and Dominic Calvert-Lewin was denied a goal in the first half by an excellent intervention from Joel Ward when the scores were level.
That was before Palace thought they should have had a second penalty when Branthwaite challenged Eze, but, after a lengthy video assistant referee check, Eze was booked for simulation.
Everton started the second half like they did the first period and scored early again through Doucoure before Edouard’s leveller.
Gueye then snatched all three points with the winning goal in the 86th minute. The midfielder was played through on goal by Doucoure and slotted the ball through Tyrick Mitchell’s legs and into the corner of the net to win the game.
The victory is only Everton’s second in eight visits to Selhurst Park and continues their run of good form.
In his news conference before this match manager Sean Dyche said Everton’s performances were starting to get the points they deserved, and he was proved correct again after this match.
The Toffees came flying out of the traps at the start of the game and took advantage of some sloppy defending to take an early lead.
Dyche’s side had a clear game plan and their crosses into the Palace area caused numerous problems with Calvert-Lewin and Dwight McNeil going close from them.
Dyche made the decision to bring off Amadou Onana at half-time for Gueye and that substitution was rewarded with the winning goal.
Pickford had to make a good save from Eze, but Everton did enough to get over the line and take three points.
The news that Eze had signed a new three-and-a-half-year contract was celebrated as a major piece of business for Palace, and, after watching his performance against Everton, you could see why.
Eze scored 10 goals and had four assists last season, but in this campaign he has stepped up to fill the void left by talisman Wilfried Zaha, who joined Galatasaray in the summer.
The majority of Palace’s attacking play flowed through Eze and he scored his penalty on the back of one of his trademark dribbles.
On another day he could have been awarded another spot-kick and not given a yellow card for simulation, with the decision deemed not to be a clear and obvious error.
The 25-year-old’s performance was a positive for Palace in defeat but the result leaves them with one win from their past five games.