City, meanwhile, found themselves 2-0 down in the first half at the Liberty stadium, but rallied in the second period.
But Bernardo Silva gave City a lifeline with a fierce strike 20 minutes from the end and the Portuguese playmaker insists anything is still possible.
In the end, Guardiola was left to sing the praises of Aguero, who had the final say in an exhilarating contest as he made up for lost time in dramatic fashion with his late victor.
Manchester City's quest for a quadruple is still on, thanks to a bit of luck and some help from the referee, but Manchester United has only one remaining chance for silverware after being dumped out of the FA Cup by Wolverhampton Wanderers.
Yet Championship side Swansea were left aggrieved at the awarding of two late goals to City which nearly certainly would not have stood if VAR had been in operation.
One was a penalty call against Swansea's Cameron Carter-Vickers who produced what appeared a fine challenge on Raheem Sterling but as the City player went down in the box, a penalty was awarded by Andre Mariner.
Swansea then carved City open with a slick passing move and it was superbly finished off by Celina. "That's why I say "chapeau" for all my players and the staff".
"Yeah, I'm exhausted", the City manager said on Saturday, bowing his head as he spoke.
Aguero restored parity in fortuitous circumstances when his spot-kick hit a post and went in via Nordfeldt's back.
Yet there was no VAR to turn to for a review, with the English footballing authorities having ruled that only Cup matches at Premier League grounds would have the video technology available.
AGUERO would make it 3-2 and score the victor off a headed goal but not without controversy.
Guardiola, though, was full of praise for the character of his side, who came from two goals down to win a game for the first time in all competitions since he took charge of the club.
Jota then finished off a lightning counter attack to double the lead, which never looked under threat until Marcus Rashford pulled back a consolation goal deep into injury time.
Championship side Millwall host Premier League side Brighton on Sunday.