Newport, 13th in League Two, set about Claude Puel's Leicester side with relish and went ahead as Robbie Willmott raced down the right and crossed for centre forward Jamille Matt to glance home a header.
Denis Odoi had given Fulham the lead before Sam Surridge pulled Oldham level but Fulham were left to rue Aleksandar Mitrovic's missed penalty at 1-1, as Callum Lang scored an 89th-minute victor to send Oldham into the fourth round.
"I think we had enough quality on the pitch to qualify", said Puel, who was sharply criticised for naming an under-strength side for the Carabao Cup quarter-final defeat to makeshift Manchester City opponents last month.
Strikes from Sterling, Phil Foden, Gabriel Jesus, Riyad Mahrez, Nicolas Otamendi, Leroy Sane and an own goal from Semi Ajayi got the Citizens into the fourth round of the FA Cup in an emphatic fashion.
"It is Roy of the Rovers stuff", said Oldham boss Pete Wild.
Aleksandar Mitrovic could not repeat Surridge's feat when his spot-kick was parried wide by Daniel Iversen, with the on-loan Leicester goalkeeper's heroics paying further dividends in the 88th minute.
New Captain Marvel teaser is all sass, Fury and explosions
Louis D'Esposito, Victoria Alonso, Jonathan Schwartz, Patricia Whitcher and Stan Lee are the executive producers. Despite both being female-led superhero films, Schwartz says Captain Marvel is its own film.
"It's unbelievable", Barnet captain Callum Reynolds said after the north London club's triumph.
Premier League sides come into play in the third round proper (round of 64), with lower league teams battling it out in the early stages of the competition. We couldn't beat Stevenage last weekend and they had 10 men for an hour.
Manchester City brushed aside Championship side, Rotherham United courtesy of an emphatic 7 nil victory.
Bidwell then headed in the victor with 15 minutes to go from a set-piece.
Elsewhere, in an all-Premier League clash, Brighton won 3-1 away to south coast rivals Bournemouth.
Premier League leaders Liverpool will complete a third round that started with Tottenham´s 7-0 thrashing of Tranmere on Friday when they travel to top-flight rivals Wolves on Monday.