The Foxes have risen to seventh in the Premier League table following a 1-0 win over Everton on New Year's Day, while their opponents now sit 12th in League Two.
There was no so-called "giant-killing" at Manchester City's Etihad Stadium, though, with the Premier League champions sweeping aside second-tier Rotherham in a 7-0 win in Pep Guardiola's biggest victory in charge of the club.
However, moments later a penalty was awarded to Newport when the ball struck Marc Albrighton's arm, which was converted by Padraig Amond in the 85th minute.
But they could not add to their lead and Oldham were level with 15 minutes to go after Ryan Sessegnon brought down Peter Clarke and Sam Surridge fired in from the spot, sending Marcus Bettinelli the wrong way.
Bidwell then headed in the victor with 15 minutes to go from a set-piece.
Claudio Ranieri's Fulham reign has got off to a hard start and it got worse for the former Leicester boss at the hands of Pete Wild's Oldham. I'm so proud of the players'.
"I think we need to be ready because I know the tradition of the cup", he told a press conference.
United were the victims of a massive shock - Shaquile Coulthirst's 21st-minute penalty securing a famous win for Barnet, the lowest-ranked side now left in the competition.
Newport are yet to concede a goal in this season's FA Cup.
Marcelo Bielsa's much-changed Leeds side once again fell in the third round after a 2-1 defeat away at Queens Park Rangers.
Raheem Sterling scored early on and youngster Phil Foden's first home goal for the club and Semi Ajayi's own goal made the result a formality before halftime.