That "small but energetic part of his base" includes the KKK, who held a victory parade in North Carolina after Trump was elected, and former KKK grand wizard David Duke who said the Unite the Right rally in Charlottesville "fulfills the promises of Donald Trump".
The president doubled down on that dismissiveness at Tuesday's press conference, distancing Bannon from his unexpectedly successful presidential campaign.
House Minority leader Nancy Pelosi, D-Calif., said in a statement that Trump's initial response was a "direct reflection of the fact that his chief strategist, Steve Bannon, is an alt-right white supremacist sympathizer and a shameless enforcer of those un-American beliefs".
One source with direct knowledge of White House staffing told Fox News they expect Bannon will be the next one ousted, pointing to the icy nature of his relationship with Trump.
"Look, I like Mr. Bannon, he's a friend of mine, but Mr. Bannon came on very late - you know that".
Kelly's hire has prompted a renewed effort from Bannon's rivals inside and outside the White House to try to secure the former Breitbart chief's ouster.
"McMaster isn't saying Bannon is the mastermind behind the campaign, but he does think Bannon could stop it if he wanted to", said one of McMaster's defenders.
On Sunday, McMaster declined to put to rest the speculation about a rift between the two.
Bannon didn't respond to requests for comment.
Scaramucci, who infamously was the communications director at the White House for just 10 days, felt that Bannon was responsible for some of the leaks that have dogged the Trump administration since it formed in January. The chief of staff is embarking on a weeks-long personnel review of West Wing staff and has indicated to aides that significant changes could be coming, according to an official familiar with Kelly's plans but not authorized to speak publicly about private conversations. But, we'll see what happens with Mr. Bannon.
He has pushed to scrap the 2015 nuclear-weapons agreement with Iran, which McMaster argues should remain in place, and has also proposed using contractors to fight the war in Afghanistan rather than expanding US forces there, as McMaster has advocated.
It was only last week that Gorka, in an appearance on a Breitbart News Daily radio show, defended the white nationalist movement.
Conservative commentator Mike Cernovich is a Bannon ally and has been a vocal critic of McMaster, even levelling personal attacks against him.
"That's not a fight he's going to win", the Republican said of Mr. Bannon.
The silence isn't surprising, since Bannon rarely talks (on the record) to the mainstream media, which he has described as an "opposition party" relative to the Trump administration.