For one, the fact that Brett and Neil and Amy told one thing to Lisa Murkowski and Susan Collins and they did another? Or how about the wife of one of the SCOTUS members was involved in the planning for halting the inauguration after the 2020 Election, yet we're all supposed to believe that member never interacted on those matters and he doesn't need to recuse himself about questions related to that event? Or maybe it's the fact that they kinda pick and choose on cases related to gerrymandering / VRA as to whether it's their role to intervene?
Maybe some of those reasons are less justified than others, but it doesn't really change the fact that these are actions that weigh on the court's credibility for a lot of folks. It just is what it is.