This problem is mainly due to the current offsets. The way the wearables are positioned on players.
But yeah most of the offsets are pretty terrible imo, the glasses still clip super badly with the minigolf ball. I've been wanting this to be fixed personally for so long. However it's not a priority, the devs want to focus on character customization last so it doesn't get in the way of new gameworlds and new content in the plaza/condos.