you also cannot collect fruits while in the orb, but if you unequip the orb you still stay in the area while never entering the minigame, meaning that you can use the jetpack to fly around and collect fruits, this has no effect however.
I think this also might relate to the part of my post: [0.7.0.3] General weird orb behaviour where the location doesn’t change from where you originally equipped the orb
Don’t worry, a long time ago the devs made it so that it isn’t possible to steal fruit from other players.
So you can only trick yourself out of earning points.