I wouldn’t. After all, if the tree is making them, not let them grow? I think for citrus it’s more related to young branches being brittle and they might break. Perhaps it’s only a question of how many fruits instead of all/none
I would say the same thing – it can get all the nutrients it needs from the ground. Removing some fruits to favor the growth of a few with already significant size then pruning should do the job in this case.