The categorization of geometric objects is one of the most fundamental problems all intelligent systems have to deal with in dynamic environments in which objects' geometrical configuration constantly changes. Animals, including humans, do not treat all geometrical differences equally: they ignore some geometrical features when it comes to generalization but not others. So far, no theory has been presented that explains this cognitive phenomenon. We here propose and empirically test such a theory. The theory identifies and relies on the invariant referents existing in 3D (i.e., gravity) and 2D (e.g., any 2D frame) environments to predict the geometrical differences reasoners consider as important or irrelevant for object categorization. We test and confirm a novel central prediction of the theory, namely that human reasoners categorize objects differently in 3D and 2D environments. These findings cast new light on core cognitive abilities that minds use to make sense of the world.