Number concepts are often thought to originate from counting and the successor function, or from a refinement of the approximate number sense. Here we argue for a third origin: a shared language-of-thought for geometry and arithmetic, with primitives of repetition, concatenation, and recursive embedding. Applied to sets, starting from 1, those primitives engender concepts of exact integers through recursive applications of additions and multiplications. Links between geometry and arithmetic also explain the emergence of higher-level notions (squares, primes, etc.). Under our hypothesis, understanding a number means possessing one or several mental expressions for it, and their minimal description length determines how easily they can be mentally manipulated. Several historical, developmental, linguistic, and brain-imaging phenomena provide preliminary support for our proposal.