A signature feature of the animal kingdom is the presence of epithelia: sheets of polarized cells that both insulate the organism from its environment and mediate interactions with it. Epithelial cells display a marked apico-basal polarity, which is highly conserved across the animal kingdom, both in terms of morphology and of molecular regulators. How did this architecture first evolve? Although the last eukaryotic common ancestor almost certainly possessed a simple form of apico-basal polarity (marked by the presence of one or several flagella at a single cellular pole), comparative genomics and evolutionary cell biology reveal that the polarity regulators of animal epithelial cells have a surprisingly complex and stepwise evolutionary history. Here, we retrace their evolutionary assembly. We suggest that the "polarity network" that polarized animal epithelial cells evolved by integration of initially independent cellular modules that evolved at distinct steps of our evolutionary ancestry. The first module dates back to the last common ancestor of animals and amoebozoans and involved Par1, extracellular matrix proteins, and the integrin-mediated adhesion complex. Other regulators, such as Cdc42, Dlg, Par6 and cadherins evolved in ancient unicellular opisthokonts, and might have first been involved in F-actin remodeling and filopodial dynamics. Finally, the bulk of "polarity proteins" as well as specialized adhesion complexes evolved in the metazoan stem-line, in concert with the newly evolved intercellular junctional belts. Thus, the polarized architecture of epithelia can be understood as a palimpsest of components of distinct histories and ancestral functions, which have become tightly integrated in animal tissues.