The site of contact between T lymphocytes and antigen-presenting cells becomes, upon antigen recognition, an organized junction named the immunological synapse. Various T cell organelles polarize, together with microtubules, toward the antigen-presenting cell. Among them, intracellular vesicular compartments, such as the Golgi apparatus, the recycling endosomal compartment, or cytotoxic granules help to build the immunological synapse and ensure effector functions, such as polarized secretion of cytokines by helper T cells, or exocytosis of lytic granules by cytotoxic T cells. Lymphotropic retroviruses, such as the human immunodeficiency virus type 1, the human T cell leukemia virus type 1, or the Herpesvirus saimiri, can subvert some of the vesicle traffic mechanisms impeding the generation and function of the immunological synapses. This review focuses on the polarization of vesicle traffic, its regulation, and its role in maintaining the structure and function of the immunological synapse. We discuss how some lymphotropic viruses target the vesicle traffic in T lymphocytes, inhibiting the formation of immunological synapses and modulating the response of infected T cells.