Hepatitis C virus (HCV) infection triggers Golgi fragmentation through the Golgi-resident protein immunity-related GTPase M (IRGM). Here, we report the roles of NLRP3 (NOD-, LRR- and pyrin domain-containing protein 3) and ASC (apoptosis-associated speck-like protein containing a caspase activation and recruitment domain [CARD]), two inflammasome components, in the initial events leading to this fragmentation. We show that ASC resides at the Golgi with IRGM at homeostasis. Upon infection, ASC dissociates from both IRGM and the Golgi and associates with HCV-induced NLRP3. NLRP3 silencing inhibits Golgi fragmentation. ASC silencing disrupts the Golgi structure in both control and infected cells and reduces the localization of IRGM at the Golgi. IRGM depletion in the ASC-silenced cells cannot totally restore the Golgi structure. These data highlight a role for ASC, upstream of the formation of the inflammasome, in regulating IRGM through its control on the Golgi. A similar mechanism occurs in response to nigericin treatment, but not in cells infected with another member of the Flaviviridae family, Zika virus (ZIKV). We propose a model for a newly ascribed function of the inflammasome components in Golgi structural remodeling during certain stimuli. IMPORTANCE Numerous pathogens can affect cellular homeostasis and organelle dynamics. Hepatitis C virus (HCV) triggers Golgi fragmentation through the immunity-related GTPase M (IRGM), a resident Golgi protein, to enhance its lipid supply for replication. Here, we reveal the role of the inflammasome components NLRP3 and ASC in this process, thus uncovering a new interplay between effectors of inflammation and viral infection or stress. We show that the inflammasome component ASC resides at the Golgi under homeostasis and associates with IRGM. Upon HCV infection, ASC is recruited to NLRP3 and dissociates from IRGM, causing Golgi fragmentation. Our results uncover that aside from their known function in the inflammation response, these host defense regulators also ensure the maintenance of intact intracellular structure in homeostasis, while their activation relieves factors leading to Golgi remodeling.