Recently, the immune system has been implicated in the regulation of adipose tissue homeostasis and function. Here, we highlight new insights into the distinct phenotypes and functions of adipose resident leukocytes, including a growing body of work on their role in adipose tissue remodeling and thermogenesis. We discuss how the protective capacity of the adipose immune system can be diminished during obesity, where immune cells promote sterile inflammation leading to insulin resistance. Understanding the role of adipose immune cells across different physiological states and processes is important in understanding the full breadth of adipose immunity and the possibilities to harness immune cells in fat tissue for the treatment of chronic metabolic conditions, including obesity.