Physically inactive populations are more exposed to the occurrence of chronic non-communicable diseases and to a higher risk of complications from COVID-19. During the pandemic, health behaviours in the adult population changed, including the practice of leisure-time physical activity (LTPA). The objective of this work is to identify barriers to the practice of physical activity in leisure-time during the COVID-19 pandemic. In 2020, a survey on COVID-19 was carried out with participants from ELSA-Brasil. 5,639 civil servants from 5 Brazilian teaching and research institutions were interviewed, and 57.7% of whom were women. The median age was 59 years. LTPA during the period of social distancing was reported by 46.2% of the participants, however, 19.0% of those who did it in the pre-pandemic period stopped doing it. Among those engaged, 53.0% were women, half of them were married, 30.2% cared for children or the elderly, 71.3% reported meeting the recommendations to stay/work at home and 46.5% increased weight. Among the physically inactive, 45.0% were married, 70.1% complied with the recommendations to stay/work at home, 37.5% took care of children or the elderly, 50.8% gained weight and 97.6% performed housework. The profile of men is similar except that around 80.0% of them, active or inactive, were married. According to the results, the COVID-19 pandemic decreased the practice of LTPA among study participants. The inactive men and women performed more household chores, took care of children and the elderly and had an economic impact on the family, whether due to the loss of a source of income or because a member was left without a job, constituting possible barriers to the practice of LTPA. This finding could be used in the elaboration of policies aimed at promoting health by focusing on different aspects of barriers to LTPA.