In Uganda, the Karamojong Pastoralist and Batwa forest pygmy communities are disproportionately affected by the increasing frequency and magnitude of climate change impacts. Though these communities have long-standing traditional systems to adapt to a changing climate, policymakers and researchers often disregard the adaptations. Programs in the Karamoja region aim to make pastoralism more resilient to climate change, but most ignore pastoralism’s resilience to climate variability and instead focus on changing livestock systems, reducing livestock numbers, adopting crop growing, and diversifying to other economic activities. On the other hand, Batwa forest pygmies have long adapted to climate change in tropical forests by integrating their health systems to the ecosystem. This chapter maps out the policy implications of resilience building in poor communities marginalized by public policy. Gaps, constraints, and opportunities are discussed, in addition to lessons from existing community adaptations that build resilience to climate change.