Heritability is a central concept in the long-standing debate about nature versus nurture in biological and social sciences. However, existing notions of heritability are based on strong modeling assumptions, and the models are statistical rather than causal. We propose a new, counterfactual definition of heritability by adopting the potential outcomes model in causal inference. Our counterfactual heritability measures the importance of genetic inheritance by the average magnitude of difference between an individual with their hypothetical non-identical twin’that is exposed to the exact same environment. We provide bounds on the counterfactual heritability that can, in principle, be computed from observational data. We then compare counterfactual heritability and its associated bounds with common notions of heritability in population-based studies, twin and sibling studies, and plant breeding experiments. Our results and comparisons highlight the importance of clarifying the causal structural assumptions and counterfactual comparisons in reasoning about heritability.