Mathematical construction of a low-bias high-resolution deprivation index for the United States
The construction of deprivation indices is complicated by the inherent ambiguity in defining deprivation as well as the potential for partisan manipulation. Nevertheless, deprivation indices provide an essential tool for mitigating the effects of deprivation and reducing it through policy interventions. Here we demonstrate the construction of a deprivation index using diffusion maps, a manifold learning technique capable of finding the variables that optimally describe the variations in a dataset in the sense of preserving pairwise relationships among the data points. The method is applied to the 2010 US decennial census. In contrast to other methods the proposed procedure does not select particular columns from the census, but rather constructs an indicator of deprivation from the complete dataset. Due to its construction the proposed index does not introduce biases except those already present in the source data, does not require normative judgment regarding the desirability of certain life styles, and is highly resilient against attempts of partisan manipulation. We demonstrate that the new index aligns well with established income-based deprivation indices but deviates in aspects that are perceived as problematic in some of the existing indices. The proposed procedure provides an efficient way for constructing accurate, high resolution indices. These indices can thus have the potential to become powerful tools for the academic study of social structure as well as political decision making.