Big Data's Disparate Impact
TL;DR: In the absence of a demonstrable intent to discriminate, the best doctrinal hope for data mining's victims would seem to lie in disparate impact doctrine as discussed by the authors, which holds that a practice can be justified as a business necessity when its outcomes are predictive of future employment outcomes, and data mining is specifically designed to find such statistical correlations.
Abstract: Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with. Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society. Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm’s use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court.This Essay examines these concerns through the lens of American antidiscrimination law — more particularly, through Title VII’s prohibition of discrimination in employment. In the absence of a demonstrable intent to discriminate, the best doctrinal hope for data mining’s victims would seem to lie in disparate impact doctrine. Case law and the Equal Employment Opportunity Commission’s Uniform Guidelines, though, hold that a practice can be justified as a business necessity when its outcomes are predictive of future employment outcomes, and data mining is specifically designed to find such statistical correlations. Unless there is a reasonably practical way to demonstrate that these discoveries are spurious, Title VII would appear to bless its use, even though the correlations it discovers will often reflect historic patterns of prejudice, others’ discrimination against members of protected groups, or flaws in the underlying dataAddressing the sources of this unintentional discrimination and remedying the corresponding deficiencies in the law will be difficult technically, difficult legally, and difficult politically. There are a number of practical limits to what can be accomplished computationally. For example, when discrimination occurs because the data being mined is itself a result of past intentional discrimination, there is frequently no obvious method to adjust historical data to rid it of this taint. Corrective measures that alter the results of the data mining after it is complete would tread on legally and politically disputed terrain. These challenges for reform throw into stark relief the tension between the two major theories underlying antidiscrimination law: anticlassification and antisubordination. Finding a solution to big data’s disparate impact will require more than best efforts to stamp out prejudice and bias; it will require a wholesale reexamination of the meanings of “discrimination” and “fairness.”
Citations
More filters
TL;DR: In this paper, a taxonomy of recent contributions related to explainability of different machine learning models, including those aimed at explaining Deep Learning methods, is presented, and a second dedicated taxonomy is built and examined in detail.
2,827 citations
Proceedings Article•
05 Dec 2016TL;DR: This work proposes a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features and shows how to optimally adjust any learned predictor so as to remove discrimination according to this definition.
Abstract: We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy. We enourage readers to consult the more complete manuscript on the arXiv.
2,690 citations
TL;DR: It is suggested that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.
Abstract: Health systems rely on commercial prediction algorithms to identify and help patients with complex health needs. We show that a widely used algorithm, typical of this industry-wide approach and affecting millions of patients, exhibits significant racial bias: At a given risk score, Black patients are considerably sicker than White patients, as evidenced by signs of uncontrolled illnesses. Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%. The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients. Thus, despite health care cost appearing to be an effective proxy for health by some measures of predictive accuracy, large racial biases arise. We suggest that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.
2,003 citations
TL;DR: This article showed that applying machine learning to ordinary human language results in human-like semantic biases and replicated a spectrum of known biases, as measured by the Implicit Association Test, using a widely used, purely statistical machine-learning model trained on a standard corpus of text from the World Wide Web.
Abstract: Machine learning is a means to derive artificial intelligence by discovering patterns in existing data. Here, we show that applying machine learning to ordinary human language results in human-like semantic biases. We replicated a spectrum of known biases, as measured by the Implicit Association Test, using a widely used, purely statistical machine-learning model trained on a standard corpus of text from the World Wide Web. Our results indicate that text corpora contain recoverable and accurate imprints of our historic biases, whether morally neutral as toward insects or flowers, problematic as toward race or gender, or even simply veridical, reflecting the status quo distribution of gender with respect to careers or first names. Our methods hold promise for identifying and addressing sources of bias in culture, including technology.
1,874 citations
TL;DR: It is argued that while this law will pose large challenges for industry, it highlights opportunities for computer scientists to take the lead in designing algorithms and evaluation frameworks which avoid discrimination and enable explanation.
Abstract: We summarize the potential impact that the European Union’s new General Data Protection Regulation will have on the routine use of machine learning algorithms. Slated to take effect as law across the EU in 2018, it will restrict automated individual decision-making (that is, algorithms that make decisions based on user-level predictors) which “significantly affect” users. The law will also effectively create a “right to explanation,” whereby a user can ask for an explanation of an algorithmic decision that was made about them. We argue that while this law will pose large challenges for industry, it highlights opportunities for computer scientists to take the lead in designing algorithms and evaluation frameworks which avoid discrimination and enable explanation.
1,500 citations
Cites background from "Big Data's Disparate Impact"
...Barocas & Selbst(2016) sum the problem up succinctly: “Big data claims to be neutral....
[...]
...Consequently, “unthinking reliance on data mining can deny members of vulnerable groups full participation in society” (Barocas & Selbst, 2016)....
[...]
...The link between geography and income may be obvious, but less obvious correlations—say between browsing time and income—are likely to exist within large enough datasets and can lead to discriminatory effects (Barocas & Selbst, 2016)....
[...]
...ising that concerns over discrimination have begun to take root in discussions over the ethics of big data. Barocas and Selbst sum the problem up succinctly: “Big data claims to be neutral. It isn’t” [4]. As the authors point out, machine learning depends upon data that has been collected from society, and to the extent that society contains inequality, exclusion or other traces of discrimination, so...
[...]
References
More filters
TL;DR: It is shown that easily accessible digital records of behavior, Facebook Likes, can be used to automatically and accurately predict a range of highly sensitive personal attributes including: sexual orientation, ethnicity, religious and political views, personality traits, intelligence, happiness, use of addictive substances, parental separation, age, and gender.
Abstract: We show that easily accessible digital records of behavior, Facebook Likes, can be used to automatically and accurately predict a range of highly sensitive personal attributes including: sexual orientation, ethnicity, religious and political views, personality traits, intelligence, happiness, use of addictive substances, parental separation, age, and gender. The analysis presented is based on a dataset of over 58,000 volunteers who provided their Facebook Likes, detailed demographic profiles, and the results of several psychometric tests. The proposed model uses dimensionality reduction for preprocessing the Likes data, which are then entered into logistic/linear regression to predict individual psychodemographic profiles from Likes. The model correctly discriminates between homosexual and heterosexual men in 88% of cases, African Americans and Caucasian Americans in 95% of cases, and between Democrat and Republican in 85% of cases. For the personality trait “Openness,” prediction accuracy is close to the test–retest accuracy of a standard personality test. We give examples of associations between attributes and Likes and discuss implications for online personalization and privacy.
2,232 citations
TL;DR: Three approaches for making the naive Bayes classifier discrimination-free are presented: modifying the probability of the decision being positive, training one model for every sensitive attribute value and balancing them, and adding a latent variable to the Bayesian model that represents the unbiased label and optimizing the model parameters for likelihood using expectation maximization.
Abstract: In this paper, we investigate how to modify the naive Bayes classifier in order to perform classification that is restricted to be independent with respect to a given sensitive attribute. Such independency restrictions occur naturally when the decision process leading to the labels in the data-set was biased; e.g., due to gender or racial discrimination. This setting is motivated by many cases in which there exist laws that disallow a decision that is partly based on discrimination. Naive application of machine learning techniques would result in huge fines for companies. We present three approaches for making the naive Bayes classifier discrimination-free: (i) modifying the probability of the decision being positive, (ii) training one model for every sensitive attribute value and balancing them, and (iii) adding a latent variable to the Bayesian model that represents the unbiased label and optimizing the model parameters for likelihood using expectation maximization. We present experiments for the three approaches on both artificial and real-life data.
750 citations
TL;DR: This work proposes a method that can eliminate proxy effects while maintaining predictive accuracy relative to an approach that restricts the use of contentious variables outright and illustrates the value of the proposed method using data from the Worker Profiling and Reemployment Services system.
Abstract: How should statistical models used for assigning prices or eligibility be implemented when there is concern about discrimination? In many settings, factors such as race, gender, and age are prohibited. However, the use of variables that correlate with these omitted characteristics {e.g., zip codes, credit scores) is often contentious. We provide a framework to address these issues and propose a method that can eliminate proxy effects while maintaining predictive accuracy relative to an approach that restricts the use of contentious variables outright. We illustrate the value of our proposed method using data from the Worker Profiling and Reemployment Services system. (7£LC53,J15,J65,J71) is a growing debate about how predictive statistical models should be implemented in settings where discrimination is a concern. Anti-discrimination policies typically ensure that predictive models exclude characteristics such as race, and (depending on the situation) age, gender, sexual orientation, etc. However, there is a large and contentious gray area concerning other variables that may be directly predictive but may also serve as proxies for these omitted characteristics. One prominent example of the debate on this issue came in the passage of California's Proposition 103 in 1988, which limited the variables that insurance companies could use in pricing automobile insurance. Most of the debate surrounded the use of location controls (e.g., zip code) and credit scores. The insurers argued that these types of variables were directly predictive of losses. On the other side, consumer advocates argued that these variables were clearly correlated with race and income (which are banned from use in insurance pricing) and were serving as proxies for the omitted characteristics. Of course, both arguments have merits - these variables are likely neither solely predictive nor purely proxies for omitted characteristics. In the case of Proposition 103, the arguments that the variables were proxies won the day, and the state mandated that insurers price their policies on a very small range of variables that excluded zip codes or credit scores.
60 citations
Posted Content•
TL;DR: The authors examines the particular way in which the focus on the defendant's state of mind in disparate impact cases has helped undermine the theory, despite the belief of many theorists that a thorough grounding in intent may be necessary to ensure the theory's survival.
Abstract: The disparate impact theory of liability in antidiscrimination law is in its fourth decade of doctrinal development, but the theory remains misunderstood, mislabeled, and misused. Despite the fact that the effects-based theory emerged as a method of proof theoretically unconcerned with the concept of intent - whether of the "good" or "bad" variety - all of the stakeholders using disparate impact theory have remained stuck on state of mind. This Article examines the particular way in which the focus on the defendant's state of mind in disparate impact cases has helped undermine the theory, despite the belief of many theorists that a thorough grounding in intent may be necessary to ensure the theory's survival. The Article suggests that recent events and more modern conceptions of evil make it possible for us to look past "good intentions," to revisit our notions of culpability and fault, and to reinvigorate a disparate impact theory focused solely on effects.
1 citations