scispace - formally typeset
Search or ask a question
Posted Content•

AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias

Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn A. Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John T. Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, Yunfeng Zhang 
03 Oct 2018-arXiv: Artificial Intelligence-
TL;DR: A new open source Python toolkit for algorithmic fairness, AI Fairness 360 (AIF360), released under an Apache v2.0 license to help facilitate the transition of fairness research algorithms to use in an industrial setting and to provide a common framework for fairness researchers to share and evaluate algorithms.
Abstract: Fairness is an increasingly important concern as machine learning models are used to support decision making in high-stakes applications such as mortgage lending, hiring, and prison sentencing. This paper introduces a new open source Python toolkit for algorithmic fairness, AI Fairness 360 (AIF360), released under an Apache v2.0 license {this https URL). The main objectives of this toolkit are to help facilitate the transition of fairness research algorithms to use in an industrial setting and to provide a common framework for fairness researchers to share and evaluate algorithms. The package includes a comprehensive set of fairness metrics for datasets and models, explanations for these metrics, and algorithms to mitigate bias in datasets and models. It also includes an interactive Web experience (this https URL) that provides a gentle introduction to the concepts and capabilities for line-of-business users, as well as extensive documentation, usage guidance, and industry-specific tutorials to enable data scientists and practitioners to incorporate the most appropriate tool for their problem into their work products. The architecture of the package has been engineered to conform to a standard paradigm used in data science, thereby further improving usability for practitioners. Such architectural design and abstractions enable researchers and developers to extend the toolkit with their new algorithms and improvements, and to use it for performance benchmarking. A built-in testing infrastructure maintains code quality.
Citations
More filters
Posted Content•
A Survey on Bias and Fairness in Machine Learning

[...]

Ninareh Mehrabi1, Fred Morstatter1, Nripsuta Saxena1, Kristina Lerman1, Aram Galstyan1 •
Information Sciences Institute1
23 Aug 2019-arXiv: Learning
TL;DR: This survey investigated different real-world applications that have shown biases in various ways, and created a taxonomy for fairness definitions that machine learning researchers have defined to avoid the existing bias in AI systems.
Abstract: With the widespread use of AI systems and applications in our everyday lives, it is important to take fairness issues into consideration while designing and engineering these types of systems. Such systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that the decisions do not reflect discriminatory behavior toward certain groups or populations. We have recently seen work in machine learning, natural language processing, and deep learning that addresses such challenges in different subdomains. With the commercialization of these systems, researchers are becoming aware of the biases that these applications can contain and have attempted to address them. In this survey we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined in order to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and how they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.

1,571 citations

Cites background or methods from "AI Fairness 360: An Extensible Tool..."

  • ...If the algorithm is allowed to modify the training data, then pre-processing can be used [11]....

    [...]

  • ...AI Fairness 360 (AIF360) is another toolkit developed by IBM in order to help moving fairness research algorithms into an industrial setting and to create a benchmark for fairness algorithms to get evaluated and an environment for fairness researchers to share their ideas [11]....

    [...]

  • ...allowed to change the learning procedure for a machine learning model, then in-processing can be used during the training of a model— either by incorporating changes into the objective function or imposing a constraint [11, 14]....

    [...]

  • ...If the algorithm can only treat the learned model as a black box without any ability to modify the training data or learning algorithm, then only post-processing can be used in which the labels assigned by the black-box model initially get reassigned based on a function during the post-processing phase [11, 14]....

    [...]

  • ...In addition, IBM’s AI Fairness 360 (AIF360) toolkit [11] has implemented many of the current fair learning algorithms and has demonstrated some of the results as demos which can be utilized by interested users to compare different methods with regards to different fairness measures....

    [...]

Journal Article•DOI•
The What-If Tool: Interactive Probing of Machine Learning Models

[...]

James Wexler1, Mahima Pushkarna1, Tolga Bolukbasi1, Martin Wattenberg1, Fernanda B. Viégas1, Jimbo Wilson1 •
Google1
01 Jan 2020-IEEE Transactions on Visualization and Computer Graphics
TL;DR: The What-If Tool is an open-source application that allows practitioners to probe, visualize, and analyze ML systems, with minimal coding, and lets practitioners measure systems according to multiple ML fairness metrics.
Abstract: A key challenge in developing and deploying Machine Learning (ML) systems is understanding their performance across a wide range of inputs. To address this challenge, we created the What-If Tool , an open-source application that allows practitioners to probe, visualize, and analyze ML systems, with minimal coding. The What-If Tool lets practitioners test performance in hypothetical situations, analyze the importance of different data features, and visualize model behavior across multiple models and subsets of input data. It also lets practitioners measure systems according to multiple ML fairness metrics. We describe the design of the tool, and report on real-life usage at different organizations.

296 citations

Proceedings Article•DOI•
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

[...]

Michael Madaio1, Luke Stark2, Jennifer Wortman Vaughan2, Hanna Wallach2•
Carnegie Mellon University1, Microsoft2
21 Apr 2020
TL;DR: It is found that AI fairness checklists could provide organizational infrastructure for formalizing ad-hoc processes and empowering individual advocates, and highlight aspects of organizational culture that may impact the efficacy of AI fairnessChecklists.
Abstract: Many organizations have published principles intended to guide the ethical development and deployment of AI systems; however, their abstract nature makes them difficult to operationalize. Some organizations have therefore produced AI ethics checklists, as well as checklists for more specific concepts, such as fairness, as applied to AI systems. But unless checklists are grounded in practitioners' needs, they may be misused. To understand the role of checklists in AI ethics, we conducted an iterative co-design process with 48 practitioners, focusing on fairness. We co-designed an AI fairness checklist and identified desiderata and concerns for AI fairness checklists in general. We found that AI fairness checklists could provide organizational infrastructure for formalizing ad-hoc processes and empowering individual advocates. We highlight aspects of organizational culture that may impact the efficacy of AI fairness checklists, and suggest future design directions.

257 citations

Cites background or methods from "AI Fairness 360: An Extensible Tool..."

  • ...Researchers have accompanied the proliferation of AI ethics principles by creating mathematical methods and software toolkits for developing fairer [8, 75, 76], more interpretable [72], and privacy-preserving AI systems [45]....

    [...]

  • ...Both concurrently and in response to these principles, researchers have created mathematical methods and software toolkits for developing fairer [8, 75, 76], more interpretable [72], and privacy-preserving AI systems [45]....

    [...]

Posted Content•
Fairness in Machine Learning: A Survey.

[...]

Simon Caton1, Christian Haas2•
University College Dublin1, University of Nebraska Omaha2
04 Oct 2020-arXiv: Learning
TL;DR: An overview of the different schools of thought and approaches to mitigating (social) biases and increase fairness in the Machine Learning literature is provided, organises approaches into the widely accepted framework of pre-processing, in- processing, and post-processing methods, subcategorizing into a further 11 method areas.
Abstract: As Machine Learning technologies become increasingly used in contexts that affect citizens, companies as well as researchers need to be confident that their application of these methods will not have unexpected social implications, such as bias towards gender, ethnicity, and/or people with disabilities. There is significant literature on approaches to mitigate bias and promote fairness, yet the area is complex and hard to penetrate for newcomers to the domain. This article seeks to provide an overview of the different schools of thought and approaches to mitigating (social) biases and increase fairness in the Machine Learning literature. It organises approaches into the widely accepted framework of pre-processing, in-processing, and post-processing methods, subcategorizing into a further 11 method areas. Although much of the literature emphasizes binary classification, a discussion of fairness in regression, recommender systems, unsupervised learning, and natural language processing is also provided along with a selection of currently available open source libraries. The article concludes by summarising open challenges articulated as four dilemmas for fairness research.

240 citations

Posted Content•
Aequitas: A Bias and Fairness Audit Toolkit.

[...]

Pedro Saleiro, Benedict Kuester, Abby Stevens, Ari Anisfeld, Loren Hinkson, Jesse London, Rayid Ghani 
14 Nov 2018-arXiv: Learning
TL;DR: Aequitas is an open source bias and fairness audit toolkit that is an intuitive and easy to use addition to the machine learning workflow, enabling users to seamlessly test models for several bias and Fairness metrics in relation to multiple population sub-groups.
Abstract: Recent work has raised concerns on the risk of unintended bias in AI systems being used nowadays that can affect individuals unfairly based on race, gender or religion, among other possible characteristics. While a lot of bias metrics and fairness definitions have been proposed in recent years, there is no consensus on which metric/definition should be used and there are very few available resources to operationalize them. Therefore, despite recent awareness, auditing for bias and fairness when developing and deploying AI systems is not yet a standard practice. We present Aequitas, an open source bias and fairness audit toolkit that is an intuitive and easy to use addition to the machine learning workflow, enabling users to seamlessly test models for several bias and fairness metrics in relation to multiple population sub-groups. Aequitas facilitates informed and equitable decisions around developing and deploying algorithmic decision making systems for both data scientists, machine learning researchers and policymakers.

174 citations

References
More filters
UCI Machine Learning Repository

[...]

A. Asuncion
01 Jan 2007

17,341 citations

Proceedings Article•
Equality of opportunity in supervised learning

[...]

Moritz Hardt1, Eric Price2, Nathan Srebro•
Google1, University of Texas at Austin2
05 Dec 2016
TL;DR: This work proposes a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features and shows how to optimally adjust any learned predictor so as to remove discrimination according to this definition.
Abstract: We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy. We enourage readers to consult the more complete manuscript on the arXiv.

2,690 citations

"AI Fairness 360: An Extensible Tool..." refers methods in this paper

  • ...Post-processing algorithms: Equalized odds postprocessing (Hardt et al., 2016) solves a linear program to find probabilities with which to change output labels to optimize equalized odds....

    [...]

  • ..., 2012) Post-processing Equalized odds post-processing (Hardt et al., 2016) Algorithms Calibrated eq....

    [...]

Proceedings Article•
Scaling up the accuracy of Naive-Bayes classifiers: a decision-tree hybrid

[...]

Ron Kohavi
02 Aug 1996
TL;DR: A new algorithm, NBTree, is proposed, which induces a hybrid of decision-tree classifiers and Naive-Bayes classifiers: the decision-Tree nodes contain univariate splits as regular decision-trees, but the leaves contain Naïve-Bayesian classifiers.
Abstract: Naive-Bayes induction algorithms were previously shown to be surprisingly accurate on many classification tasks even when the conditional independence assumption on which they are based is violated. However, most studies were done on small databases. We show that in some larger databases, the accuracy of Naive-Bayes does not scale up as well as decision trees. We then propose a new algorithm, NBTree, which induces a hybrid of decision-tree classifiers and Naive-Bayes classifiers: the decision-tree nodes contain univariate splits as regular decision-trees, but the leaves contain Naive-Bayesian classifiers. The approach retains the interpretability of Naive-Bayes and decision trees, while resulting in classifiers that frequently outperform both constituents, especially in the larger databases tested.

1,667 citations

"AI Fairness 360: An Extensible Tool..." refers methods in this paper

  • ...We currently provide an interface to seven popular datasets: Adult Census Income (Kohavi, 1996), German Credit (Dheeru & Karra Taniskidou, 2017), ProPublica Recidivism (COMPAS) (Angwin et al., 2016), Bank Marketing (Moro et al., 2014), and three versions of Medical Expenditure Panel Surveys (AHRQ, 2015; 2016)....

    [...]

  • ...We currently provide an interface to seven popular datasets: Adult Census Income (Kohavi, 1996), German Credit (Dheeru & Karra Taniskidou, 2017), ProPublica Recidivism (COMPAS) (Angwin et al., 2016), Bank Marketing (Moro et al., 2014), and three versions of Medical Expenditure Panel Surveys (AHRQ,…...

    [...]

  • ...The processed Adult Census Income, German Credit, and COMPAS datasets contain 45,222, 1,000 and 6,167 records respectively....

    [...]

  • ...An example result for Adult Census Income dataset with race as protected attribute is shown in Figure 5....

    [...]

  • ...C.1 Datasets C.1.1 Adult Census Income For protected attribute sex, Male is privileged, and Female is unprivileged....

    [...]

Proceedings Article•
Learning Fair Representations

[...]

Rich Zemel1, Yu Wu1, Kevin Swersky1, Toni Pitassi1, Cynthia Dwork2 •
University of Toronto1, Microsoft2
16 Jun 2013
TL;DR: A learning algorithm for fair classification that achieves both group fairness (the proportion of members in a protected group receiving positive classification is identical to the proportion in the population as a whole), and individual fairness (similar individuals should be treated similarly).
Abstract: We propose a learning algorithm for fair classification that achieves both group fairness (the proportion of members in a protected group receiving positive classification is identical to the proportion in the population as a whole), and individual fairness (similar individuals should be treated similarly). We formulate fairness as an optimization problem of finding a good representation of the data with two competing goals: to encode the data as well as possible, while simultaneously obfuscating any information about membership in the protected group. We show positive results of our algorithm relative to other known techniques, on three datasets. Moreover, we demonstrate several advantages to our approach. First, our intermediate representation can be used for other classification tasks (i.e., transfer learning is possible); secondly, we take a step toward learning a distance metric which can find important dimensions of the data for classification.

1,444 citations

"AI Fairness 360: An Extensible Tool..." refers background in this paper

  • ...The metrics therein are the group fairness measures of disparate (DI) and statistical parity difference (SPD) — the ratio and difference, respectively, of the base rate conditioned on the protected attribute — and the individual fairness measure consistency defined by Zemel et al. (2013)....

    [...]

  • ..., 2017) Algorithms Learning fair representations (Zemel et al., 2013) Disparate impact remover (Feldman et al....

    [...]

  • ...Learning fair representations (Zemel et al., 2013) finds a latent representation that encodes the data well but obfuscates information about protected attributes....

    [...]

Proceedings Article•DOI•
Certifying and Removing Disparate Impact

[...]

Michael Feldman1, Sorelle A. Friedler1, John Moeller2, Carlos Scheidegger3, Suresh Venkatasubramanian2 •
Haverford College1, University of Utah2, University of Arizona3
10 Aug 2015
TL;DR: This work links disparate impact to a measure of classification accuracy that while known, has received relatively little attention and proposes a test for disparate impact based on how well the protected class can be predicted from the other attributes.
Abstract: What does it mean for an algorithm to be biased? In U.S. law, unintentional bias is encoded via disparate impact, which occurs when a selection process has widely different outcomes for different groups, even as it appears to be neutral. This legal determination hinges on a definition of a protected class (ethnicity, gender) and an explicit description of the process.When computers are involved, determining disparate impact (and hence bias) is harder. It might not be possible to disclose the process. In addition, even if the process is open, it might be hard to elucidate in a legal setting how the algorithm makes its decisions. Instead of requiring access to the process, we propose making inferences based on the data it uses.We present four contributions. First, we link disparate impact to a measure of classification accuracy that while known, has received relatively little attention. Second, we propose a test for disparate impact based on how well the protected class can be predicted from the other attributes. Third, we describe methods by which data might be made unbiased. Finally, we present empirical evidence supporting the effectiveness of our test for disparate impact and our approach for both masking bias and preserving relevant information in the data. Interestingly, our approach resembles some actual selection practices that have recently received legal scrutiny.

1,434 citations

"AI Fairness 360: An Extensible Tool..." refers background or methods in this paper

  • ...Disparate impact remover (Feldman et al., 2015) edits feature values to increase group fairness while preserving rank-ordering within groups....

    [...]

  • ...It includes several bias detection metrics as well as bias mitigation methods, including disparate impact remover (Feldman et al., 2015), prejudice remover (Kamishima et al., 2012), and two-Naive Bayes (Calders & Verwer, 2010)....

    [...]

Related Papers (5)
Fairness through awareness

[...]

08 Jan 2012
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, Richard S. Zemel 
Equality of opportunity in supervised learning

[...]

05 Dec 2016
Moritz Hardt, Eric Price, Nathan Srebro
Certifying and Removing Disparate Impact

[...]

10 Aug 2015
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, Suresh Venkatasubramanian 
Learning Fair Representations

[...]

16 Jun 2013
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, Cynthia Dwork 
Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

[...]

01 Jun 2017
Alexandra Chouldechova
Trending Questions (1)
Does AI Debias Recruitment? Race, Gender, and AI’s “Eradication of Difference”?

The paper does not directly address the question of whether AI debiases recruitment based on race and gender. The paper focuses on introducing the AI Fairness 360 toolkit for detecting and mitigating algorithmic bias.

See answers from 5 other papers