This page contains links to relevant workshops, projects, and principle documents.
Workshops and Events
- Fairness, Accountability, and Transparency in Machine Learning (FAT/ML), NIPS 2014, ICML 2015, DTL 2016, KDD 2017
- Fairness in User Modeling, Adaptation and Personalization (FairUMAP), UMAP 2018
- International Workshop on Software Fairness (FairWare), ICSE 2018
- Workshop on Responsible Recommendation (FAT/Rec), RECSYS 2017
- Workshop on Data and Algorithmic Bias, CIKM 2017
- Singapore Workshop on Fairness, Accountability and Transparency in AI and Big Data, 2017
- Ethics in Natural Language Processing, EACL 2017
- Workshop on Fairness, Accountability, and Transparency on the Web, WWW 2017
- Special Session on Explainability of Learning Machines, IJCNN 2017
- Workshop on Data and Algorithmic Transparency (DAT), 2016
- The Human Use of Machine Learning: An Interdisciplinary Workshop, IEEE SMC
- International Workshop on Privacy and Discrimination in Data Mining, IEEE ICDM 2016
- Machine Learning and the Law, NIPS 2016
- Interpretable Machine Learning for Complex Systems, NIPS 2016
- Workshop on Human Interpretability in Machine Learning, ICML 2016
- Workshop on the Ethics of Online Experimentation, WSDM 2016
- Auditing Algorithms From the Outside: Methods and Implications, ICWSM 2015
- Discrimination and Privacy-Aware Data Mining, IEEE ICDM 2012
- Workshop on Novelty and Diversity in Recommender Systems, ACM RECSYS 2011
- Governing Algorithms
- Auditing Algorithms NSF Workshop
Numerous groups are conducting research, building tools, and developing policy statements related to fairness, accountability, and transparency in socio-technical systems.
- Optimizing Government: Policy Challenges in the Machine Learning Age, University of Pennsylvania
- Responsible Data Science, Eindhoven University of Technology, Leiden University, University of Amsterdam, Radboud University Nijmegen, Tilburg University, VU University, Amsterdam Medical Center, VU Medical Center, Leiden University Medical Center, Delft University of Technology, and CWI (National Research Institute for Mathematics and Computer Science)
- Explainable Artificial Intelligence (XAI), Defense Advanced Research Projects Agency
- Computer science and legal methods for enforcing the personal rights of non-discrimination and privacy in ICT systems, Italian Fund for Basic Research
- Data Mining without Discrimination, Netherlands Organisation for Scientific Research
- On algorithmic fairness, discrimination and disparate impact, Haverford College
- The GenderMag Project, Oregon State University and many others
- Auditing Algorithms @ Northeastern, Northeastern University
- Privacy, Accountability, Compliance, and Trust in Tomorrow's Internet (imPACT), Saarland University, Center for IT-Security, Privacy, and Accountability, Max-Planck Institute for Informatics, Max Planck Institute for Software Systems
Software and libraries that implement fair learning algorithms or facilitate algorithm auditing.
- FairML: Auditing Black-Box Predictive Models, an end-to-end toolbox for auditing predictive models by quantifying the relative significance of the model’s inputs.
- Lime: Explaining the predictions of any machine learning classifier, supports explaining individual predictions for text classifiers or classifiers that act on tables (e.g. arrays of numerical or categorical data).
- Fairness Measures: Datasets and software for detecting algorithmic discrimination.
- Fairness notation, definitions, data, and legality, presents mathematical notions of fairness from the literature, helpfully rewritten using consistent notation and presented with context.
- Aequitas, an open-source bias audit toolkit for machine learning developers, analysts, and policymakers to audit machine learning models for discrimination and bias, and make informed and equitable decisions around developing and deploying predictive risk-assessment tools.
Principles and Best Practices
Guidelines and documentation developed by standards bodies, practitioners, and researchers.
A growing number of organizations inside and outside academia are dedicated to promoting the ideals of fairness, accountability, and transparency of algorithmic systems.
- AI Now Institute (New York University), @AINowInstitute
- AlgoPolitics (PersonalDataIO), @AlgoPolitics
- Algorithmic Society Lab (Imperial College London)
- Berkman Klein Center for Internet & Society (Harvard University), @BKCHarvard
- Center for Analytics in Learning and Teaching (Colorado State University)
- Center for Democracy & Technology, @cendemtech
- Center for Information Policy Research (University of Wisconsin-Milwaukee)
- Center for Information Technology Policy (Princeton University), @PrincetonCITP
- Center for Media and Citizenship (University of Virginia), @mediaandcitizen
- Citris and the Banatao Institute (University of California - Berkeley,Davis, Merced, SantaCruz), @citrisnews
- Data & Society, @datasociety
- Datafied Society Research Platform (Utrecht University)
- DataKind (New York), @datakind
- DataKind UK (London), @datakindUK
- Digital Ethics Lab (University of Oxford), @oxfordethicslab
- Future of Privacy Forum (Washington, DC), @futureofprivacy
- Information Ethics & Equity Institute (IEEI), @ethicsequity
- Internet Policy Research Initiative (IPRI) (Massachusetts Institute of Technology (MIT)), @MIT_IPRI
- ISI Foundation (Italy), @ISI_Fondazione
- LSE Data & Society (London School of Economics), @LSEDataSociety
- New York University Information Law Institute (New York University), @ILI_NYU
- Oxford Internet Institute (University of Oxford), @oiioxford
- Partnership on AI, @PartnershipAI
- PersonalDataIO (PersonalDataIO), @PersonalDataIO
- Pervasive Data Ethics for computational research (PERVADE) (University of Maryland, College Park), @pervade_team
- PrivacyInternational (United Kingdom), @privacyint
- RathenauNL (Netherlands), @RathenauNL
- Science And Technology Studies/Program on Science, Technology & Society (Harvard University), @HarvardSTS
- Technoscience Research Unit (University of Toronto)
- The Center for Technology, Society & Policy (University of California - Berkeley), @CTSPBerkeley
- The Digital Methods Initiative (University of Amsterdam)
- The Human Rights, Big Data and Technology Project, @HRBDTNews
- The LSE Truth, Trust and Technology Commission (London School of Economics)
- Upturn, @teamupturn
- Utrecht Data School (Utrecht University), @data_school
- VIRT-EU: Values and ethics in Innovation for Responsible Technology in Europe, @VIRT_EU
- Yale Information Society Project (Yale Law School), @yaleisp
- Reuters Institute for the Study of Journalism (University of Oxford), @risj_oxford
- Jheronimus Academy of Data Science (Eindhoven University of Technology / Tilburg University), @jadatascience