Awesome

Overview

As AI systems become more prevalent in society, we face bigger and tougher societal challenges. Given many of these challenges have not been faced before, practitioners will face scenarios that will require dealing with hard ethical and societal questions.

There has been a large amount of content published which attempts to address these issues through “Principles”, “Ethics Frameworks”, “Checklists” and beyond. However navigating the broad number of resources is not easy.

This repository aims to simplify this by mapping the ecosystem of guidelines, principles, codes of ethics, standards and regulation being put in place around artificial intelligence.

Quick links to sections in this page

Other relevant resources

Roadmap

Main Content

High Level Frameworks and Principles

• AI & Machine Learning 8 principles for Responsible ML - The Institute for Ethical AI & Machine Learning has put together 8 principles for responsible machine learning that are to be adopted by individuals and delivery teams designing, building and operating machine learning systems.
• An Evaluation of Guidelines - The Ethics of Ethics - A research paper that analyses multiple Ethics principles
• Association for Computer Machinery's Code of Ethics and Professional Conduct - This is the code of ethics that has been put together in 1992 by the Association for Computer Machinery and updated in 2018. The Code is designed to inspire and guide the ethical conduct of all computing professionals, including current and aspiring practitioners, instructors, students, influencers, and anyone who uses computing technology in an impactful way. Additionally, the Code serves as a basis for remediation when violations occur. The Code includes principles formulated as statements of responsibility, based on the understanding that the public good is always the primary consideration.
• From What to How: An initial review of publicly available AI Ethics Tools, Methods and Research to translate principles into practices - A paper published by the UK Digital Catapult that aims to identify and present the gap between principles and their practical applications.
• European Commission's Guidelines for Trustworthy AI - The Ethics Guidelines for Trustworthy Artificial Intelligence (AI) is a document prepared by the High-Level Expert Group on Artificial Intelligence (AI HLEG). This independent expert group was set up by the European Commission in June 2018, as part of the AI strategy announced earlier that year.
• IEEE's Ethically Aligned Design - A Vision for Prioritizing Human Wellbeing with Artificial Intelligence and Autonomous Systems that encourages technologists to prioritize ethical considerations in the creation of autonomous and intelligent technologies.
• Montréal Declaration for a responsible development of artificial intelligence - ethical principles and values that promote the fundamental interests of people and group created as an initiative by Université de Montréal
• Oxford's Recommendations for AI Governance - A set of recommendations from Oxford's Future of Humanity institute which focus on the infrastructure and attributes required for efficient design, development, and research around the ongoing work building & implementing AI standards.
• PWC's Responsible AI - PWC has put together a survey and a set of principles that abstract some of the key areas they've identified for responsible AI.
• Singapore Data Protection Govt Commission's AI Governance Principles - The Singapore government's Personal Data Protection Commission has put together a set of guiding principles towards data protection and human involvement in automated systems, and comes with a report that breaks down the guiding principles and motivations.
• Toronto Declaration Protecting the right to equality and non-discrimination in machine learning systems by accessnow.
• UK Government's Data Ethics Framework Principles - A resource put together by the Department for Digital, Culture, Media and Sport (DCMS) which outlines an overview of data ethics, together with a 7-principle framework.
• Algorithm charter for Aotearoa New Zealand - The Algorithm Charter for Aotearoa New Zealand is an evolving piece of work that needs to respond to emerging technologies and also be fit-for-purpose for government agencies.
• Montreal AI Ethics Institute State of AI Ethics June 2020 Report - A resource put together by the Montreal AI Ethics Institute that captures the most relevant research and reporting in the domain of AI ethics between March 2020 and June 2020.
• Montreal AI Ethics Institute State of AI Ethics October 2020 Report - A resource put together by the Montreal AI Ethics Institute that captures the most relevant research and reporting in the domain of AI ethics between July 2020 and October 2020.
• Technical and Organizational Best Practices - A resource put together by Foundation for Best Practices in Machine Learning (FBPML) with technical guidelines (e.g. fairness and non-discrimination, monitoring and maintenance, data quality, product traceability, explainability) and organizational guidelines (e.g. data governance, product management, human resources management, compliance and auditing). Community contributions are welcome via the FBPML Wiki.
• Understanding artificial intelligence ethics and safety - A guide for the responsible design and implementation of AI systems in the public sector by David Leslie from the Alan Turing Institute.
• Declaration for responsible and intelligent data practice - a shared vision of what best practice in data looks like by Open Data Manchester.
• Recommendation on the Ethics of Artificial Intelligence - The Recommendation by the UNESCO is a comprehensive international framework aiming to shape the development and use of AI technologies and establishes a set of values in line with the promotion and protection of human rights, human dignity, and environmental sustainability. It has been adopted by acclamation by 193 Member States at UNESCO’s General Conference in November 2021. For more information, refer to UNESCO's 2023 publication on key facts here.

Processes and Checklists

• AI RFX Procurement Framework - A procurement framework for evaluating the maturity of machine learning systems put together by cross functional teams of academics, industry practitioners and technical individuals at The Institute for Ethical AI & Machine Learning to empower industry practitioners looking to procure machine learning suppliers.
• Checklist for data science projects – Deon by DrivenData is a command line tool that allows you to easily add an ethics checklist to your data science projects.
• Designing Ethical AI Experiences Checklist and Agreement - document to guide the development of accountable, de-risked, respectful, secure, honest, and usable artificial intelligence (AI) systems with a diverse team aligned on shared ethics. Carnegie Mellon University, Software Engineering Institute.
• Ethical OS Toolkit - A toolkit that dives into 8 risk zones to assess the potential challenges that a technology team may face, together with 14 scenarios to provide examples, and 7 future-proofing strategies to help take ethical action.
• Ethics Canvas - A resource inspired by the traditional business canvas, which provides an interactive way to brainstorm potential risks, opportunities and solutions to ethical challenges that may be faced in a project using post-it note-like approach.
• Kat Zhou's Design Ethically Resources - A set of workshops that can be organised across teams to identify challenges, assess current risks and take action on potential issues around ethical challenges that may be faced.
• Markula Center's Ethical Toolkit for Engineering/Design Practice - A practical and comprehensible toolkit with seven components to aid practitioners reflect, and judge the moral grounds in which they are operating.
• San Francisco City's Ethics & Algorithms Toolkit - A risk management framework for government leaders and staff who work with algorithms, providing a two part assessment process including an algorithmic assessment process, and a process to address the risks.
• UK Government's Data Ethics Workbook - A resource put together by the Department for Digital, Culture, Media and Sport (DCMS) which provides a set of questions that can be asked by practitioners in the public sector, which address each of the principles in their Data Ethics Framework Principles.
• World Economic Forum's Guidelines for Procurement - The WEF has put together a set of guidelines for governments to be able to safely and reliably procure machine learning related systems, which has been trialled with the UK government.
• Machine Learning Assurance - Quick look at machine learning assurance: process of recording, understanding, verifying, and auditing machine learning models and their transactions.
• ODEP's Checklist for Employers: Facilitating the Hiring of People with Disabilities Through the Use of eRecruiting Screening Systems, Including AI - The Employer Assistance and Resource Network on Disability Inclusion (EARN) and the Partnership on Employment & Accessible Technology (PEAT), which are both funded through the U.S. Department of Labor's Office of Disability Employment Policy (ODEP), collaborated on an inclusive AI checklist for employers. The checklist provides direction for leadership, human resources personnel, equal employment opportunity managers, and procurement officers for reviewing AI tools used in recruiting and candidate assessment for fairness and inclusion of individuals with disabilities.
• Microsoft AI Fairness Checklist
• US NIST AI Risk Management Framework - The Framework is intended to help developers, users and evaluators of AI systems better manage AI risks which could affect individuals, organizations, society, or the environment.

Interactive and Practical Tools

• Aequitas' Bias & Fairness Audit Toolkit - The Bias Report is powered by Aequitas, an open-source bias audit toolkit for machine learning developers, analysts, and policymakers to audit machine learning models for discrimination and bias, and make informed and equitable decisions around developing and deploying predictive risk-assessment tools.
• Awesome Machine Learning Production List - A list of tools and frameworks that support the design, development and operation of production machine learning systems, currently maintained by The Institute for Ethical AI & Machine Learning.
• Cape Python - Easily apply privacy-enhancing techniques for data science and machine learning tasks in Pandas and Spark. Can be used in conjunction with Cape Core to collaborate on privacy policies and distribute those policies for data projects across teams and organizations.
• eXplainability Toolbox - The Institute for Ethical AI & Machine Learning proposal for an extended version of the traditional data science process which focuses on algorithmic bias and explainability, to ensure a baseline of risks around undesired biases can be mitigated.
• FAT Forensics is a Python toolkit for evaluating Fairness, Accountability and Transparency of Artificial Intelligence systems. It is built on top of SciPy and NumPy, and distributed under the 3-Clause BSD license (new BSD).
• IBM's AI Explainability 360 Open Source Toolkit - This is IBM's toolkit that includes large number of examples, research papers and demos implementing several algorithms that provide insights on fairness in machine learning systems.
• Linux Foundation AI Landscape - The official list of tools in the AI landscape curated by the Linux Foundation, which contains well maintained and used tools and frameworks.
• Taking action on digital ethics from Avanade
• Microsoft Fairlearn - An open-source toolkit for assessing and improving fairness in machine learning products developed by Microsoft
• Microsoft Interpret ML - An open-source toolkit for improving explainability/interpretability developed by Microsoft
• Alibi - An open-source Python library for machine learning model inspection and interpretation.

Industry standards initiatives

• ACS Code of Professional Conduct - PDF - Australian ICT (Information and Communication Technology) sector professional organization.
• Association for Computer Machinery's Code of Ethics and Professional Conduct - This is the code of ethics that has been put together in 1992 by the Association for Computer Machinery and updated in 2018. The Code is designed to inspire and guide the ethical conduct of all computing professionals, including current and aspiring practitioners, instructors, students, influencers, and anyone who uses computing technology in an impactful way. Additionally, the Code serves as a basis for remediation when violations occur. The Code includes principles formulated as statements of responsibility, based on the understanding that the public good is always the primary consideration.
• IEEE Global Initiative for Ethical Considerations in Artificial Intelligence (AI) and Autonomous Systems (AS) - IEEE Approved Standards Projects specifically focused on the Ethically Aligned Design principles, and includes 14 (P700X) standards which cover topics from data collection to privacy, to algorithmic bias and beyond.
• ISO/IEC's Standards for Artificial Intelligence - The ISO's initiative for Artificial Intelligence standards, which include a large set of subsequent standards ranging across Big Data, AI Terminology, Machine Learning frameworks, etc.

Online Courses and Learning Resources

• Udacity's Secure & Private AI Course - Free course by Udacity which introduces three cutting-edge technologies for privacy-preserving AI: Federated Learning, Differential Privacy, and Encrypted Computation.
• Data science ethics - Free course by Prof. Jagadish from the University of Michigan (via Coursera) that covers data ownership, privacy and anonymity, data validity, and algorithmic fairness.
• Practical Data Ethics - Free course by Rachel Thomas from the University of San Francisco Data Institute (via fast.ai) that covers disinformation, bias and fairness, foundations of ethics, privacy and surveillance, algorithmic colonialism
• Bias and Discrimination in AI - Free course by the University of Montreal and IVADO (via edX) about the discriminatory effects of algorithmic decision-making and responsible machine learning (institutional and technical strategies to identify and address bias).
• Intro to AI ethics - Free course by Kaggle introducing the base concepts of ethics in AI and how to mitigate related problems.

Research and Industry Newsletters

• Import AI - A newsletter curated by OpenAI's Jack Clark which curates the most resent and relevant AI research, as well as relevant societal issues that intersect with technical AI research.
• Matt's thoughts in between - Newsletter curated by Entrepreneur First CEO Matt Clifford that provides a curated critical analysis on topics surrounding geopolitics, deep tech startups, economics and beyond.
• The Machine Learning Engineer - A newsletter curated by The Institute for Ethical AI & Machine Learning that contains curated articles, tutorials and blog posts from experienced Machine Learning professionals and includes insights on best practices, tools and techniques in machine learning explainability, reproducibility, model evaluation, feature analysis and beyond.
• Montreal AI Ethics Institute Weekly AI Ethics newsletter - A weekly newsletter curated by Abhishek Gupta and his team at the Montreal AI Ethics Institute that presents accessible summaries of technical and academic research papers along with commentary on the latest in the domain of AI ethics.

Regulation and Policy

Austria

• Artificial Intelligence Mission Austria 2030 - Shaping the Future of Artificial Intelligence in Austria. The Austrian ministry for Innovation and Technology published their vision for AI until 2030.

China

• Beijing AI Principles - initiative for the research, development, use, governance and long-term planning of AI, calling for its healthy development to support the construction of a human community with a shared future, and the realization of beneficial AI for humankind and nature.
• China Internet Security Law - China's law which enacted to increase cybersecurity and national security, safeguard cyberspace sovereignty and public interest, protect the legitimate rights and interests of citizens, legal persons and other organisations, and promote healthy economic and social development (and was argued by the Chinese ministry for industry and information that this law justified the means of pursuing the "Going Out" strategy China has persisted on ever since 1999). KPMG's summary of the Cybersecurity Law. Center for strategic & international studies overview of China's new Data Privacy law
• China's Personal Information Security Specification (Translation) - The Chinese Government's first major digital privacy rules which took effect in May 2018, which lays out granular guidelines for consent and how personal data should be collected, used and shared. Center for strategic & international studies overview of the specification
• China's Administrative Provisions on Information Services on Microblogs - China's provisions which require microblogging sites (social media sites) to obtain relevant credentials by law, verify users' real identities, establish mechanisms for dispelling and refuting rumors, etc. Summary of rules by the US Law Library of Congress.
• Decision on strengthening the protection of online information - The Standing Committee of the National People's Congress (NPC) of the People's Republic of China adopted the decision on strengthening the protection of online information - this is an act that contains 12 clauses applicable to entities both in the public and private sectors in respect to the collection and processing of electronic personal information on the internet.
• Personal Data Protection Act - The personal data protection act of the Republic of China, which is enacted to regulate the collection, processing and use of personal data as so to prevent harm on personality rights, and to facilitate the proper use of personal data.

Dubai

• Smart Dubai Artificial Intelligence Principles and Ethics - Ethical AI Toolkit - created to provide practical help across a city ecosystem. It supports industry, academia and individuals in understanding how AI systems can be used responsibly. It consists of principles and guidelines, and a self-assessment tool for developers to assess their platforms.

European Union

• Ethics Guidelines for Trustworthy AI - European Commission document prepared by the High-Level Expert Group on Artificial Intelligence (AI HLEG).
• General Data Protection Regulation GDPR - Legal text for the EU GDPR regulation 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC
• GDPR.EU Guide - A project co-funded by the Horizon 2020 Framework programme of the EU which provides a resource for organisations and individuals researching GDPR, including a library of straightforward and up-to-date information to help organisations achieve GDPR compliance (Legal Text).

Singapore

• Data Protection Act 2012 - The Personal Data Protection Act 2012 (the "Act") sets out the law on data protection in Singapore. Apart from establishing a general data protection regime, the Act also regulates telemarketing practices.
• Protection from Online Falsehoods and Manipulation Act 2019 - An act to prevent the electronic communication in Singapore of false statements of fact, to suppress support for and counteract the effects of such communication, to safeguard against the use of online accounts for such communication and for information manipulation, to enable measures to be taken to enhance transparency of online political advertisements and for related matters.

United States of America

• The White House Executive Order in AI - The USA Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence
• California Consumer Privacy Act (CCPA) - Legal text for California's consumer privacy act
• EU-U.S. and Swiss-U.S. Privacy Shield Frameworks - The EU-U.S. and Swiss-U.S. Privacy Shield Frameworks were designed by the U.S. Department of Commerce and the European Commission and Swiss Administration to provide companies on both sides of the Atlantic with a mechanism to comply with data protection requirements when transferring personal data from the European Union and Switzerland to the United States in support of transatlantic commerce.
• Fair Credit Reporting Act 2018 - The Fair Reporting Act is a federal law that regulates the collection of consumers' credit information and access to their credit reports.
• Gramm-Leach-Billey Act (for financial institutions) - The Graham-Leach-Billey Act requires financial institutions (companies that offer consumers financial projects or services like loans, financial, or investment advice, or insurance) to explain their information-sharing practices to their customers and to safeguard sensitive data.
• Health Insurance Portability and Accountability Act of 1996 - The HIPAA required the secretary of the US department of health and human services (HHS) to develop regulations protecting the privacy and security of certain health information, which then HHS published what is known as the HIPAA Privacy Rule, and the HIPAA Security Rule.
• Executive Order on Maintaining American Leadership in AI - Official mandate by the President of the US to
• Privacy Act of 1974 - The privacy act of 1974 which establishes a code of fair information practices that governs the collection, maintenance, use and dissemination of information about individuals that is maintained in systems of records by federal agencies.
• Privacy Protection Act of 1980 - The Privacy Protection Act of 1980 protects journalists from being required to turn over to law enforcement any work product and documentary materials, including sources, before it is disseminated to the public.
• DoD's Ethical Principles for AI - The U.S.A. Department of Defense responsible AI guidelines for tech contractors. The guidelines provide a step-by-step process to follow during the planning, development, and deployment phases of the technical lifecycle.

United Kingdom

• UK Data Protection Act of 2018 - The DPA 2018 enacts the GDPR into UK Law, however in doing so has included various "derogations" as permitted by the GDPR, resulting in some key differenced (which although small are not of insignificance impact and may have a greater impact after Brexit).
• The Information Commissioner's Office guide to Data Protection - This guide is for data protection officers and others who have day-to-day responsibility for data protection. It is aimed at small and medium-sized organisations, but it may be useful for larger organisations too.