A Framework for Building Trustworthy and Actionable AI | by Tanusree De
How to Minimize Risks of AI and Accelerate its Adoption
Artificial Intelligence has come a long way! What was once a fantasy of the imagination of some famous science fiction writers, is all-pervasive today. AI has become a part of our lives; it is used everywhere, and it has made lot of things convenient for us. In many industries, AI powered cognitive automation has helped in automating complex business processes which in turn has largely improved process efficiency. AI systems are also being incorporated into a wide variety of decision-making processes, both in industry and policy fields. In healthcare, AI has brought in some incredible breakthroughs. However, in many cases, there are also potential risks associated with the usage of AI and there have been real examples where AI has been misused, either knowingly or unknowingly. There can be possibility of AI having ethical issues, or inadequate security, or not being safe or not preserving privacy of personal or sensitive data. Moreover, an AI system is a black box. It gives us a decision, but it does not tell us about the rationale behind the decision, how it has arrived at the decision. Due to its lack of transparency, it becomes difficult to trust the decisioning of AI systems. All in all, though people are aware of the incredible power and benefits of AI, they are equally concerned about the potential risks associated with it. This has naturally called for regulation of AI and the need for Trustworthy and Actionable AI.
What is AI Regulation?
The regulation of artificial intelligence is the development of public sector policies and laws to integrate ethical standards into the design and implementation of AI-enabled technologies.
Now, how to define ethical standards for AI? What should be the premise? The first and foremost principle is to ensure responsible use of AI. Even before an AI system is developed for a use case, one should analyze the potential risks and benefits associated with the use of AI for that particular use case. Depending on the extent of potential risk versus the benefit it yields, one should take decisions to integrate some stringent requirements in the design of the system to make it safe, secure, ethical, transparent and reliable. This is exactly what the European Union Commission has mentioned in its proposal for Regulation of AI published on 21 April 2021. Source: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206
The EU Commission has mentioned that their key objective for regulation of AI is to ensure safety and adherence to fundamental rights and Union values. The Commission has proposed a risk-based approach for regulation of AI, wherein, they have come up with three categories of risk, viz. unacceptable risk, high risk and low or minimum risk. When AI is used for an unacceptable risk, it should be banned is what EU Commission has proposed.
Unacceptable risk is, if the AI system poses threat to safety, livelihood and rights, for example, autonomous weapons; or if the AI system manipulates human behavior to circumvent user’s free will, for example, toys that promote violence, cheer killing and encourage dangerous behavior of minors or if the AI system allows social scoring by Government, which may lead to discriminatory outcomes and the exclusion of certain groups [1].
High risk is when AI system is used for the following [1]:
1. Critical infrastructure (transport) -> eg. autonomous vehicle.
2. Educational training -> eg. scoring of exams.
3. Employee’s selection, management -> eg. resume screening system.
4. Safety components of products -> eg. robot-assisted surgery.
5. Essential private and public services -> eg. credit scoring denying citizens opportunity to obtain a loan.
6. Law enforcement -> eg. evaluation of the reliability of evidence.
AI system related to each of the above has a potential risk of harmful impact on health, safety and fundamental rights. The EU Commission proposes a regulatory framework with more stringent regulatory requirements for these high-risk AI.
Low or Minimum risk is when AI system represents minimal or no risk for citizen’s rights or safety, for example a web search engine or a product recommendation engine; or when AI system needs specific transparency obligation where users should be aware, they are interacting with a machine, for example, chatbot. For these low or minimum risk AI, the EU Commission has proposed a code of conduct to be drawn up voluntarily by individual providers of AI systems or organizations representing them [1].
Regulation Requirements for High-Risk AI
The EU Commission has proposed the following regulatory requirements for High-Risk AI [1]:
1. Adequate risk assessment and mitigation systems.
2. High quality of the datasets: to minimize risks and discriminatory outcomes.
3. Transparency: Clear and adequate information to the user
4. Logging of activity to ensure traceability of results.
5. High level of robustness, security and accuracy.
6. Appropriate human oversight measures to minimize risk.
7. Detailed documentation for authorities to assess its compliance.
The EU and US are starting to align on AI regulation. The National Institute for Standards and Technology (NIST) is in the process of developing an AI risk management framework. “The Framework aims to foster the development of innovative approaches to address characteristics of trustworthiness including accuracy, explainability and interpretability, reliability, privacy, robustness, safety, security (resilience), and mitigation of unintended and/or harmful bias, as well as of harmful uses”. Source: https://www.nist.gov/itl/ai-risk-management-framework
All the above seven regulatory requirements that EU Commission has laid out for high-risk AI, is aligned to the five principles of Trustworthy and Actionable AI:
Whenever we build an AI system, it should be inherently ethical and trustworthy, in other words, we should embed fairness, explainability, safety, security, robustness and accountability in the building blocks of AI system.
How to embed principles of Trustworthy and Actionable AI in Business Applications?
Ideally, we should embed the principles of Trustworthy and Actionable AI at every stage of the AI lifecycle, as shown below.
1. Data is the core of AI. An AI system needs to learn from data in order to be able to fulfil its functions. So, it is of utmost importance to ensure quality of training data in terms of fairness, consistency and privacy protection.
Fairness means unbiasedness or the absence of any prejudice or favoritism towards an individual or a group based on their intrinsic traits (gender, race, color, religion, disability, national origin, marital status, age, economic status and so on) in the context of decision-making. The principle of fairness plays a vital role in ensuring quality of training data.
a. Fairness of training data means the data should be representative of the population to which the model is going to be applied. More explicitly, the data should have a fair representation by outcome and sensitive-attribute sub-group. If there is bias in the data with regard to a sensitive-attribute sub-group, the AI model will learn the bias and give biased decision against that group, which is unacceptable.
So, to build a Trustworthy AI System, it is extremely important to detect and mitigate bias in the data before using it for training. There are several metrices and tests for detecting data bias such as Disparity Impact, Fisher’s Exact Test, Chi-square Test of Independence, Class Imbalance etc. which need to be appropriately applied to detect bias in the data.
Similarly, there exists various techniques of bias mitigation or balancing the data, such as over-sampling, under-sampling, SMOTE and its variants.
b. Consistency of training data: Another dimension of quality of training data is to ensure the data has no anomalies. Anomaly refers to the patterns in data that do not conform to expected behavior. Anomalous data are rare or abnormal events which differ significantly from the majority of the data. If there are anomalous data points in the training data, the model will learn the anomalous behavior and overfit the data; it will not be able to generalize and will make inaccurate predictions on new data. So, to build a Trustworthy AI System, it is extremely important to detect anomalies in the training data and remove it or treat it appropriately before using the data for training a machine learning model. There are various un-supervised, supervised and semi-supervised machine learning techniques to detect anomaly, popular among which are Density-based techniques like KNN, Isolation Forest, Clustering-based techniques such as K-Means, DBSCAN; One-class SVM, Neural Networks, autoencoders, hidden Markov models and could also be a combination of these techniques.
c. Privacy protection of training data: This is another important dimension of data quality from a Trustworthy Data & AI standpoint. We need to ensure the training data is appropriately masked or suppressed or encrypted as applicable so that confidentiality and integrity of personal and sensitive data is maintained.
Applying the principles of Trustworthy Data & AI to the training data, once we are able to confirm high quality of the training data, in terms of data free from bias, anomaly and one which has protection for personal or sensitive information, it can be used for training.
Once we ensure quality of training data, the next most important thing is to incorporate principles of Robustness, Fairness and Explainability in the training process of a machine learning workflow
2. Robustness refers to algorithmic stability. It means how effective your algorithm is while being tested on the new independent (but similar) dataset. In other words, the robust algorithm is the one, the testing error of which is close to the training error. Another aspect of robustness is “robustness to noise”, that describes the stability of the algorithm’s performance after adding some noise to the data. This kind of model stability is to ensure that the AI System is secure from external attacks. It is able to deal with adversarial attacks when someone feeds erroneous inputs and parameters that trick the AI model in making incorrect predictions or give away sensitive information.
3. Fairness of AI algorithm is of utmost importance when AI is used for decision making that has risk of affecting someone’s life, health, safety, freedom, education, career, profession and so on. Some real examples, where AI system has given unfair or biased decision against minority groups or historically disadvantageous groups are given below.
a) There is a commercial tool called COMPAS (which stands for Correctional Offender Management Profiling for Alternative Sanctions), powered by machine learning algorithm, which predicts a criminal defendant’s likelihood of re-offending (or recidivism). In 2016, a study which compared predictions from this tool with actuals, found that the tool was predicting higher risk of recidivism for Black defendants than they were and lesser risk of recidivism for white defendants than they actually were. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
b) There were some commercial face recognition online services provided by Microsoft, Face++, and IBM respectively, which showed much higher accuracy in determining the gender of light-skinned men and much lower accuracy in determining the gender of darker-skinned women. The reason being, the algorithm was trained on highly imbalance data which had disproportionately high representation of images of male and people with light-skin. These systems were later improved. https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212; https://blogs.microsoft.com/ai/gender-skin-tone-facial-recognition-improvement/
c) “Amazon’s experimental AI based hiring tool used to score job candidates was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. The reason being, the model was trained predominantly on male resumes submitted to the company over a 10-year period and so the model was giving more weightage to male-biased words in the resume and penalized resumes that included the word “women’s”, as in “women’s chess club captain””. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
It is imperative to ensure there is no bias in the model so that it gives fair decisions. Bias is detected and mitigated in the pre-processing, in-processing or post-processing stages of the model development. Pre-processing means bias in the training data, which has already been discussed under 1a. In-processing means bias in algorithm creeping in during training process and this happens due to certain assumptions made in training the model or due to applying some optimization constraints. There are a number of metrices to detect algorithmic bias, such as equalized odds, equal opportunity, demographic parity and so on. Algorithmic bias can be mitigated through various techniques, such as regularization of learning parameters, adjusting class-weights, balanced bagging, adversarial training procedure and few others. Post-processing means bias is mitigated after the training takes place. With post hoc explainability of model outcome it is possible to pinpoint the exact reason for bias and thereby it is critical for ethical AI.
4. Explainability of AI Model Decision is about making the outcome from “Black Box” machine learning models interpretable and explainable even to a naïve user, irrespective of the complexities of the underlying models. Machine learning or deep learning models are very complex non-linear models that do not explain, in a comprehensible manner, the rationale behind a prediction or how it has arrived at a decision. This lack of transparency in machine or deep learning model’s reasoning has been termed as the “Black Box” problem. One may wonder, if comprehensibility is a concern with machine learning model, then why not use traditional methods, like regression techniques or decision trees which produce models that are more amenable to human understanding? The answer to this question is that, today, as huge volumes of data and high-performance computing with the help of GPUs are available, machine learning models can better utilize the data to learn the highly complex, non-linear patterns and relationships in it and tune the parameters of the model optimally, which reduces the bias in the model and improves the model performance significantly over that of the traditional models. So, due to their high performance, machine learning models should be leveraged, but with explainability. To quote Professor Pedro Domingos, “when a new technology is as pervasive and game changing as machine learning, it is not wise to let it remain a black box. Opacity opens the door to error and misuse”. We need explainability to build trust in the AI system and also to guide downstream action in business applications.
Let us consider an example. If a customer is rejected for a home loan, as predicted by a neural network, the lender does not usually say it is because the sigmoid of a weighted, scaled combination of your loan-to-value ratio, your total value of delinquent accounts, your worst status (two or more payments in arrears) in last six months, your total outstanding balance in last three months, number of searches on loan you made in last six months, your occupation and your credit history length is equal to 0.57. Even though that may be how the model decided to reject the customer, lenders are typically required to break down the complex explanation and attempt to explain it to the customer in simple terms, using the most important original input variables — for example, stating that your loan-to-value ratio is greater than 80%, which is too high, your total outstanding balance on your credit card in last three months increased and you are self-employed. This simple, comprehensible explanation can be achieved by developing a surrogate model that approximates the trained complex machine learning model used for prediction. This surrogate model is called Explainable AI model, which is transparent and provides human-interpretable explanation for the prediction outcome given by the black-box model.
There are various approaches for Explainable AI: global versus local and model agnostic versus model specific explanation. By global explanation is meant, explanation in terms of the entire model. It is about analyzing the importance of features, in terms of their contributions to the output of the model. By local explanation is meant, explanation to understand the reasoning behind each individual prediction. Explanation at an instance level is usually more actionable than at model level, as it guides the explanation and action for the given transaction. For example, two individuals can have a high probability of defaulting on a loan but for completely different reasons.
By model agnostic is meant an explainer must be able to explain any model, i.e. treating the original model as a black box. This provides flexibility to explain any classifier. Whereas a model specific explanation considers the structure of the model to derive explanation for the prediction. The approach for generating explanation is not universal across all kinds of classifiers but uniquely developed for each classifier taking into consideration the algorithm’s inner working process.
Explanation at an instance level is usually more actionable than at model level, as it guides the explanation and action for the given transaction. Here’s a list of papers which present model-specific, local explanation:
i) “Explainable AI: A Hybrid Approach to Generate Human-Interpretable Explanation for Deep Learning Prediction”, published by ELSEVIER in Procedia Computer Science. Authors: Tanusree De, Prasenjit Giri, Ahmeduvesh Mevawala, Ramyasri Nemani, and Arati Deo.
ii) “Explainable NLP: A Novel Methodology to Generate Human-Interpretable Explanation for Semantic Text Similarity”, published by Springer in Advances in Signal Processing and Intelligent Recognition Systems. Authors: Tanusree De and Debapriya Mukherjee.
iii) “An Explainable AI powered Early Warning System to address Patient Readmission Risk”, published by IEEE in IEEE Xplore. Authors: Tanusree De, Ahmeduvesh Mevawala, and Ramyasri Nemani.
iv) “Comparative study of xAI layer-wise algorithms with a Robust Recommendation framework of Inductive Clustering for Polyp Segmentation and Classification”, published by IEEE in IEEE Xplore. Authors: Shiddalingeshwar Mallayya Javali, Raghavender Surya Upadhyayula, and Tanusree De.
What is needed to operationalize Trustworthy and Actionable AI?
Here, I present my point of view on a framework for Trustworthy AI Implementation & Governance.
As depicted above, five things that are key to build and operationalize Trustworthy Data & AI are Strategy & Roadmap, Toolkit, Process, People, Audit and Governance.
1. AI Strategy & Roadmap
The journey should begin with an assessment of the potential risk of using AI; and based on the assessment create an AI strategy and roadmap to collect the right set of data, check and improve the quality of the training data and embed the principles of Trustworthy Data & AI in the data and the model, like data bias mitigation, data privacy protection, model robustness, fairness, explainability and accountability. Now, to implement the strategy, we need a set of toolkits.
2. Toolkit
Basically, a set of assets and accelerators that enable you to build explainability and also to test any kind of bias in the data or in the algorithm and mitigate it. There are various open-source tools and proprietary tools that organizations are building, which are basically low-code/no-code platforms.
These toolkits serve as accelerators and enable researchers and developers to do more experimentations in a given timeframe; thereby empower them to build Trustworthy Data & AI solutions faster and with better performance. Along with tools, there should be a well-defined process or workflow that developers should follow to deliver technically sound and ethical AI systems.
3. Process:
It is very important to have a process in place and best practices around it.
(i) Research: The process should start with deep research on various approaches and methodologies that are already there, take ideas from them and innovate and design new approaches and methodologies that will yield better results.
(ii) Design: A human-centered approach should be taken to design the solution. Keeping human needs, behaviors and goals at the center will help to build AI solutions that are more trustable, reliable and free from any kind of bias.
(iii) Creation of ML Workflow: Once the solution is designed, the machine learning workflow should be created incorporating the elements of fairness, robustness, explainability as depicted in fig 1.
(iv) Development: Following the workflow, the solution development process should be executed using the toolkits and applying the best practices.
(v) Model Risk Assessment: Once the model is developed, the risk associated with the model, in terms of bias, accuracy, robustness, data privacy, explainability etc. should be assessed to finalize the model.
(vi) Documentation: In this whole process, a very important task is a detailed documentation of the design, workflow, the data and all the steps that have been followed, from data preprocessing to model training, validation, integration, model performance metrics, performance tracking mechanism, assumptions made at various stages of the model lifecycle, thresholds of various parameters of the model, limitations of the model and associated risks and so on. Documentation is very important reference for risk assessment and human oversight measures to maintain the system and address any issues whenever there is an indication of potential failure.
And at the center of everything that is needed for developing a Robust, Explainable and Ethical AI system, is people. It is humans who train and build an AI system. On the other hand, the end users of AI are also humans.
4. People:
To build an AI System, we need diverse set of people, i.e., technical, functional, data and domain experts; and we also need potential users of the AI system, or at least people who are representative of end users. It is extremely important to incorporate perspectives of the end users, their needs and requirements, and their feedback on usability in designing and developing the product to make it human-centered and efficient.
For example, to build an AI Powered Disease Detection System, you need to involve doctors or people from Life Sciences, who can provide specific domain knowledge of the disease, symptoms, medical parameters etc. that needs to be incorporated in training the system. A doctor will be able to do the right benchmarking and come up with domain-specific metrics to measure accuracy and precision of the system during training as well as in production.
Similarly for building AI System for legal domain, lawyers should be consulted; or for an AI Based Exam Review, human examiners should be involved and so on. It is all about training the system with specific knowledge a human expert would apply to accomplish a task.
By similar logic Functional experts in HR, IT, Marketing, Risk & Compliance should be involved in building AI systems that address business problems in these functional domains.
There are many AI applications today that are trained purely on huge volumes of text data, where linguistic experts and translators can play a big role in providing valuable inputs for data pre-processing, model fine-tuning and validation.
The team building the AI system should have diversity in terms of gender, race, marital status and so on, so that perspectives of different groups of people can be incorporated in the design and development of the AI system which will reduce bias and make the AI system more inclusive.
The team should also have people playing the role of AI Ethicists who can provide guidance on ethical use of AI, help with regulatory compliance, ensure AI system gives ethical decisions and build accountability frameworks to address any unintended risk posed by the AI system.
Once an AI system is built with the right set of people, process and toolkit, the next most important task is audit.
5. Audit
Every organization that are building AI system, should have an AI audit practice. The role of auditors is to review and inspect the standards and procedures followed at every stage of the AI lifecycle, and compliance with laws, policies and organization’s strategies to ensure the system is technically robust and safe, secure, fair, explainable, protects privacy, and overall reliable and responsible.
To be more specific, this means inspecting for:
a. Detailed documentation: of all stages of the AI lifecycle, from detailed overview of the data, to data cleansing, data filtering, data pre-processing applied, to modeling processes followed, model information and artifacts involved in the production of an AI model, and also an analysis of the outcomes of the model.
b. Methods used for designing and developing the algorithmic system: what algorithm was used, how the algorithm was trained, what assumptions were made, what kind of input data was used, whether the input data had sensitive variables and whether there was any bias in the data that was mitigated or not and how.
c. Methods used to test and validate the algorithmic system: information about the data used to test and validate. Information about Train-Test discrepancy and how well the model generalized on Test or new data. Whether there was over-fitting, under-fitting or bias in terms of sensitive attributes.
d. Explainability of outcomes: Is there explainability for the outcome of the black-box AI model? How intuitive, logical and comprehensible the explanations are? How well the explanations can be used for taking decisions? This is critical for high-risk use cases of AI, like medical image analysis or disease detection where explainability is a must to trust the model outcome and take appropriate decisions.
Auditors should also review the process in place for data governance and post market monitoring.
e. Data governance: Inspect the standards, processes and technologies used to manage and protect the company’s data assets. The best practices followed to ensure regular cleansing, updating and purging the data to maintain high quality of the data. Conformity with data privacy laws and regulations.
f. Post market monitoring: Auditors should check, there is a robust process in place for tracking and reporting serious incidence and of malfunctioning, to ensure AI system is not used irresponsibly when there is a problem in it that needs to be rectified.
And, to ensure overall conformity and compliance, including audit, there has to be governance.
6. Governance
AI Governance is a system for overseeing the way organization implements each element of the Trustworthy AI framework as described above and how they are ensuring accountability. Basically, AI governance needs to encompass the entire life cycle of the AI system. There should be governance for:
i) Ethical review of AI usage from the perspective of end-user.
ii) Ensuring alignment with regulatory requirements and organization’s strategy, culture and vision.
iii) Reviewing AI risk and approaches for model design & development, model integration, deployment & maintenance.
iv) Ensuring appropriate standards for quality of training data, data preparation methods, model development processes, testing model performance and robustness, bias checks, explainability of model outcome and model maintenance.
Another very important aspect of AI Governance is that AI Governance should not take place in isolation, rather it should be part of the overall governance system of an organization. For effective AI Governance, it should be connected with corporate, IT and data governance. For example, how an organization would leverage AI to advance public interest is decided by Corporate Governance and alignment with this objective is an important aspect of AI governance. Similarly, when AI algorithms are integrated into an organization’s information system, the organization’s AI governance and IT governance should be aligned. Again, as AI systems are built on data, and on the other hand, AI is also leveraged to enrich the data, certain aspects of Data governance are vital to AI governance and vice versa and the two should work in tandem.
Conclusion
Embedding transparency, explainability and ethical standards and values in Data and AI is the need of the hour to mitigate risk, build trust and make AI actionable. All these will eventually increase adoption of AI in transforming society and business both.
References
[1] European Commission, Proposal for REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL, LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS (2021), EUR-Lex,
https://eur-lex.europa.eu/legalcontent/EN/TXT/?uri=CELEX%3A52021PC0206
[2] E. Ntoutsia et al., Bias in Data-driven AI Systems — An Introductory Survey (2020), arXiv, https://arxiv.org/abs/2001.09762
[3] R. Guidotti et al., A Survey of Methods for Explaining Black Box Models (2018), arXiv, https://arxiv.org/abs/1802.01933
[4] Nicole Turner Lee et al., Algorithmic Bias Detection and Mitigation: Best Practices and Policies to reduce Consumer Harm (2019), BROOKINGS, https://www.brookings.edu/research/algorithmic-bias-detection-and-mitigation-best-practices-and-policies-to-reduce-consumer-harms/
[5] David J Hand and Shakeel Khan, Validating and Verifying AI Systems (2020), Patterns, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7660449/#
How to Minimize Risks of AI and Accelerate its Adoption
Artificial Intelligence has come a long way! What was once a fantasy of the imagination of some famous science fiction writers, is all-pervasive today. AI has become a part of our lives; it is used everywhere, and it has made lot of things convenient for us. In many industries, AI powered cognitive automation has helped in automating complex business processes which in turn has largely improved process efficiency. AI systems are also being incorporated into a wide variety of decision-making processes, both in industry and policy fields. In healthcare, AI has brought in some incredible breakthroughs. However, in many cases, there are also potential risks associated with the usage of AI and there have been real examples where AI has been misused, either knowingly or unknowingly. There can be possibility of AI having ethical issues, or inadequate security, or not being safe or not preserving privacy of personal or sensitive data. Moreover, an AI system is a black box. It gives us a decision, but it does not tell us about the rationale behind the decision, how it has arrived at the decision. Due to its lack of transparency, it becomes difficult to trust the decisioning of AI systems. All in all, though people are aware of the incredible power and benefits of AI, they are equally concerned about the potential risks associated with it. This has naturally called for regulation of AI and the need for Trustworthy and Actionable AI.
What is AI Regulation?
The regulation of artificial intelligence is the development of public sector policies and laws to integrate ethical standards into the design and implementation of AI-enabled technologies.
Now, how to define ethical standards for AI? What should be the premise? The first and foremost principle is to ensure responsible use of AI. Even before an AI system is developed for a use case, one should analyze the potential risks and benefits associated with the use of AI for that particular use case. Depending on the extent of potential risk versus the benefit it yields, one should take decisions to integrate some stringent requirements in the design of the system to make it safe, secure, ethical, transparent and reliable. This is exactly what the European Union Commission has mentioned in its proposal for Regulation of AI published on 21 April 2021. Source: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206
The EU Commission has mentioned that their key objective for regulation of AI is to ensure safety and adherence to fundamental rights and Union values. The Commission has proposed a risk-based approach for regulation of AI, wherein, they have come up with three categories of risk, viz. unacceptable risk, high risk and low or minimum risk. When AI is used for an unacceptable risk, it should be banned is what EU Commission has proposed.
Unacceptable risk is, if the AI system poses threat to safety, livelihood and rights, for example, autonomous weapons; or if the AI system manipulates human behavior to circumvent user’s free will, for example, toys that promote violence, cheer killing and encourage dangerous behavior of minors or if the AI system allows social scoring by Government, which may lead to discriminatory outcomes and the exclusion of certain groups [1].
High risk is when AI system is used for the following [1]:
1. Critical infrastructure (transport) -> eg. autonomous vehicle.
2. Educational training -> eg. scoring of exams.
3. Employee’s selection, management -> eg. resume screening system.
4. Safety components of products -> eg. robot-assisted surgery.
5. Essential private and public services -> eg. credit scoring denying citizens opportunity to obtain a loan.
6. Law enforcement -> eg. evaluation of the reliability of evidence.
AI system related to each of the above has a potential risk of harmful impact on health, safety and fundamental rights. The EU Commission proposes a regulatory framework with more stringent regulatory requirements for these high-risk AI.
Low or Minimum risk is when AI system represents minimal or no risk for citizen’s rights or safety, for example a web search engine or a product recommendation engine; or when AI system needs specific transparency obligation where users should be aware, they are interacting with a machine, for example, chatbot. For these low or minimum risk AI, the EU Commission has proposed a code of conduct to be drawn up voluntarily by individual providers of AI systems or organizations representing them [1].
Regulation Requirements for High-Risk AI
The EU Commission has proposed the following regulatory requirements for High-Risk AI [1]:
1. Adequate risk assessment and mitigation systems.
2. High quality of the datasets: to minimize risks and discriminatory outcomes.
3. Transparency: Clear and adequate information to the user
4. Logging of activity to ensure traceability of results.
5. High level of robustness, security and accuracy.
6. Appropriate human oversight measures to minimize risk.
7. Detailed documentation for authorities to assess its compliance.
The EU and US are starting to align on AI regulation. The National Institute for Standards and Technology (NIST) is in the process of developing an AI risk management framework. “The Framework aims to foster the development of innovative approaches to address characteristics of trustworthiness including accuracy, explainability and interpretability, reliability, privacy, robustness, safety, security (resilience), and mitigation of unintended and/or harmful bias, as well as of harmful uses”. Source: https://www.nist.gov/itl/ai-risk-management-framework
All the above seven regulatory requirements that EU Commission has laid out for high-risk AI, is aligned to the five principles of Trustworthy and Actionable AI:
Whenever we build an AI system, it should be inherently ethical and trustworthy, in other words, we should embed fairness, explainability, safety, security, robustness and accountability in the building blocks of AI system.
How to embed principles of Trustworthy and Actionable AI in Business Applications?
Ideally, we should embed the principles of Trustworthy and Actionable AI at every stage of the AI lifecycle, as shown below.
1. Data is the core of AI. An AI system needs to learn from data in order to be able to fulfil its functions. So, it is of utmost importance to ensure quality of training data in terms of fairness, consistency and privacy protection.
Fairness means unbiasedness or the absence of any prejudice or favoritism towards an individual or a group based on their intrinsic traits (gender, race, color, religion, disability, national origin, marital status, age, economic status and so on) in the context of decision-making. The principle of fairness plays a vital role in ensuring quality of training data.
a. Fairness of training data means the data should be representative of the population to which the model is going to be applied. More explicitly, the data should have a fair representation by outcome and sensitive-attribute sub-group. If there is bias in the data with regard to a sensitive-attribute sub-group, the AI model will learn the bias and give biased decision against that group, which is unacceptable.
So, to build a Trustworthy AI System, it is extremely important to detect and mitigate bias in the data before using it for training. There are several metrices and tests for detecting data bias such as Disparity Impact, Fisher’s Exact Test, Chi-square Test of Independence, Class Imbalance etc. which need to be appropriately applied to detect bias in the data.
Similarly, there exists various techniques of bias mitigation or balancing the data, such as over-sampling, under-sampling, SMOTE and its variants.
b. Consistency of training data: Another dimension of quality of training data is to ensure the data has no anomalies. Anomaly refers to the patterns in data that do not conform to expected behavior. Anomalous data are rare or abnormal events which differ significantly from the majority of the data. If there are anomalous data points in the training data, the model will learn the anomalous behavior and overfit the data; it will not be able to generalize and will make inaccurate predictions on new data. So, to build a Trustworthy AI System, it is extremely important to detect anomalies in the training data and remove it or treat it appropriately before using the data for training a machine learning model. There are various un-supervised, supervised and semi-supervised machine learning techniques to detect anomaly, popular among which are Density-based techniques like KNN, Isolation Forest, Clustering-based techniques such as K-Means, DBSCAN; One-class SVM, Neural Networks, autoencoders, hidden Markov models and could also be a combination of these techniques.
c. Privacy protection of training data: This is another important dimension of data quality from a Trustworthy Data & AI standpoint. We need to ensure the training data is appropriately masked or suppressed or encrypted as applicable so that confidentiality and integrity of personal and sensitive data is maintained.
Applying the principles of Trustworthy Data & AI to the training data, once we are able to confirm high quality of the training data, in terms of data free from bias, anomaly and one which has protection for personal or sensitive information, it can be used for training.
Once we ensure quality of training data, the next most important thing is to incorporate principles of Robustness, Fairness and Explainability in the training process of a machine learning workflow
2. Robustness refers to algorithmic stability. It means how effective your algorithm is while being tested on the new independent (but similar) dataset. In other words, the robust algorithm is the one, the testing error of which is close to the training error. Another aspect of robustness is “robustness to noise”, that describes the stability of the algorithm’s performance after adding some noise to the data. This kind of model stability is to ensure that the AI System is secure from external attacks. It is able to deal with adversarial attacks when someone feeds erroneous inputs and parameters that trick the AI model in making incorrect predictions or give away sensitive information.
3. Fairness of AI algorithm is of utmost importance when AI is used for decision making that has risk of affecting someone’s life, health, safety, freedom, education, career, profession and so on. Some real examples, where AI system has given unfair or biased decision against minority groups or historically disadvantageous groups are given below.
a) There is a commercial tool called COMPAS (which stands for Correctional Offender Management Profiling for Alternative Sanctions), powered by machine learning algorithm, which predicts a criminal defendant’s likelihood of re-offending (or recidivism). In 2016, a study which compared predictions from this tool with actuals, found that the tool was predicting higher risk of recidivism for Black defendants than they were and lesser risk of recidivism for white defendants than they actually were. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
b) There were some commercial face recognition online services provided by Microsoft, Face++, and IBM respectively, which showed much higher accuracy in determining the gender of light-skinned men and much lower accuracy in determining the gender of darker-skinned women. The reason being, the algorithm was trained on highly imbalance data which had disproportionately high representation of images of male and people with light-skin. These systems were later improved. https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212; https://blogs.microsoft.com/ai/gender-skin-tone-facial-recognition-improvement/
c) “Amazon’s experimental AI based hiring tool used to score job candidates was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. The reason being, the model was trained predominantly on male resumes submitted to the company over a 10-year period and so the model was giving more weightage to male-biased words in the resume and penalized resumes that included the word “women’s”, as in “women’s chess club captain””. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
It is imperative to ensure there is no bias in the model so that it gives fair decisions. Bias is detected and mitigated in the pre-processing, in-processing or post-processing stages of the model development. Pre-processing means bias in the training data, which has already been discussed under 1a. In-processing means bias in algorithm creeping in during training process and this happens due to certain assumptions made in training the model or due to applying some optimization constraints. There are a number of metrices to detect algorithmic bias, such as equalized odds, equal opportunity, demographic parity and so on. Algorithmic bias can be mitigated through various techniques, such as regularization of learning parameters, adjusting class-weights, balanced bagging, adversarial training procedure and few others. Post-processing means bias is mitigated after the training takes place. With post hoc explainability of model outcome it is possible to pinpoint the exact reason for bias and thereby it is critical for ethical AI.
4. Explainability of AI Model Decision is about making the outcome from “Black Box” machine learning models interpretable and explainable even to a naïve user, irrespective of the complexities of the underlying models. Machine learning or deep learning models are very complex non-linear models that do not explain, in a comprehensible manner, the rationale behind a prediction or how it has arrived at a decision. This lack of transparency in machine or deep learning model’s reasoning has been termed as the “Black Box” problem. One may wonder, if comprehensibility is a concern with machine learning model, then why not use traditional methods, like regression techniques or decision trees which produce models that are more amenable to human understanding? The answer to this question is that, today, as huge volumes of data and high-performance computing with the help of GPUs are available, machine learning models can better utilize the data to learn the highly complex, non-linear patterns and relationships in it and tune the parameters of the model optimally, which reduces the bias in the model and improves the model performance significantly over that of the traditional models. So, due to their high performance, machine learning models should be leveraged, but with explainability. To quote Professor Pedro Domingos, “when a new technology is as pervasive and game changing as machine learning, it is not wise to let it remain a black box. Opacity opens the door to error and misuse”. We need explainability to build trust in the AI system and also to guide downstream action in business applications.
Let us consider an example. If a customer is rejected for a home loan, as predicted by a neural network, the lender does not usually say it is because the sigmoid of a weighted, scaled combination of your loan-to-value ratio, your total value of delinquent accounts, your worst status (two or more payments in arrears) in last six months, your total outstanding balance in last three months, number of searches on loan you made in last six months, your occupation and your credit history length is equal to 0.57. Even though that may be how the model decided to reject the customer, lenders are typically required to break down the complex explanation and attempt to explain it to the customer in simple terms, using the most important original input variables — for example, stating that your loan-to-value ratio is greater than 80%, which is too high, your total outstanding balance on your credit card in last three months increased and you are self-employed. This simple, comprehensible explanation can be achieved by developing a surrogate model that approximates the trained complex machine learning model used for prediction. This surrogate model is called Explainable AI model, which is transparent and provides human-interpretable explanation for the prediction outcome given by the black-box model.
There are various approaches for Explainable AI: global versus local and model agnostic versus model specific explanation. By global explanation is meant, explanation in terms of the entire model. It is about analyzing the importance of features, in terms of their contributions to the output of the model. By local explanation is meant, explanation to understand the reasoning behind each individual prediction. Explanation at an instance level is usually more actionable than at model level, as it guides the explanation and action for the given transaction. For example, two individuals can have a high probability of defaulting on a loan but for completely different reasons.
By model agnostic is meant an explainer must be able to explain any model, i.e. treating the original model as a black box. This provides flexibility to explain any classifier. Whereas a model specific explanation considers the structure of the model to derive explanation for the prediction. The approach for generating explanation is not universal across all kinds of classifiers but uniquely developed for each classifier taking into consideration the algorithm’s inner working process.
Explanation at an instance level is usually more actionable than at model level, as it guides the explanation and action for the given transaction. Here’s a list of papers which present model-specific, local explanation:
i) “Explainable AI: A Hybrid Approach to Generate Human-Interpretable Explanation for Deep Learning Prediction”, published by ELSEVIER in Procedia Computer Science. Authors: Tanusree De, Prasenjit Giri, Ahmeduvesh Mevawala, Ramyasri Nemani, and Arati Deo.
ii) “Explainable NLP: A Novel Methodology to Generate Human-Interpretable Explanation for Semantic Text Similarity”, published by Springer in Advances in Signal Processing and Intelligent Recognition Systems. Authors: Tanusree De and Debapriya Mukherjee.
iii) “An Explainable AI powered Early Warning System to address Patient Readmission Risk”, published by IEEE in IEEE Xplore. Authors: Tanusree De, Ahmeduvesh Mevawala, and Ramyasri Nemani.
iv) “Comparative study of xAI layer-wise algorithms with a Robust Recommendation framework of Inductive Clustering for Polyp Segmentation and Classification”, published by IEEE in IEEE Xplore. Authors: Shiddalingeshwar Mallayya Javali, Raghavender Surya Upadhyayula, and Tanusree De.
What is needed to operationalize Trustworthy and Actionable AI?
Here, I present my point of view on a framework for Trustworthy AI Implementation & Governance.
As depicted above, five things that are key to build and operationalize Trustworthy Data & AI are Strategy & Roadmap, Toolkit, Process, People, Audit and Governance.
1. AI Strategy & Roadmap
The journey should begin with an assessment of the potential risk of using AI; and based on the assessment create an AI strategy and roadmap to collect the right set of data, check and improve the quality of the training data and embed the principles of Trustworthy Data & AI in the data and the model, like data bias mitigation, data privacy protection, model robustness, fairness, explainability and accountability. Now, to implement the strategy, we need a set of toolkits.
2. Toolkit
Basically, a set of assets and accelerators that enable you to build explainability and also to test any kind of bias in the data or in the algorithm and mitigate it. There are various open-source tools and proprietary tools that organizations are building, which are basically low-code/no-code platforms.
These toolkits serve as accelerators and enable researchers and developers to do more experimentations in a given timeframe; thereby empower them to build Trustworthy Data & AI solutions faster and with better performance. Along with tools, there should be a well-defined process or workflow that developers should follow to deliver technically sound and ethical AI systems.
3. Process:
It is very important to have a process in place and best practices around it.
(i) Research: The process should start with deep research on various approaches and methodologies that are already there, take ideas from them and innovate and design new approaches and methodologies that will yield better results.
(ii) Design: A human-centered approach should be taken to design the solution. Keeping human needs, behaviors and goals at the center will help to build AI solutions that are more trustable, reliable and free from any kind of bias.
(iii) Creation of ML Workflow: Once the solution is designed, the machine learning workflow should be created incorporating the elements of fairness, robustness, explainability as depicted in fig 1.
(iv) Development: Following the workflow, the solution development process should be executed using the toolkits and applying the best practices.
(v) Model Risk Assessment: Once the model is developed, the risk associated with the model, in terms of bias, accuracy, robustness, data privacy, explainability etc. should be assessed to finalize the model.
(vi) Documentation: In this whole process, a very important task is a detailed documentation of the design, workflow, the data and all the steps that have been followed, from data preprocessing to model training, validation, integration, model performance metrics, performance tracking mechanism, assumptions made at various stages of the model lifecycle, thresholds of various parameters of the model, limitations of the model and associated risks and so on. Documentation is very important reference for risk assessment and human oversight measures to maintain the system and address any issues whenever there is an indication of potential failure.
And at the center of everything that is needed for developing a Robust, Explainable and Ethical AI system, is people. It is humans who train and build an AI system. On the other hand, the end users of AI are also humans.
4. People:
To build an AI System, we need diverse set of people, i.e., technical, functional, data and domain experts; and we also need potential users of the AI system, or at least people who are representative of end users. It is extremely important to incorporate perspectives of the end users, their needs and requirements, and their feedback on usability in designing and developing the product to make it human-centered and efficient.
For example, to build an AI Powered Disease Detection System, you need to involve doctors or people from Life Sciences, who can provide specific domain knowledge of the disease, symptoms, medical parameters etc. that needs to be incorporated in training the system. A doctor will be able to do the right benchmarking and come up with domain-specific metrics to measure accuracy and precision of the system during training as well as in production.
Similarly for building AI System for legal domain, lawyers should be consulted; or for an AI Based Exam Review, human examiners should be involved and so on. It is all about training the system with specific knowledge a human expert would apply to accomplish a task.
By similar logic Functional experts in HR, IT, Marketing, Risk & Compliance should be involved in building AI systems that address business problems in these functional domains.
There are many AI applications today that are trained purely on huge volumes of text data, where linguistic experts and translators can play a big role in providing valuable inputs for data pre-processing, model fine-tuning and validation.
The team building the AI system should have diversity in terms of gender, race, marital status and so on, so that perspectives of different groups of people can be incorporated in the design and development of the AI system which will reduce bias and make the AI system more inclusive.
The team should also have people playing the role of AI Ethicists who can provide guidance on ethical use of AI, help with regulatory compliance, ensure AI system gives ethical decisions and build accountability frameworks to address any unintended risk posed by the AI system.
Once an AI system is built with the right set of people, process and toolkit, the next most important task is audit.
5. Audit
Every organization that are building AI system, should have an AI audit practice. The role of auditors is to review and inspect the standards and procedures followed at every stage of the AI lifecycle, and compliance with laws, policies and organization’s strategies to ensure the system is technically robust and safe, secure, fair, explainable, protects privacy, and overall reliable and responsible.
To be more specific, this means inspecting for:
a. Detailed documentation: of all stages of the AI lifecycle, from detailed overview of the data, to data cleansing, data filtering, data pre-processing applied, to modeling processes followed, model information and artifacts involved in the production of an AI model, and also an analysis of the outcomes of the model.
b. Methods used for designing and developing the algorithmic system: what algorithm was used, how the algorithm was trained, what assumptions were made, what kind of input data was used, whether the input data had sensitive variables and whether there was any bias in the data that was mitigated or not and how.
c. Methods used to test and validate the algorithmic system: information about the data used to test and validate. Information about Train-Test discrepancy and how well the model generalized on Test or new data. Whether there was over-fitting, under-fitting or bias in terms of sensitive attributes.
d. Explainability of outcomes: Is there explainability for the outcome of the black-box AI model? How intuitive, logical and comprehensible the explanations are? How well the explanations can be used for taking decisions? This is critical for high-risk use cases of AI, like medical image analysis or disease detection where explainability is a must to trust the model outcome and take appropriate decisions.
Auditors should also review the process in place for data governance and post market monitoring.
e. Data governance: Inspect the standards, processes and technologies used to manage and protect the company’s data assets. The best practices followed to ensure regular cleansing, updating and purging the data to maintain high quality of the data. Conformity with data privacy laws and regulations.
f. Post market monitoring: Auditors should check, there is a robust process in place for tracking and reporting serious incidence and of malfunctioning, to ensure AI system is not used irresponsibly when there is a problem in it that needs to be rectified.
And, to ensure overall conformity and compliance, including audit, there has to be governance.
6. Governance
AI Governance is a system for overseeing the way organization implements each element of the Trustworthy AI framework as described above and how they are ensuring accountability. Basically, AI governance needs to encompass the entire life cycle of the AI system. There should be governance for:
i) Ethical review of AI usage from the perspective of end-user.
ii) Ensuring alignment with regulatory requirements and organization’s strategy, culture and vision.
iii) Reviewing AI risk and approaches for model design & development, model integration, deployment & maintenance.
iv) Ensuring appropriate standards for quality of training data, data preparation methods, model development processes, testing model performance and robustness, bias checks, explainability of model outcome and model maintenance.
Another very important aspect of AI Governance is that AI Governance should not take place in isolation, rather it should be part of the overall governance system of an organization. For effective AI Governance, it should be connected with corporate, IT and data governance. For example, how an organization would leverage AI to advance public interest is decided by Corporate Governance and alignment with this objective is an important aspect of AI governance. Similarly, when AI algorithms are integrated into an organization’s information system, the organization’s AI governance and IT governance should be aligned. Again, as AI systems are built on data, and on the other hand, AI is also leveraged to enrich the data, certain aspects of Data governance are vital to AI governance and vice versa and the two should work in tandem.
Conclusion
Embedding transparency, explainability and ethical standards and values in Data and AI is the need of the hour to mitigate risk, build trust and make AI actionable. All these will eventually increase adoption of AI in transforming society and business both.
References
[1] European Commission, Proposal for REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL, LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS (2021), EUR-Lex,
https://eur-lex.europa.eu/legalcontent/EN/TXT/?uri=CELEX%3A52021PC0206
[2] E. Ntoutsia et al., Bias in Data-driven AI Systems — An Introductory Survey (2020), arXiv, https://arxiv.org/abs/2001.09762
[3] R. Guidotti et al., A Survey of Methods for Explaining Black Box Models (2018), arXiv, https://arxiv.org/abs/1802.01933
[4] Nicole Turner Lee et al., Algorithmic Bias Detection and Mitigation: Best Practices and Policies to reduce Consumer Harm (2019), BROOKINGS, https://www.brookings.edu/research/algorithmic-bias-detection-and-mitigation-best-practices-and-policies-to-reduce-consumer-harms/
[5] David J Hand and Shakeel Khan, Validating and Verifying AI Systems (2020), Patterns, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7660449/#
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.