AI Risk Management

AI risk management has become a best practice among enterprises in order to measure and mitigate risk to meet organizational requirements. To fully understand this, we must first define AI risk.

AI risk can be expressed as a probabilistic expectation:

AI risk = (likelihood of an AI model error or exploit) x (its potential effect)

This definition is instructive because it provides a recipe for action. AI errors and vulnerabilities are many, occur frequently, and vary across model tasks and modalities. Examples include arbitrary code execution, data poisoning, prompt injection, model extraction, hallucinations, data drift, unexpected behavior, bias predictions, and toxic output.

The effects of a model error depend largely on the use case. They can be financial, legal, or reputational. More importantly, they can have devastating consequences for users, such as leaked personal information, denied healthcare coverage, or declined loan approval decisions.

AI models are not just a function of the underlying “code”; they are fundamentally also a function of the data. In traditional software functions, one defines rigorous abstractions rooted in procedural software logic in order to transform an input into a given output. Yet the very advantage of AI is that it can “learn” the logic from a corpus of data, rather than the user explicitly defining it. Because the user cannot directly observe this logic, this makes the burden of testing much harder on the user. Simply testing on a few datapoints and edge cases is no longer enough to assert “correctness”. The user must now rigorously test the performance of the model with evaluation datasets, and assert that the model can also make correct predictions on a potentially infinite set of possible data distributions.

A corollary is that the metrics used to evaluate the performance of a AI model are different from those used to test software. This is because AI models are approximate learners on noisy data; in most practical settings, it is impossible for models to be 100% correct on the entire data distribution. For example, accuracy is a common metric for evaluating the performance of a classification model, whereas software is often tested for functionality and usability. In addition, simply testing top-level metrics on a dataset is not enough. There are many other dimensions in which AI models can fail.

AI models also come in a wide array of modalities/tasks, ranging from binary classification on tabular data to language-based responses from generative AI. Each modality and tasks come with their own unique failure modes and testing challenges in the dimensions mentioned above. Moreover, since AI involves automating critical decisions, there are test that we would like to run that pertain to other aspects of integrity of the system. This includes testing against bias and fairness, as well as abuses to sensitive information that may be encoded in the model and data.

AI is presents unique challenges that necessitate a new paradigm to mitigate risk.

AI risk management refers to a suite of tools and practices deployed to proactively protect organizations and end users from the distinctive risks of AI. It involves measuring the risks, and implementing solutions to minimize them. As per the definition of AI risk, it can be minimized by reducing the likelihood of a failure or by reducing the impact severity. Although it may resemble some software engineering best practices, AI risk management is entirely different.

According to NIST, as defined in their AI Risk Management Framework: