Explainability

The three pillars of explainability

This diagram presents a framework for defining the « right » level of explainability based on technical, legal and economic considerations.

The approach involves three logical steps:

• Define the main contextual factors, such as who is the recipient of the explanation, the operational context, the level of harm that the system could cause, and the legal/regulatory framework. This step will help characterize the operational and legal needs for explanation, and the corresponding social benefits.
• Examine the technical tools available, including post-hoc approaches (input perturbation, saliency maps…) and hybrid AI approaches.
• As a function of the first two steps, choose the right levels of global and local explanation outputs, taking into the account the costs involved.

Explainibility requirements

There are operational reasons for explainability, including the need to make algorithms more robust.

There are also ethical and legal reasons for explainability, including protection of individual rights.

It’s important to keep these two sets of reasons separate.

From transparency to accountability

Explainability takes raw information and makes it understandable to humans.

Explainability is a value-added component of transparency.

Both explainability and transparency enable other important functions, such as traceability, auditability, and accountability.

Global explanations

Global explanations give an overview of the whole algorithm.

Local explanations

Local explanations provide precise information on a given algorithmic decision.

Explainability requires information about the learning algorithm and the trained algorithm

Explanations may be needed regarding the learning algorithm, including information on the training data used.

Explanations relating to the learned algorithm will generally focus on a particular algorithmic decision (local explanations).

Technical approaches

Post-hoc approaches try to imitate the functioning of the black-box model.

Hybrid approaches try to put explainability in the model itself.

Examples of a hybrid approach

A hybrid approach may teach the algorithm to look at the right area of the image, based on domain expertise, in this case radiology.

Three approaches to explainability by design

Hybrid AI approaches can focus on the inputs, on the network itself, or on the outputs.

Benefits and costs of explaination

Finding the right level of explainability requires a consideration of the benefits and costs associated with an explanation in a given situation. For an explanation to be socially-useful, total benefits should exceed costs.