On the Variability of Concept Activation Vectors

One of the most pressing challenges in artificial intelligence is to make models more transparent to their users. Recently, explainable artificial intelligence has come up with numerous methods to tackle this challenge. A promising avenue is to use concept-based explanations, that is, high-level concepts instead of plain feature importance scores. Among this class of methods, Concept Activation Vectors (CAVs, Kim et al., 2018) stand out as one of the main protagonists. One interesting aspect of CAVs is that their computation requires sampling random examples from the train set. Therefore, the actual vectors obtained may vary depending on the randomness of this sampling. In this paper, we propose a fine-grained theoretical analysis of CAV construction in order to quantify their variability. Our results, confirmed by experiments on several real-life datasets of four different modalities, point to an universal result: the variance of CAVs declines roughly as $1/N$, where $N$ is the number of random examples. Based on this, we give practical recommendations for a resource-efficient application of the method.

We study a problem with TCAV, a method that explains AI models using human ideas like "striped," "female," or "positive words.'" TCAV finds a direction inside a model that represents one of these ideas. This direction is called a Concept Activation Vector, or CAV. The problem is that CAVs depend on random comparison examples. If we run TCAV twice with different random examples, we can get different results. This means people may draw different conclusions from the same model. We show that CAVs become more stable when more random examples are used. In simple terms, if we increase the number of random examples, the CAV becomes less noisy. But TCAV scores can still stay unstable because they depend on whether effects are counted as positive or negative, and small changes can flip borderline cases. Therefore, we recommend using many random examples for stable CAV directions, and several independent runs with averaging for more stable TCAV scores.