We begin in ‘Understanding the inner knowledge of AI models’ by demonstrating how we can understand the internal knowledge of AI models by searching and describing the semantic space. ‘Auditing concept alignment with expected reasoning’ describes how these functionalities provide the basis for effectively auditing the alignment of the reasoning of the model with respect to human expectation. We demonstrate how to spot flaws in medical models and improve robustness and safety in ‘Towards robust and safe medical models’. Last, in ‘Evaluating human-interpretability of model components’, computable measures for the human-interpretability of model components are introduced, enabling one to rate and improve interpretability at scale.
The different sets of experiments reported in this paper were conducted on a variety of models, including convolutional neural networks with the ResNet24 and VGG25 architectures as well as different vision transformers (ViTs)26. Additionally, we used two large vision datasets, namely ImageNet27 and ISIC 201928, along with several foundation models, including Mobile-CLIP29, DINOv2 (ref. 30) and WhyLesionCLIP31. Further details about the experimental setting can be found in Supplementary Note B. More analyses are reported in Supplementary Notes C to G.
Understanding the inner knowledge of AI models
In the following, SemanticLens is used to systematically analyse the knowledge (learned concepts) encoded by neurons of ResNet50v2 trained on the ImageNet classification task27. The individual neurons are embedded as vectors ϑ into the multimodal and semantically organized space of the Mobile-CLIP foundation model29, as illustrated in Fig. 1 and described in ‘Methods’.
Search capability
The first capability of SemanticLens that we demonstrate is its search capability, which allows one to quickly browse through all neurons of the ResNet50v2 model and identify encoded concepts that a user is interested in, such as potential biases (for example, gender or racial), data artefacts (for example, watermarks) or specific knowledge. The search is based on a (cosine) similarity comparison between a probing vector ϑprobe, representing the concept we are looking for (for example, the concept ‘person’), and the set of embedded neurons ϑ’s of the ResNet model. The shared vision/text embedding space of Mobile-CLIP enables one to query concepts using either textual descriptions (the word ‘person’) or visual examples (an image of a person). In the following experiments, we focus on text-based probing. Details of the construction of probing vectors and the retrieval process can be found in ‘Methods’.
As illustrated in Fig. 2a, the neurons in the ResNet50v2 model that encode for person-related concepts can be identified. Two embedded neurons that most closely resemble the addtext-based probing vector represent different, non-obvious and potentially discriminative aspects of a person, such as a hijab (neuron 1216) and dark skin (neuron 1454). This is, in principle, a valid strategy for representing different object subgroups sharing certain visual features by specialized neurons. However, if these sensitive attribute-encoding neurons are used for other purposes, for example, the dark skin-related neuron is used in the classification of ‘steel drum’ (Fig. 3b), then this may hint at potential fairness issues.
a, Using search-engine-like queries, one can probe for knowledge referring to, for example, (racial) biases, data artefacts or specific knowledge of interest (here done with the text modality). b, A low-dimensional UMAP projection of the semantic embeddings provides a structured overview of the knowledge in the model. Each point corresponds to the encoded concept of a model component. By searching for human-defined concepts, we can add descriptions to all parts of the semantic space. c, Having grouped the knowledge into concepts, attribution graphs reveal where concepts are encoded in the model and how they are used (and interconnected) for inference. When predicting ‘ox’, we learn that ox-cart-related background concepts are used. Importantly, we can also identify relevant knowledge that could not be labelled and should be manually inspected by the user. d, The set of unexpected concepts includes the ‘Indian person’, ‘palm tree’ and ‘watermark’ concepts, which correlate in the dataset with ‘ox’. We can find other affected output classes, for example, ‘butcher shop’, ‘scale’ and ‘ricksha’ for the ‘Indian person’ concept. Watermarks depicted in images of panels a and d were manually overlaid to simulate artefacts. Credit: images from Pexels and Unsplash.
a, (1) In the first step, a set of valid and spurious concepts is defined with text descriptions, for example, ‘curved horns’ or ‘palm tree’ for ‘ox’ detection. (2) Afterwards, we evaluate the alignment of each model component (defined in ‘Auditing concept alignment’) with spurious (y axis) and valid (x axis) concepts, to determine if the concepts it encodes are primarily valid, spurious, a combination of both or neither. The size of each dot in the chart represents the importance of a component for ‘ox’ detections. We learn that ResNet50v2 relies on the ‘Indian person’, ‘palm tree’ and ‘cart’ concepts. (3) Last, we can test our model and try to distinguish the ‘ox’ output logits on ‘ox’ images (from the test dataset) and diffusion-based images with spurious features only. When several spurious features are present, as for ‘Indian person pulling a cart under palm trees’, the model outputs become more difficult to separate, indicated by a lower AUC score. b, When auditing the alignment of ResNet to valid concepts for 26 ImageNet classes, we find that in all cases, spurious or background concepts are used. For each class, we show a violin plot depicting the distribution of alignment scores for the top n = 20 most relevant components. Special markers indicate the median, minimum and maximum alignment. The radius of each dot reflects the highest relevance of a component on the test set. Credit: images in a, Unsplash; b, Pexels.
We also query the model for the concept ‘watermark’ through text-based probing. The retrieved neurons encode watermarks and other text superimposed on an image. Such data artefacts may become part of the prediction strategy of the model, known as shortcut learning12,32 or the Clever Hans phenomenon15, and massively undermine its trustworthiness (the model predicts correctly but for the wrong reason33). Although previous works have unmasked such watermark-encoding neurons more or less by chance15,34, SemanticLens allows one to intentionally query the model for the presence of such neurons.
In addition to searching for bias- or artefact-related neurons, we can also query the model for specific knowledge, for example, the concept ‘bioluminescence’. The results show that this concept has been learned by the ResNet50v2 model. Such specific knowledge queries can help ensure that the model has learned all the relevant concepts needed to solve a task, as demonstrated in the ABCDE rule for melanoma detection in ‘Auditing concept alignment with expected reasoning’. Notably, SemanticLens not only allows one to query the model for specific concepts but also to identify the output classes for which concepts are used and the respective (training) data, as shown in Fig. 2d. More examples, comparisons between models and details are provided in Supplementary Note C.
Describing what knowledge exists and how it is used
Another feature of SemanticLens is its ability to describe and systematically analyse what concepts the model has learned and how they are used. Figure 2b provides an overview of the internally encoded concepts in the ResNet50v2 model (by penultimate-layer neurons) as a Uniform Manifold Approximation and Projection (UMAP) projection of the semantic embeddings ϑ. Here, for example, a text-based search for ‘animal’ results in aligned embeddings on the left (indicated in red), whereas transport-related embeddings are in the centre (blue). Even more insights can be gained when systematically searching and annotating semantic embeddings, as described in the following.
Labelling and categorizing knowledge
To structure the learned knowledge systematically, we assign a text-form concept label (from a user-defined set) to a neuron embedding if its alignment exceeds the alignment with a baseline, which is an empty text label. The labelled embeddings can then be grouped according to their annotation. For example, all embeddings matching ‘dog’ are grouped together, which reduces the complexity, especially if there are many neurons with similar semantic embeddings. For example, the results for ResNet indicate that the model has over a hundred neurons related to dog, as illustrated in Fig. 2b, which shows the overall top-aligned label from the expected set for clusters of semantic embeddings ϑ. Further details (including labels) and examples are provided in Supplementary Notes D.1 and D.2, respectively.
It is possible to ‘dissect’14 the knowledge in a model at different levels of complexity, ranging from broad categories such as ‘objects’ and ‘animals’ to more fine-grained concepts such as ‘bicycle’ or ‘elephant’. For instance, in Supplementary Note D, we categorize the model components relevant to the ‘ox’ class into ‘breeds’ like ‘water buffalo’, ‘work’-related concepts such as ‘ploughing’, and ‘physical attributes’ such as ’horns’. Importantly, labelling not only facilitates the assessment of what the model has learned but also identifies gaps in its knowledge, that is, cases where no neuron aligns with a user-defined concept. In the studied ResNet model, for instance, no neuron encodes the ox breeds ‘Angus’ or ‘Hereford’, indicating areas where further training data could enhance model performance. Notably, the faithfulness of labels is important35, which is evaluated in Supplementary Note D.4.
Understanding how knowledge is used
Understanding how the model uses learned knowledge is as crucial as knowing what knowledge exists. For instance, although ‘wheels’ can be a valid concept for detecting sports cars, it should not be relevant for detecting an ox, which is, however, measurable for ResNet. Figure 2c shows the attribution graph for the class ‘ox’, constructed by computing conditioned attribution scores15. The graph reveals associations between neuron groups with the same concept label. For the class ‘ox’, in the attribution graph in Fig. 2c, for example, next to the ‘wheels’ concept there is another highly relevant ‘long fur’ concept encoded by neuron 179 in layer 3, which in turn relies on a ‘grass’ concept in the preceding layer, indicating that neuron 179 encodes long-furred animals on green grass. Attribution graphs, thus, not only describe what and how concepts are used but also enhance our understanding of subgraphs (‘circuits’) within the model. A complete graph is detailed in Supplementary Note D.5.
The link among knowledge, data and predictions
Notably, some components do not align with any of the predefined concept labels, yielding embedding similarities that were equal to or lower than those obtained using an empty text prompt. As shown in Fig. 2d, manual inspection of these unexpected concepts reveals associations with ‘Indian person’, ‘palm tree’ and ‘watermark’, traced to neurons 179, 1560 and 800 in layer 3, respectively. All three concepts correspond to spurious correlations in the dataset, for example, farmers using an ox to plough a field, palm trees in the background or overlaid watermarks, where the training data responsible can be generally identified by retrieving highly activating samples \({\mathcal{E}}\). The same neurons are also relevant to other ImageNet classes: ‘butcher shop’, ‘scale’, and ‘rickshaw’ for ‘Indian person’; ‘thatch’, ‘bell cote’ and ‘swim trunk’ for ‘palm tree’; and ‘Lakeland terrier’, ‘bulletproof vest’ and ‘safe’ for ‘watermark’. By inherently connecting data, model components and predictions, SemanticLens is an effective and actionable tool for model debugging, as further described in ‘Towards robust and safe medical models’.
Identifying common and unique knowledge
So far, we have investigated a single model in semantic space. However, this space allows the embedding and comparing of several models with different architectures, layers or model parts. Thus, the influence on learned concepts when changing the network architecture or training hyperparameters, such as the training duration, can be studied.
Supplementary Note E compares using SemanticLens two ResNet50 models trained on ImageNet. One (ResNet50v2) was trained more extensively, which results in higher test accuracy. As illustrated in Supplementary Fig. E.1, both models share common concepts, for example, bird-related ones. However, whereas the better-trained ResNet50v2 learns more class-specific concepts, like the unique fur texture of the Komondor dog breed, ResNet50 tends to learn more abstract, shared concepts. For instance, ResNet50 detects a Komondor using a mop-like concept also used for class ‘mop’, whereas ResNet50v2 learns a concept specific to ‘Komondor’. This is in line with works that study the generalization of neural networks after long training regimes, which observed that latent model components become more structured and class-specific36. We further provide quantitative comparisons through network dissection in Supplementary Note D.3. Alternatively, SemanticLens allows one to compare models quantitatively without access to concept labels by evaluating the similarity between the knowledge in the models. In Supplementary Note E, we discuss the alignment of various pretrained neural networks across layers and architectures.
Auditing concept alignment with expected reasoning
The analyses introduced in ‘Understanding the inner knowledge of AI models’ enable the quantification of the alignment of a model with human expectations by measuring its reliance on valid, spurious or unexpected concepts. The steps of an alignment audit, outlined in Fig. 3a, include (1) defining concepts, (2) evaluating concept alignment and (3) testing model behaviour.
Defining a set of expected concepts
First, a set of valid and spurious concepts is defined and compared against the concepts actually used by the model. For illustration, we revisit the ox example. Valid concepts include ‘curved horns’, ‘wide muzzle’ and ‘large muscular body’, as shown in Fig. 3a (left). On the other hand, we are also aware of spurious correlations, such as ‘palm tree’, ‘Indian person’ and ‘watermark’. Notably, all these concepts can be defined within the modality of the data domain of the model (with example images) or, as demonstrated here, simply with text prompts when using a multimodal foundation model to compute embeddings.
Evaluating alignment to valid and spurious concepts
The alignment of the knowledge in a model with user-defined spurious or valid concepts is visualized in the scatter plot in Fig. 3a (middle) when detecting ‘ox’. Concretely, we calculate the maximum alignment between an embedding ϑ and all probing embeddings ϑprobe within a set (valid or spurious), with mathematical formulations detailed in ‘Auditing concept alignment’. Each dot in the plot represents a neuron in the penultimate layer, with its size indicating its highest importance (shown in parentheses) during inference on the test set.
Several spurious concepts, such as ‘palm tree’, ‘Indian person’ and ‘cart’, are identified besides valid concepts such as ‘short’, ‘rough fur’ and ‘curved horns’. Notably, neurons that do not align to any user-defined concept can be manually inspected, as illustrated in Fig. 2d, and incorporated into the set of spurious or valid concepts. As discussed for a VGG model in Supplementary Note F, lower overall alignment scores can also result for neurons that encode for highly abstract concepts or that exhibit ‘polysemantic’ behaviour, encoding several semantics simultaneously.
Testing models for spurious behaviour
Although SemanticLens enables the quantification of the reliance of a model on valid or spurious features (for example, by sharing spuriously aligned components), it is equally important to assess the actual impact of identified spurious features on inference. Here we use a model test evaluating the separability of two sets of outputs37: one generated from images containing valid features (associated with the ‘ox’ class) and the other from images with spurious features, as illustrated in Fig. 3a (right). When testing the model on images (generated with Stable Diffusion) for a single concept (‘Indian person’, ‘palm tree’ or ‘cart’), the model output logits for ‘ox’ are clearly distinguishable from those attained from ‘ox’ images, achieving area under the curve (AUC) scores above 0.98. However, when several spurious features are presented simultaneously and we test the model on images combining all three concepts, the ‘ox’ output logits are further amplified. Specifically, the ‘ox’ class ranks among the top five predictions in over half of the spurious samples, resulting in an AUC of 0.91, as further detailed in Supplementary Note F.
Problematic concept reliance everywhere
The previous example highlights the presence of unexpected spurious correlations, such as the association of palm trees with ‘ox’. Expanding on this, we evaluate the alignment of model components with valid concepts across 26 other ImageNet classes, including ‘shovel’, ‘steel drum’ and ‘screwdriver’. Figure 3b presents the resulting highest alignment scores with a valid concept for neurons, where size again indicates relevance for ‘ox’. Notably, no class shows complete alignment of all relevant model components with valid concepts. In every case, spurious or background features are relevant, including snow for ‘shovel’, ‘Afro-American person’ for ‘steel drum’, and ‘child’ for ‘screwdriver’. A comprehensive overview of the concepts used by the model is provided in Supplementary Note F.
Unaligned models are often challenging to interpret
Analysing popular pretrained models on ImageNet reveals substantial variation in alignment with valid expected concepts, often due to learned concepts that are neither clearly valid nor spurious, as shown for VGG-16 in Supplementary Note F. For instance, VGG-16 contains several polysemantic neurons that perform several roles in decision-making, which generally reduces alignment. More performant and wider models tend to have more specialized and monosemantic neurons, as quantified in ‘Evaluating human-interpretability of model components’, and exhibit higher alignment scores, as shown in Supplementary Note F, thus highlighting the link between interpretability and auditability. After assessing the issue of hidden concepts across components in Supplementary Note F.1.4 for ResNet50v2, we suggest that they have a minimal impact on audit faithfulness.
Towards robust and safe medical models
A popular medical use case for AI is melanoma detection in dermoscopic images, as shown in Fig. 4a. In the following, we demonstrate how to debug a VGG-16 model with SemanticLens that is trained to discern melanoma from other irregular or benign (referred to as ‘other’) cases in a public benchmark dataset28,38,39.
a, The ABCDE rule is a popular guide to visual melanoma clues. We expect models to learn several concepts corresponding to the ABCDE rule, as well as other melanoma-unrelated indications (such as regular border) or spurious concepts, including hairs or a band-aid. b, In a semantic space visualized with a UMAP projection, we can identify valid concepts, such as blue-white veil for ‘melanoma’, but also spurious ones such as red skin or ruler. c, When investigating the importance of concepts, red skin or band-aid concepts are strongly used for the ‘other’ (non-melanoma) class. Ruler concepts are used with slightly higher relevance for ‘melanoma’. d, We can improve the safety and robustness of our model either by changing the model and removing spurious components or by retraining it on augmented data. Whereas both approaches lead to improved clean performance, the influence of artefacts is only substantially reduced through retraining. Images in a, b and d are adapted with permission from ref. 28 under a Creative Commons license CC BY 4.0 and ref. 39 under a Creative Commons license CC BY 4.0.
ABCDE rule for melanoma detection
Dermatologists have created guidelines for visual melanoma detection, such as the ABCDE rule, short for asymmetry, border, colour, diameter and evolving40. We will use SemanticLens to evaluate whether the model captures corresponding concepts, such as ‘asymmetric lesion’ (A), ‘ragged border’ (B), ‘blue-white veil’ (C), ‘large lesion’ (D) and ‘crusty surface’ (E). In addition, we also define concepts for benign conditions, other skin diseases and known spurious correlations41,42, including ‘hairs’, ‘band-aids’, ‘red-hued skin’, ‘rulers’, ‘vignetting’ and ‘skin markings’. Refer to Supplementary Note F.2.1 for a full list of concepts.
Finding bugs in medical models
We embed neurons in the last convolutional layer in VGG into the semantic space of a CLIP model trained on skin lesion data31 and probe for concepts using textual embeddings. As shown in Fig. 4b, the semantic embeddings are structured, aligning to concepts related to ‘irregular’ at the top (red), ‘melanoma’ at bottom left (blue) and ‘regular’ at bottom right (green). Relevant melanoma indicators, such as ‘blue-white veil’ and ‘irregular streaks’, emerge, alongside benign features like ‘regular border’. However, several spurious components are also detected: neuron 403 encodes for measurement scale bars, 508 detects ‘blue band-aids and 272 responds to red skin.
To quantify how concepts are used by the model, we computed their highest importance for predicting the ‘melanoma’ or ‘other’ class on the test set, as shown in Fig. 4c. Alarmingly, spurious concepts are highly relevant: ‘red skin’ and ‘blue band-aid’ are strongly used for ‘other’, whereas ‘measurement scale bar’ is slightly more strongly used for ‘melanoma’.
Model correction and evaluation
SemanticLens allows one to identify model components and associated data, which can be used to reduce the influence of background features, such as red skin, plasters and rulers. To debug the model43, we apply two approaches: (1) pruning 40 identified spurious neurons without retraining and (2) retraining on a cleaned and augmented dataset. For retraining, we remove training samples containing artefacts, identified by studying the highly activating samples of our labelled components. We further overlay hand-crafted artefacts onto training images to desensitize the model, as illustrated in Fig. 4d (left).
The results in Fig. 4d (right) show that both strategies, pruning and retraining, lead to increased accuracy on a clean test set (without artefact samples), especially for melanoma (from 71.4% to 72.8%). We further modify the data by artificially inserting artefacts: cropping out ruler and plasters from real test samples and overlaying them onto clean test samples, as done in ref. 34. Additionally, for red skin, we introduce a reddish hue, as detailed in Supplementary Note F.2.3. Interestingly, the pruned model remains highly sensitive to artefacts, with the test accuracy for non-melanoma samples still dropping by over 20% when adding red colour. Although computationally more expensive, only retraining leads to a strong reduction in artefact sensitivity. Further details and discussions are provided in Supplementary Note F.2.3.
Evaluating human-interpretability of model components
Deciphering the meaning of concept examples \({\mathcal{E}}\) can be challenging, especially when neurons are polysemantic and encode for several semantics. We introduce a set of easily computable measures that assess how ‘clear’, ‘similar’ and ‘polysemantic’ concepts are perceived by humans, as inferred from their concept examples \({\mathcal{E}}\). Additionally, we introduce a measure to quantify the ‘redundancies’ present within a set of concepts. All measures are based on evaluating similarities of concept examples \({\mathcal{E}}\) in semantic space \({\mathcal{S}}\), with mathematical definitions provided in ‘Human-interpretability measures for concepts’.
Alignment of interpretability measures with human perception
Aiming to assess human-interpretability, we first evaluate the alignment between human judgements and our proposed measures (similarity, clarity and polysemanticity) through user studies. Over 218 participants were recruited through Amazon Mechanical Turk for 15-min tasks in which they evaluated concept examples from the ImageNet object detection task. For each interpretability measure, we designed an independent study consisting of both qualitative and quantitative experiments. Further details regarding the study design, the models used and the data-filtering procedures can be found in Supplementary Note G.1.
We find high alignment between our measures and human perception, with correlation scores above 0.74 (Fig. 5a), consistent with recent works using textual concept examples44. Regarding concept similarity, human-alignment varies across foundation models, namely DINOv2 (ref. 30; unimodal), CLIP-OpenAI23, CLIP-LAION45 and the most recent CLIP-Mobile29 (specific variants are reported in Supplementary Note G.1). Our results indicate that more recent and more performant CLIP models are also more aligned with human perception. Other hyperparameter choices, such as the similarity measure used, are compared in ‘Methods’. In an odd-one-out task, where participants identified outlier concepts, our similarity measures are often better than those of the participants, indicating that computational measures can be more reliable than humans. However, Amazon Mechanical Turk participants may prioritize speed over accuracy, potentially affecting performance.
a–c, ‘Clarity’ refers to how clear and easy it is to understand the common theme of concept examples. Polysemanticity indicates whether several distinct semantics are present in the concept examples. Similarity refers to the similarity of concepts. Redundancy describes the degree of redundancy in a set of concepts. a, Our computable measures align with human perception in user studies, resulting in correlation scores above 0.73. Generally, more recent and performant foundation models have higher correlation scores. For each scatter plot, we report the number n of data points. Each point corresponds to a study question and its average perceived user rating. b, Interpretability differs strongly for common pretrained models. Usually, ViTs or smaller and less performant convolutional models show lower interpretability. c, We can optimize model interpretability with respect to hyperparameter choices, such as dropout or activation sparsity regularization, during training. Although dropout leads to more redundancies besides improved clarity of concepts, applying a sparsity loss improves interpretability overall. Green arrows indicate a positive effect on interpretability, gray arrows a neutral effect and red arrows a negative effect.
Rating and improving interpretability
The difficulty of understanding the role of components in pretrained models can vary strongly, as observed in the sections ‘Understanding the inner knowledge of AI models’ and ‘Search capability’. This is confirmed by evaluating various popular neural networks trained on ImageNet using our introduced measures for the penultimate-layer neurons, as illustrated in Fig. 5b. Larger and broader models, like ResNet101, show higher degrees of redundancy, as more neurons per layer allow redundancies to increase, for example, to increase robustness. By contrast, narrow models, such as ResNet18, have a smaller neural basis, potentially leading to superimposed signals and higher polysemanticity46.
The convolution-based ResNet architecture shows higher concept clarity compared to transformer-based ViTs. The nonlinearities in the rectified linear units (ReLUs) in ResNet allow them to associate a high neuronal activation with a specific active input pattern. ViTs often refrain from using ReLUs, which enables them to superimpose signals throughout model components, ultimately leading to high polysemanticity47. Recent efforts to improve large language model interpretability are addressing this by introducing sparse autoencoders (SAEs) with ReLUs to enhance interpretability48. Additionally, more extensively trained models, such as ResNet50v2, show clearer, more interpretable components than ResNet50, indicating that training parameters may influence latent interpretability. We explore this further below.
Dropout regularization is effective for reducing overfitting, which prevents a high reliance on a few features by randomly setting a fraction of component activations to zero during training. Our results shown in Fig. 5c indicate that VGG-13 model components become more redundant but also clearer when dropout is applied during training on a subset of ImageNet (standard errors given by grey error bars for eight runs each). It can be expected that more redundancies form, as redundancies improve robustness when components are pruned. On the other hand, neurons are measured to become more class-specific and thus clearer. Notably, different architectures respond differently, with ResNet34 and ResNet50 less impacted. Qualitative examples of concepts, detailed training procedures and results are provided in Supplementary Fig. G.2.
L1 sparsity regularization on neuron activations, as common for SAEs, improves interpretability, resulting in more specific, less polysemantic and semantically redundant neurons. We further investigate the effect of task complexity, number of training epochs and data augmentation on latent interpretability in Supplementary Note G.2.
