Multiscale knowledge distillation with attention based fusion for robust human activity recognition

Overview

Overall, we propose a knowledge distillation architecture with multiple teacher networks and a single student network, as illustrated in Fig. 1. This architecture first learns modal knowledge from wearable sensors through pre-training the teacher networks. Subsequently, the acquired knowledge is transferred to the student network through knowledge distillation, guiding the training of the student network. Finally, the student network achieves a performance similar to that of inputting multimodal data when only video data is provided.

In Fig. 1, we use virtual images generated from one-dimensional time-series data from wearable sensors as inputs to the teacher network. Multiple teacher networks correspond to different types of wearable sensor data. The input to the student network is a sequence of 32 frame images constituting a video frame sequence.

Both the teacher and student networks consist of four stages. After each stage, both modal networks obtain corresponding feature maps. The teacher network transfers heterogeneous knowledge from intermediate layers to the student network through the semantic graph mapping (SGM) module and calculates semantic loss (SE Loss) and soft target loss (ST Loss) at the final stage for training the student network.

Details of stage

The Fig. 2 provides detailed specifics of each stage. For image recognition, most traditional methods rely on convolutional neural networks (CNNs). However, CNNs have the drawback of a relatively small receptive field during the early stages of feature extraction may lead to the inability of the model to effectively identify detailed features in images at the beginning. Therefore, in the teacher network, we adopt the feature extraction method of the Swin Transformer (Swin), which achieves global understanding through self-attention mechanisms and window-based attention mechanisms that CNNs lack.

For feature extraction in videos, traditional recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) suffer from problems such as forgetting long-term frame relationships, making it difficult to focus on long-term temporal relationships. In the student network, we introduce video swin transformer (Video Swin), which simultaneously considers both temporal and spatial dimensions. This helps the model systematically extract motion information from videos. Both approaches are based on Swin, providing inherent compatibility between them.

Next, we briefly introduce the working principle of Swin Transformer. Given an image $I \in \mathbb {R}^{h\times w\times 3}$, we first divide it into patches of size $p$, and then map each patch to dimensions recognizable by Swin Transformer, denoted as $x \in \mathbb {R}^{H \times W \times C}$, where $H=h/p$, $W=w/p$, and $C=C\times p^2$. Except for the first stage, each stage begins with a patch merging operation to obtain the input for the next Stage Block, denoted as $x^i \in \mathbb {R}^{H/2\times W/2\times 2C}$, where $i$ represents the $i$-th stage, $i>1$, and $H,W,C$ are from the previous stage. After several stages, we obtain our final output.

To leverage the complementarity and consistency among different types of sensor data, we design the Fusion and Tuning (FT) module in the first three stages of the teacher network. This module is used to fuse and adjust features from different types of sensors. First, in the teacher network, we map the virtual images from wearable sensors to feature embeddings of dimensionality through patch embedding. Then, we extract features from different types of data using Swin blocks. Subsequently, through the FT module, we fuse the features from different types of sensors and adjust the original features of different types of data. This process helps us maintain consistency within the data while incorporating complementary knowledge from different types of data.

Since traditional knowledge distillation only transfers knowledge at the final feature output layer, neglecting the rich feature knowledge in intermediate layers, we propose the SGM module to transfer heterogeneous knowledge from the intermediate layer of the teacher network to the student network. In the student network, we first map the video frame sequence to feature embeddings of dimensionality through patch embedding. Then, we perform knowledge distillation between the corrected features from the teacher network and the features of the student network through the SGM module. Subsequently, the feature maps of each network undergo patch merging before entering the next block. Finally, SE loss and ST loss calculations are performed on the output features of the last stage to train the student network.

We seamlessly integrate the FT module and the SGM module with the two Swin architectures and transfer heterogeneous knowledge through knowledge distillation. In the final implementation, we achieve both approximate multimodal effects and maintain high accuracy in the absence of data.

Virtual image generation

The raw sensor data typically only contains motion information, thus concealing time-related details. Relying solely on raw data for analysis overlooks the correlation of each time step, which is not conducive to the final prediction.To preserve local temporal relationships in one-dimensional time series data from wearable sensors and better convey their content to RGB video data, we employed gramian angular field (GAF) to convert the one-dimensional time series data into two-dimensional, three-channel color images.

Since wearable sensor data contain time series signals from three axes (x, y, z), we represent the signal data from one axis as $X=\{x_1, x_2,…, x_n\}$. Then, we normalize the original signal X using the min-max normalization method to scale it within the range of $[-1, 1]$, obtaining the normalized signal $\hat{X}$:

$$\begin{aligned} \hat{X}=\frac{(X-\text {max}(X))+(X-\text {min}(X))}{\text {max}(X)-\text {min}(X)} \end{aligned}$$

(1)

Next, the normalized signal $\hat{X}$ is represented in polar coordinates using a transformation function f. Specifically, this involves using the timestamp as the radial coordinate $\rho$, and expressing the normalized signal $\hat{X}$ as the polar angle $\theta$ in the range $[0, \pi ]$. Ultimately, the one-dimensional normalized signal $\hat{X}$ is transformed into two-dimensional polar coordinates $(\rho , \theta )$:

$$\begin{aligned} f(x_i,t_i,N)= {\left\{ \begin{array}{ll} \theta _i= \arccos (\hat{x_i}), &{}x_i\in \hat{X} \\ \rho _i = \frac{t_i}{N} \end{array}\right. } \end{aligned}$$

(2)

Where $\hat{x_i}$ represents the normalized temporal signal, $t_i$ denotes the temporal position of the signal, and N represents the number of data points in the temporal signal. After encoding the one-dimensional time series signal into polar coordinates, the calculation of the sum of triangles between points facilitates the easy derivation of the correlation coefficient between two points within local time. As correlation coefficients can be computed using the cosine of the angle between vectors, the correlation between time i and time j can be expressed as $\cos (\theta _i + \theta _j)$ . Consequently, we obtain the GAF correlation matrix G:

$$\begin{aligned} G=\begin{pmatrix} \cos (\theta _1 + \theta _2) &{} \dots &{} \cos (\theta _1 + \theta _n) \\ \vdots &{} \ddots &{} \vdots \\ \cos (\theta _n + \theta _1) &{} \dots &{} \cos (\theta _n + \theta _n) \end{pmatrix} \end{aligned}$$

(3)

Since $\cos (\theta )$ is uniquely determined within the interval $\theta \in [0, \pi ]$, this implies that the method can retain the characteristics of the original data. By computing the correlation matrix, it is possible to augment the local temporal relationships lacking in the original data.

This approach can preserve local temporal relationships in the form of temporal correlations as the number of timestamps increases. The wearable sensor data is formatted as three-axis time series (x, y, and z axes), assuming a time length of n for each axis. Using the aforementioned method, we obtain a $n \times n$ GAF matrix for each axis. Subsequently, the GAF matrices of the three axes are concatenated along the channel dimension to form a three-channel image of size $n \times n \times 3$, denoted as $P = \{ G_x, G_y, G_z \}$.

Through this method, we successfully convert one-dimensional time series data into a two-dimensional image format, followed by the stacking of the images of the three axes to obtain the final three-dimensional image format. This RGB image generated by encoding sensor data serves as the input to the teacher network.

Fusion and tuning

In the structure of the teacher network, we employ the Swin Transformer (Swin) as the feature extractor. Due to its utilization of global attention mechanisms, the Swin Transformer can better capture the global relationships within the data compared to traditional convolutional neural networks. However, for different types of sensor data, using the same backbone will result in different attention feature matrices. Existing attention mechanisms fail to adequately focus on the correlations between features from different types of sensors, often overlooking such correlations during multi-source feature fusion. To fully exploit the correlation information among different types of data, we integrate the similarity within features and the correlation knowledge between multiple sensor features between each Swin-Stage module. Subsequently, the fused activation signals are used to adjust the features of various sensor types. The similarity fusion module between two types of sensors is illustrated in Fig. 3.

Intra-source global context modeling

Incorporating the FT module into each stage module of Swin for feature fusion of multi-source data.Suppose there are m types (e.g. different types of sensor data), and the backbone network consists of s Stage modules. Given an input batch size of B, the output feature maps $F_m^s$ of different types of data m at a specific Stage block s in the network are represented as $F_m^s\in \mathbb {R}^{B\times L\times C}$, where B is the batch size, L is the number of patches, and C is the feature dimension per patch. Inspired by attention-based knowledge transfer methods^51,53, this approach utilizes activation correlation for knowledge transfer. First, reshape the feature maps $F_m^s$ into $R_m^s\in \mathbb {R}^{B\times L\cdot C}$, then perform matrix multiplication with its transpose and L2 normalization to obtain the similarity matrix within each type of data, denoted as $G_m^s\in \mathbb {R}^{B\times B}$.

$$\begin{aligned} \hat{R}_m^s= & {} R_m^s \times R_m^{s\mathsf T} \end{aligned}$$

(4)

$$\begin{aligned} G_m^s= & {} \frac{\hat{R}_m^s}{ \begin{Vmatrix} \hat{R}_m^s \end{Vmatrix}_2} \end{aligned}$$

(5)

where $\begin{Vmatrix} \cdot \end{Vmatrix}_2$ represents the L2 norm. After obtaining the similarity matrix within the data, we use it to guide the construction of global context within the data. First, we employ one-dimensional adaptive average pooling to compress the feature maps outputted by the Stage, obtaining the compressed feature vectors $V_m^s\in \mathbb {R}^{B\times C}$. Then, we perform matrix multiplication between the similarity matrix $G_m^s$ and the compressed feature vectors $V_m^s$ to obtain the global context information within the modality $S_m^s \in \mathbb {R}^{B\times C}$, with the calculation formula as follows:

$$\begin{aligned} V_m^s(B,C)=\frac{1}{L}\sum _{i=1}^{L}F_m^s(B,i,C) \end{aligned}$$

(6)

$$\begin{aligned} S_m^s=G_m^s \times V_m^s \end{aligned}$$

(7)

Feature fusion and tuning

The global context information $S_m^s$ include similarity features within the data, which can effectively reflect the characteristics of each type of data. To fully exploit the correlation and complementary information between different types of data, we employ three joint representation forms for the global context information $S_m^s$ across different types: concatenation, summation, and Hadamard product. These methods have been proven to be effective in previous studies⁵⁴. Below are the formulas for generating the three joint representation activation signals:

$$\begin{aligned} E_{con}^s= & {} W_{con}^s[S_1^s, \dots ,S_m^s]+b_{con}^s \end{aligned}$$

(8)

$$\begin{aligned} E_{sum}^s= & {} W_{add}^s \left( \sum _{i=1}^m S_i^s \right) +b_{sum}^s \end{aligned}$$

(9)

$$\begin{aligned} E_{had}^s= & {} W_{had}^s \left( \prod _{i=1}^m S_i^s \right) +b_{had}^s \end{aligned}$$

(10)

where $[\cdot ,\cdot ]$ represents concatenation operation, $\prod _{i=1}^m$ denotes element-wise multiplication (Hadamard product) operation on the global context information, W and b are learnable parameters, $[W_{con}^s,b_{con}^s],[W_{sum}^s,b_{sum}^s],[W_{had}^s,b_{had}^s]$ respectively represent the weights and biases of the concatenation, summation, and Hadamard product layers. These three joint representation layers scale the channel features of $S_m^s$ and apply the GELU activation function after the scaling operation to enhance the model’s generalization capability.

After obtaining the activation signals of the three aggregated global context information, we use them to correct the original feature maps to incorporate complementary information from different types. We use the GELU activation function to introduce non-linearity to the activation signals, then simply add them to obtain the fusion knowledge for correcting the original type features. The formulas for correcting different type features are as follows:

$$\begin{aligned} \hat{F}^s_m=(GELU(E_{con}^s)+GELU(E_{sum}^s)+GELU(E_{had}^s)) \odot F_m^s \end{aligned}$$

(11)

where $\odot$ denotes element-wise multiplication operation in the C dimension. Through the FT module, we can achieve multi-source fusion of attention features and re-adjustment of features of different types, allowing different types of features to calibrate each other while maintaining the similarity within the data and the correlation between different types of data.

Semantic graph mapping

Traditional knowledge distillation methods typically transfer knowledge at the final feature output layer, overlooking the rich information in the intermediate layers. When dealing with heterogeneous data such as motion sensors and visual sensors, the complementary information in the intermediate layers is crucial for transferring heterogeneous knowledge. To better achieve heterogeneous knowledge transfer from the teacher network to the student network, we designed a semantics-based knowledge distillation module, which operates on the first s-1 stage modules to generate semantic-aware attention heatmaps for the intermediate layers of the teacher and student networks, aiming to train the student network accordingly. The objective of this module is to highlight important semantic regions in the output feature maps of each Stage and train the student network based on the differences between the teacher and student networks.

Previous research⁵⁵ has demonstrated that ablating certain units in a model can assess the importance of those units for specific category recognition. Therefore, this paper combines original images with their channel-zeroed ablated data to generate semantic ablation decreases, obtaining a good multimodal semantic visual interpretation through semantic ablation decreases.

We take the student network as an example to explain the operation principle of the SGM module in Fig. 4. During the training of the student network, the student network takes a combination of the original data $I \in \mathbb {R} ^{B\times C\times D\times H\times W}$and the ablated data $I_a \in \mathbb {R}^{B\times C\times D\times H\times W}$ with zeroed channel values as input, denoted as combined data $[I;I_a]\in \mathbb {R}^{2B\times C\times D\times H\times W}$. Firstly, we obtain the network’s classification prediction and find the corresponding semantics. Let y represent the predicted category of the original data, and $y_a$ represent the predicted category of the ablated data. Then, we utilize Bert⁵⁶ to compute the semantic embeddings for the original dataB and the ablated data $B_a$:

$$\begin{aligned} \begin{aligned} B&= Bert(y) \in \mathbb {R}^{B\times 768}\\ B_a&= Bert(y_a) \in \mathbb {R}^{B\times 768} \end{aligned} \end{aligned}$$

(12)

Next, leveraging manifold learning⁵⁷ which can extract intrinsic structures from data, we constructed two graphs for the class score vectors of I and $I_a$. In these graphs, vertices represent feature vectors used for classification, while edges represent relationships between feature vectors, determined by Gaussian similarity. Assuming the input feature vectors are $x_i$ and $x_j$, the weight of the edge between them is given by:

$$\begin{aligned} W_{i,j}=exp\left( -\frac{ \begin{Vmatrix} x_i-x_j \end{Vmatrix}^2}{2}\right) \end{aligned}$$

(13)

Once we obtained the graph structure W, we introduced normalized graph Laplacians⁵⁸ and applied them to W, which helps further extract the internal partitioning features of W, i.e. $L=D^{-\frac{1}{2}}WD^{-\frac{1}{2}}$. Here, D is a diagonal matrix where the value on the diagonal of the $i^{th}$ row is equal to the sum of the $i^{th}$ row of W. This way, the manifold structure of the data can be effectively represented in the graph matrix $L\in \mathbb {R}^{B\times B}$.

To preserve the structure of the original features, we first multiply the semantic embeddings by the graph matrix, which is essentially a form of manifold regularization. Next, we define a slope metric $k\in \mathbb {R}^{B\times 768}$ to measure the rate of semantic change from the original data to the ablated data, and based on this metric, we allocate attention weights to the feature maps. The computation of k is given by Eq. (14):

$$\begin{aligned} k = \frac{L\times B-L_a\times B_a}{L\times B} \end{aligned}$$

(14)

where L and $L_a$ are the normalized graph similarity matrices of the original data and the ablated data, respectively. By computing the rate of semantic change, we can represent the importance values of features in terms of the magnitude of the slope while preserving the essence of the data. Finally, we perform a Hadamard product operation between the semantic change rate and the original features to obtain the semantic graph mapping, which will be used for the loss calculation in knowledge distillation.

$$\begin{aligned} M_m^s=LN(k\cdot F_m^s) \end{aligned}$$

(15)

where $LN(\cdot )$ represents the LayerNorm function⁵⁹. We utilize the semantic graph mappings generated by the teacher network as soft targets, along with those generated by the student network, to construct the SGM knowledge distillation loss. Assuming we have m teacher modalities and one student modality with s Stage modules, we employ the mean squared error (MSE) loss of SGM to transfer cross-modal heterogeneous knowledge.

$$\begin{aligned} L_{SGM}=\frac{\sum _{i=1}^{s-1}\sum _{j=1}^{m}MSE(M_S^i,M_j^i)}{m\times (s-1)} \end{aligned}$$

(16)

where we only utilize the last $s-1$ stage modules to compute the SGM loss, and the last module is used for calculating the semantic loss; $M_S^i$ represents the semantic graph mapping of the i-th Stage of the student network.

Training

Our framework is a multi-scale knowledge distillation architecture. We utilize a sensor image generation model to convert one-dimensional time-series sensor data into RGB image data recognizable by Swin through GAF, thereby preserving the original features while enhancing local temporal relationships for subsequent feature extraction. After the attention computation is completed in the Stage modules of the teacher network, the feature maps are input into the Fusion and Tuning (FT) module to obtain similarity features of different types of wearable sensor data and perform multi-type feature fusion. During the training of the student network, we use the Semantic Graph Mapping (SGM) module to map the semantic key parts of intermediate feature maps and calculate knowledge distillation loss using the semantic feature maps of the teacher network as soft targets. We seamlessly integrate these modules into the Swin knowledge distillation architecture to address the heterogeneous HAR problem between sensor data and visual data.

We first independently train the teacher network, as shown in Fig. 2, using Swin as the backbone network to construct the teacher network, which primarily takes sensor image data as input and undergoes fine-tuning training on Swin with pre-trained weights. After the training of the teacher network is completed, it guides the training of the student network. Our ultimate goal is to use the student network, with video data as input, to generate predictions similar to those trained with multimodal data. The teacher network’s fused features are used together with the features learned by the student network for computing the semantic graph mapping loss, thereby achieving knowledge transfer.

To train the teacher network, we use the output features of the last stage as the semantic feature output of the entire network and compute the mean squared error (MSE) loss between it and the target true semantic embeddings, which we refer to as the semantic loss. Assuming we have m teacher modalities, the total loss of the teacher network, combined with semantic loss and predicted cross-entropy loss, is shown in Eq. (17):

$$\begin{aligned} L_{SE}= & {} \frac{1}{m}\sum _{i=1}^m MSE(F_i,B_i) \end{aligned}$$

(17)

$$\begin{aligned} L_T= & {} \frac{1}{2\times m}\left( \sum _{i=1}^m MSE(F_i,B_i)+\sum _{i=1}^m CE(P_i,T_i)\right) \end{aligned}$$

(18)

where $F_i$ represents the semantic features of modality i in the network, $B_i$ represents the true semantic embeddings, $P_i$ represents the model’s predicted results, $T_i$ represents the ground truth labels, $MSE(\cdot )$ denotes the mean squared error loss function, and $CE(\cdot )$ denotes the cross-entropy loss function.

After the training of the teacher network is completed, we utilize the knowledge of the teacher network to train the student network. The loss of the student network includes semantic loss, predicted cross-entropy loss, SGM knowledge distillation loss, and soft target knowledge distillation loss. Particularly, the soft target knowledge distillation loss focuses on the differences in similarity of classification score features and utilizes KL divergence for loss computation. The specific formula is as follows:

$$\begin{aligned} L_{ST}^S= & {} \frac{1}{m}\sum _{i=1}^m KL\left( \frac{P_i}{T},\frac{P_S}{T}\right) \end{aligned}$$

(19)

$$\begin{aligned} L_S= & {} \alpha L_{ST}^S+\beta L_{SGM}^S+\gamma L_{SE}^S+L_{CE}^S \end{aligned}$$

(20)

where $\alpha , \beta , \gamma$ are hyperparameters, we set $\alpha =0.1, \beta =1, \gamma =1$ and T is the temperature parameter in the KL divergence, $L_{ST}^S$ is the soft target loss of the student network, $L_{SGM}^S$ is the semantic graph mapping loss of the student network, $L_{SE}^S$ is the semantic loss of the student network, and $L_{CE}^S$ is the predicted cross-entropy loss of the student network. We train the student network with the total loss of the student network.

Source link