A dual-stream deep learning framework for skin cancer classification using histopathological-inherited and vision-based feature extraction

In this section, we introduce the proposed approach for skin lesion diagnosis (see Fig. 1), which uses a dual-stream framework combining histopathological-inherited and vision-based feature extraction. The methodology begins with Data Acquisition, where dermatoscopic images are collected and preprocessed to ensure high-quality input for the model. These images are then passed through a U-Net Model for precise lesion segmentation, generating predicted regions of interest (ROIs) that highlight the affected areas.

The segmented ROIs are processed in parallel by two streams: the embedded stream network using Virchow2 for histopathologically inherited feature extraction and the vision stream network using Nomic for vision-based feature extraction. The features extracted from both streams are fused and integrated to create a comprehensive representation of the lesion, which is then used for classification. The model is trained via a carefully designed loss function and evaluated via a set of performance metrics to ensure robust and accurate diagnosis.

Preprocessing of the dataset

The HAM10K dataset, consisting of 10,015 dermatoscopic images across seven common skin lesion types, was preprocessed to ensure consistency and compatibility with the dual-stream framework. The preprocessing pipeline included several key steps to enhance data quality and facilitate robust model training.

First, all images were resized to a uniform resolution of $224 \times 224$ pixels to standardize input dimensions while preserving critical details for feature extraction. This resizing step also ensured computational efficiency during training and inference. To address class imbalance, a combination of undersampling and oversampling techniques was applied, ensuring that each skin lesion type was adequately represented in the training set.

Next, data augmentation was performed to simulate real-world variations in lesion appearance and improve the model’s generalizability. Augmentation techniques included random rotations, horizontal and vertical flipping, scaling, and color jittering. These transformations introduced variability in lesion orientation, size, and color, mimicking the diversity observed in clinical settings.

The dataset was then split into training, validation, and testing subsets using a stratified approach to maintain proportional representation of each lesion type across all splits. Specifically, 70% of the dataset was allocated for training, 15% for validation, and the remaining 15% for testing. This partitioning ensures a robust evaluation of the model’s performance on unseen data, as described in Equation 1 where N represents the total number of images in the dataset.

$$\begin{aligned} N_{\text {train}} = 0.7 \times N, \quad N_{\text {val}} = 0.15 \times N, \quad N_{\text {test}} = 0.15 \times N \end{aligned}$$

(1)

Finally, pixel intensity normalization was applied to all images to standardize the range of input values. Each image’s pixel values were scaled to the range [0, 1], ensuring consistent input distributions for the deep learning models. This normalization step improved convergence speed during training and enhanced the stability of the model. By implementing these preprocessing steps, the dataset was optimized for use in the dual-stream framework, enabling accurate and efficient training while ensuring the model’s ability to generalize across diverse clinical scenarios.

Lesion segmentation using U-Net

Accurate lesion segmentation is a critical step in skin lesion diagnosis, as it enables precise localization of affected areas, which is essential for subsequent feature extraction and classification⁴⁶. This study employs the U-Net architecture for lesion segmentation because of its proven effectiveness in medical image analysis tasks.

U-Net is a convolutional neural network (CNN) designed explicitly for biomedical image segmentation and is characterized by its encoder-decoder structure with skip connections that facilitate the preservation of spatial information⁴⁷. The encoder captures contextual features through a series of convolutional and pooling layers. In contrast, the decoder reconstructs the spatial resolution of the feature maps to produce a pixel-wise segmentation mask⁴⁸.

The U-Net architecture is defined by the following key components: an encoder, a decoder, and a loss function. The encoder consists of a series of convolutional layers followed by max-pooling layers⁴⁹. Each convolutional layer applies a set of filters to the input image, extracting hierarchical features. The output of the i-th convolutional layer can be expressed as in Equation 2, where $F_i$ represents the feature maps at layer i, $W_i$ and $b_i$ are the learnable weights and biases, $*$ denotes the convolution operation, and $\sigma$ is the activation function (e.g., ReLU). The max pooling operation reduces the spatial dimensions of the feature maps, enabling the network to capture broader contextual information.

$$\begin{aligned} F_i = \sigma (W_i *F_{i-1} + b_i) \end{aligned}$$

(2)

The decoder pathway reconstructs the spatial resolution of the feature maps through upsampling and convolutional layers. The upsampling operation is followed by concatenation with the corresponding feature maps from the encoder pathway via skip connections⁵⁰. This process can be described as in Equation 3, where $F_j$ represents the feature maps at the j-th decoder layer, and $\text {Upsample}(\cdot )$ denotes the upsampling operation. The skip connections ensure that fine-grained details from the encoder are preserved, enabling precise localization of lesion boundaries.

$$\begin{aligned} F_j = \sigma (W_j *\text {Upsample}(F_{j+1}) + b_j) \end{aligned}$$

(3)

The utilized U-Net model (see Fig. 2) is trained via the Dice loss function, which is particularly suitable for segmentation tasks with imbalanced class distributions⁵¹. The Dice loss is defined as in Equation 4, where $y_p$ and $\hat{y}_p$ represent the ground truth and predicted segmentation masks, respectively, for pixel p. Dice loss minimizes the discrepancy between the predicted and ground truth masks, ensuring accurate lesion segmentation.

$$\begin{aligned} \mathcal {L}_{\text {Dice}} = 1 – 2 \times \frac{\sum _p y_p \times \hat{y}_p}{\sum _p y_p + \sum _p \hat{y}_p} \end{aligned}$$

(4)

One of the key advantages of U-Net is its reduced complexity compared with other segmentation models, such as DeepLab or Mask R-CNN. This reduction in complexity not only simplifies the training process but also reduces latency, making the model more suitable for real-time applications⁵². Furthermore, the lesions in the HAM10000 dataset are generally large enough to be effectively analyzed via a moderately complex model such as U-Net⁵³.

Employing an overly complex and advanced model for this task would be computationally inefficient and unnecessary, as the additional complexity would not yield significant improvements in segmentation accuracy. Instead, U-Net strikes an optimal balance between performance and computational efficiency, making it an ideal choice for this study.

The output of the U-Net model is a binary segmentation mask that precisely delineates the lesion region. This mask then isolates the lesion from the background, enabling focused feature extraction and classification in subsequent pipeline stages. By utilizing U-Net’s ability to capture fine-grained details while maintaining computational efficiency, this study ensures accurate and efficient lesion segmentation, laying the foundation for robust skin lesion diagnosis.

Dual-stream feature extraction

Recent advancements in transformer-based architectures, such as Vision Transformers (ViTs) and Swin Transformers^54,55,56, have demonstrated superior performance in capturing long-range dependencies within images, making them particularly well-suited for medical image analysis⁵⁷. In line with this, the proposed dual-stream framework utilizes transformer-based models for both feature extraction streams.

Specifically, Virchow2, a self-supervised Vision Transformer pretrained on 3.1 million whole-slide histopathology images, is employed to extract high-level histopathological features. Its architecture inherently excels at capturing intricate patterns and subtle morphological characteristics of skin lesions, enabling state-of-the-art performance in various computational pathology tasks^58,59. Similarly, Nomic, the second stream’s feature extractor, utilizes a vision-based transformer architecture to analyze spatial and contextual information from dermatoscopic images⁶⁰.

By integrating these transformer-based models, the framework ensures optimal extraction of both global and local features, addressing the reviewer’s recommendation to enhance the capture of subtle lesion characteristics. This design choice underscores the robustness of the proposed approach in overcoming the challenges associated with skin lesion diagnosis.

Virchow2 incorporates a multilayer perceptron (MLP) with gated linear unit (GLU) style gating and sigmoid linear unit (SiLU) activation functions. The GLU mechanism allows for dynamic feature selection, enhancing the model’s ability to focus on relevant patterns in the histopathological data. Register tokens are ignored during processing, ensuring that only meaningful patch and class tokens contribute to the final embeddings.

The embeddings are derived from both class tokens $\textbf{c}$ and patch tokens $\textbf{P}$, which are concatenated to form a comprehensive feature representation as in Equation 5, where $\textbf{c} \in \mathbb {R}^d$ is the class token, $\textbf{P}_i \in \mathbb {R}^d$ represents the i-th patch token, N is the number of patches, and $\Vert$ denotes concatenation. This ensures that the model captures both the global and local features critical for accurate diagnosis.

$$\begin{aligned} \textbf{E}_{\text {Virchow2}} = \left[ \textbf{c} \, \Vert \, \frac{1}{N} \times \sum _{i=1}^{N} \textbf{P}_i \right] \end{aligned}$$

(5)

The second track employs Nomic, a vision-based feature extraction model, to analyze spatial and contextual information from the same set of images. Nomic processes the images through a transformer-based architecture, extracting embeddings $\textbf{E}_{\text {Nomic}} \in \mathbb {R}^d$ that encode visual patterns such as texture, color, and structural irregularities. The Nomic model uses a Nomic Processor to preprocess the input images, followed by a transformer backbone that generates a last hidden state. The class token from this hidden state is extracted and normalized via L2 normalization, as defined in Equation 6, to ensure consistent scaling of the feature vectors where $\textbf{h}_{\text {cls}} \in \mathbb {R}^d$ represents the class token extracted from the last hidden state of the transformer.

$$\begin{aligned} \textbf{E}_{\text {Nomic}} = \frac{\textbf{h}_{\text {cls}}}{\Vert \textbf{h}_{\text {cls}}\Vert _2} \end{aligned}$$

(6)

Combining these embeddings with the histopathological-inherited features from Virchow2 provides the model with a more holistic understanding of the lesion characteristics, enhancing its diagnostic capabilities. The fusion process integrates global and local histopathological features with spatial and contextual visual patterns, enabling the model to achieve superior performance in skin lesion classification.

Feature fusion and integration

The features extracted from both tracks are integrated through a fusion mechanism that combines the strengths of histopathologically inherited and vision-based data. The embeddings from Virchow2 and Nomic are normalized and concatenated, ensuring that the model retains the discriminative power of both feature sets. The fusion process can be expressed as in Equation 7, where $\text {Norm}(\cdot )$ denotes L2 normalization.

$$\begin{aligned} \textbf{E}_{\text {fused}}= & \text {Norm}(\textbf{E}_{\text {Virchow2}}) \, \Vert \, \text {Norm}(\textbf{E}_{\text {Nomic}}) \end{aligned}$$

(7)

$$\begin{aligned} \text {Norm}(\textbf{x})= & \frac{\textbf{x}}{\Vert \textbf{x}\Vert _2} \end{aligned}$$

(8)

This fusion process is followed by a series of fully connected (FC) dense layers, which further refine the combined feature representation. Each dense block consists of an FC layer, a LeakyReLU activation function, and a dropout layer to prevent overfitting. Integrating these features is critical for capturing the complex interplay between morphological and visual characteristics, enabling the model to make more informed decisions during classification.

Classification model

The classification model is built upon a deep neural network architecture incorporating fused feature representations. The model consists of a multilayer perceptron (MLP) with dropout and batch normalization layers to prevent overfitting and improve generalizability. The output logits $\textbf{z}$ are computed via Equation 9, where $\textbf{W}_1, \textbf{W}_2$ are weight matrices, $\textbf{b}_1, \textbf{b}_2$ are bias terms, and ReLU is the activation function.

$$\begin{aligned} \textbf{z} = \textbf{W}_2 \times \text {ReLU}(\textbf{W}_1 \times \textbf{E}_{\text {fused}} + \textbf{b}_1) + \textbf{b}_2 \end{aligned}$$

(9)

The final probabilities $\textbf{p}$ are obtained via the Softmax function presented in Equation 10, where C is the number of classes.

$$\begin{aligned} p_i = \frac{\exp (z_i)}{\sum _{j=1}^C \exp (z_j)} \end{aligned}$$

(10)

The model is trained via a cross-entropy loss function, as expressed in Equation 11, where $y_i$ is the ground truth label. The inclusion of both histopathologically inherited and vision-based features allows the model to achieve high classification accuracy while maintaining robustness to variations in the input data.

$$\begin{aligned} \mathcal {L} = -\sum _{i=1}^C y_i \times \log (p_i) \end{aligned}$$

(11)

To ensure efficient training and convergence, the learning rate was dynamically adjusted using a ReduceLROnPlateau scheduler. This scheduler reduces the learning rate when the validation loss plateaus, as defined in Equation 12, where $\eta _t$ is the learning rate at epoch t, $\eta _{t-1}$ is the learning rate at the previous epoch, and $\alpha$ is the reduction factor applied when the validation loss stops improving.

$$\begin{aligned} \eta _t = {\left\{ \begin{array}{ll} \eta _{t-1} & \text {if } \mathcal {L}_{\text {val}}
(12)

This approach effectively balances computational efficiency and model performance, ensuring stable convergence without excessive computational overhead. While techniques such as Cosine Annealing or OneCycleLR can be potential alternatives for improving convergence speed and avoiding local minima, the current strategy already demonstrates robust performance across multiple trials. Nevertheless, future suggestions is to explore the integration of these advanced learning rate schedules to further optimize the model’s training dynamics and enhance its adaptability to complex datasets.

Model evaluation and validation

The performance of the proposed model is evaluated via a comprehensive set of metrics, including precision, recall, F1 score, accuracy, and specificity. These metrics are computed via weighted averaging techniques to account for class imbalance in the dataset⁶¹. For example, weighted precision $\text {precision}_{\text {weighted}}$ is calculated via Equation 13, where $w_i = \frac{n_i}{N}$ is the weight for class i, $n_i$ is the number of samples in class i, and N is the total number of samples.

$$\begin{aligned} \text {Precision}_{\text {weighted}} = \sum _{i=1}^C w_i \times \frac{\text {TP}_i}{\text {TP}_i + \text {FP}_i} \end{aligned}$$

(13)

Similarly, the weighted F1 score is computed via Equation 14, where $\text {precision}_i$ and $\text {recall}_i$ are the precision and recall for class i, respectively⁶².

$$\begin{aligned} \text {F1}_{\text {weighted}} = \sum _{i=1}^C w_i \times \frac{2 \times \text {Precision}_i \times \text {Recall}_i}{\text {Precision}_i + \text {Recall}_i} \end{aligned}$$

(14)

Weighted accuracy is defined as in Equation 15, where $\text {TN}_i$ and $\text {FN}_i$ are the true negatives and false negatives for class i, respectively.

$$\begin{aligned} \text {Accuracy}_{\text {weighted}} = \sum _{i=1}^C w_i \times \frac{\text {TP}_i + \text {TN}_i}{\text {TP}_i + \text {TN}_i + \text {FP}_i + \text {FN}_i} \end{aligned}$$

(15)

The model is validated through a series of experiments, including k-fold cross-validation and independent testing on unseen data. Additionally, dimensionality reduction techniques such as PCA, t-SNE, and UMAP are employed to visualize the feature embeddings, providing insights into the model’s ability to separate different classes in the latent space. For example, PCA reduces the dimensionality of the embeddings $\textbf{E}_{\text {fused}} \in \mathbb {R}^d$ to $\textbf{E}_{\text {PCA}} \in \mathbb {R}^2$ via Equation 16, where $\textbf{U}$ and $\Sigma$ are the eigenvectors and eigenvalues of the covariance matrix of $\textbf{E}_{\text {fused}}$.

$$\begin{aligned} \textbf{E}_{\text {PCA}} = \textbf{U} \times \Sigma \end{aligned}$$

(16)

The confusion matrix is used to analyze the model’s performance at the granular level, identifying potential areas for improvement. The results demonstrate that the dual-stream approach significantly enhances diagnostic accuracy compared to single-track models, making it a promising solution for detecting skin cancer.

To illustrate the feature extraction and fusion process, consider a skin lesion image $\textbf{I}$. Virchow2 first processes the image to extract histopathologically inherited features $\textbf{E}_{\text {Virchow2}}$, and Nomic is used to extract vision-based features $\textbf{E}_{\text {Nomic}}$. These features are normalized and concatenated to form the fused embedding $\textbf{E}_{\text {fused}}$. The fused embedding is then passed through the classification model to produce the final probabilities $\textbf{p}$, as shown in the following flow:

1.

Input Image: $\textbf{I}$
2.

Feature Extraction:
- Virchow2: $\textbf{E}_{\text {Virchow2}} = \left[ \textbf{c} \, \Vert \, \frac{1}{N} \times \sum _{i=1}^{N} \textbf{P}_i \right]$
- Nomic: $\textbf{E}_{\text {Nomic}}$
3.

Feature Fusion: $\textbf{E}_{\text {fused}} = \text {Norm}(\textbf{E}_{\text {Virchow2}}) \, \Vert \, \text {Norm}(\textbf{E}_{\text {Nomic}})$
4.

Classification: $\textbf{z} = \textbf{W}_2 \times \text {ReLU}(\textbf{W}_1 \times \textbf{E}_{\text {fused}} + \textbf{b}_1) + \textbf{b}_2$
5.

Output Probabilities: $p_i = \frac{\exp (z_i)}{\sum _{j=1}^C \exp (z_j)}$

The proposed framework pseudocode

The pseudocode for the proposed framework is outlined in Algorithm 1. The algorithm provides a step-by-step description of the dual-stream deep learning framework for skin lesion diagnosis, encompassing data preprocessing, feature extraction using Virchow2 and Nomic, feature fusion, model training, and performance evaluation. Each step is designed to ensure robustness, generalizability, and interpretability of the model while addressing challenges such as class imbalance and complex lesion characteristics. The modular structure of the pseudocode reflects the key components of the framework, enabling reproducibility and facilitating future enhancements.