Robust multiclass classification of crop leaf diseases using hybrid deep learning and Grad-CAM interpretability

In this research, plant disease detection and identification are performed based on banana, cherry, and tomato leaf images. Various pre-trained convolutional and transformer models have been evaluated to compare the performance of the methods, which include: EfficientNetV2, ConvNeXt, Swin Transformer, and ViT. In addition to these existing architectures, we benchmarked a proposed Hybrid ConvNet-ViT model that combines ConvNets and vision transformer layers. The objective is to evaluate the performance of these models in terms of their ability to detect a range of leaf disease states. The subsequent sections provide a detailed description of the model architectures, implementation framework, evaluation, and performance analysis.

Materials

A publicly available dataset has been utilized in this study to evaluate the effectiveness of both pre-trained and proposed deep learning models for plant leaf disease detection and classification³⁰. The dataset comprises images of banana, cherry, and tomato plant leaves, encompassing a variety of healthy and diseased conditions. To ensure the systematic comparison and fair performance assessment, the data set is tripartite into three parts, which are used for training (70%), validation (15%), and test (15%) in Table 2. This split enables iterative model optimization while in development and a proper, fair, and objective evaluation on unseen data. The dataset in this study consists of labeled images representing banana, cherry, and tomato leaves, and it is associated with nine classes, among which eight classes are for eight diseases and healthy status, and one class is for healthy cases. Classes: BH, BU, CH, and CPM. Five categories are planted, placed, Tomato Septoria Leaf Spot (TSL), Tomato Spider Mites (TSM), Tomato Target Spot (TTS), Tomato Mosaic Virus (TTM), and Tomato Yellow Leaf Curl Vera Virus (TTY), respectively. The dataset was partitioned into three subsets for experimentation: 70% for training, 15% for validation, and 15% for testing. For example, the TTY class contains the most significant samples with 3,750 images for training, 804 for validation, and 804 for testing. The BH class has 109 training images and 23 each for validation and testing. This balanced and diverse distribution of classes ensures that the model can learn to differentiate between a wide range of leaf conditions across different crop types, enhancing the generalization capability of the disease detection system. Figure 1 depicts the sample dataset of leaf images of all diseases. (Fig. 2).

Preprocessing technique

Contemporary DL architectures like EfficientNetV2, ConvNeXt, Swin Transformer, Vision Transformer (ViT), and hybrid ConvNet-ViT models may each have different requirements on the input image size³¹. EfficientNetV2 and ConvNeXt are usually applied to 224 × 224or 384 × 384sized images, while transformer-based models (ViT and Swin) require fixed-size patches, and they also gain their success with the help of fixed-size patches. As our dataset has images at different resolutions, we needed a consistent and comparable preprocessing pipeline across those models for consistency in experimentation. We employed a unified image preprocessing and augmentation pipeline to provide consistent training across models with varying input size requirements, such as EfficientNetV2, ConvNeXt, Swin Transformer, ViT, and hybrid ConvNet-ViT architectures. The preprocessing begins by resizing all images to a common base resolution \(\:{H}_{target}\times\:{W}_{target}\), typically 224 × 224 or 384 × 384 pixels, depending on the model requirements. Use bilinear interpolation, and the following Eq. (1) is represented,

Following resizing, data augmentation helps improve generalization. This typically includes random cropping, where a region \(\:(h,\:w)\) is randomly selected from the resized image and scaled back to \(\:{H}_{target}\times\:{W}_{target}\) and it is written in Eq. (2),

Random horizontal flipping (Eq. (3)) is applied with a probability \(\:p\), usually 0.5, to simulate mirror-image variations,

For rotational invariance, apply random rotation by an angle θ within a fixed range (e.g., −15° to + 15°).

After augmentation, pixel values are normalized on a per-channel basis using the mean and standard deviation from the dataset,

For transformer-based models such as ViT and Swin Transformer, which operate on fixed-size patch embeddings (16 × 16), the input dimensions must be divisible by the patch size . Given input size \(\:{H}_{target}\times\:{W}_{target}\), the number of patches is,

We used several data augmentation strategies to improve the generalization properties and avoid overfitting. Random cropping: We randomly crop a portion from the image and then scale it to the target resolution, which adds spatial variance. Random horizontal flipping (with a probability of 0.5) is used to observe reflection-invariant training features. Moreover, random rotation in the range of ± 15° makes the model more robust to orientation variance, which is also essential since, in general, objects are positioned differently. All these modifications have a realistic effect on image acquisition and object placement. Images are then transformed to have the type casting to float and re-scaled per-channel by the ImageNet mean and standard deviation. This will normalise our pixel values to a consistent range and distribution, and in turn, help our training to converge far quicker and also enable us to utilize pre-trained weights. The normalization can be seen from Eq. (5). For transformer-based architectures like ViT and Swin Transformer, the images are processed as a sequence of fixed-sized patches (16 × 16), and the documents are a sequence of words. To accommodate this, the resized image dimensions must be divisible by the patch size . The total number of non-overlapping patches is determined using Eq. (6). Ensuring that \(\:{H}_{target}\) and \(\:{W}_{target}\) are divisible by \(\:P\) enables efficient and accurate patch embedding, which is critical for the self-attention mechanisms in these models. Finally, all pre-processed images are batched and arranged into a tensor format of \(\:(B,C,H,W)\) where \(\:B\) is the batch size, \(\:C\) the number of channels, \(\:H,\:W\) are the spatial dimensions. This unified format ensures compatibility across all selected architectures and supports efficient loading and training.

Methods

This study investigated the five state-of-the-art deep learning architectures for detecting diseases in tomato leaves: EfficientNetV2, ConvNeXt, Swin Transformer, ViT, and a Hybrid ConvNet-ViT model. EfficientNetV2 and ConvNeXt are strong ConvNets tailored for efficiency and accuracy. In contrast, Swin Transformer and ViT are two other transformer-based models designed for capturing global contextual information, which can benefit the multi-level image classification work. Hybrid ConvNet-ViT uses the strengths of both ConvNets and transformers, i.e., local feature extraction by convolution layers and global attention of transformers. We believe these multiple models from different backgrounds provide a wide coverage for assessing various architectural strengths against the tasks of tomato disease classification.

EfficientNetV2

EfficientNetV2, an updated version of EfficientNet, achieves faster training and better performance owing to the fused-MBConv and better progressive learning, and is suitable for image-based plant disease detection tasks³². The present work, EfficientNetV2 was used to classify banana, cherry, and tomato plant leaf images into infected and healthy categories. The model was chosen because it can achieve high accuracy at low computational cost and is produced by compound scaling with depth and resolution. Despite the diverse symptoms and textures of the leaves between different plant species, efficient feature extraction of EfficientNetV2 contributed to the accurate discrimination of the disease patterns, namely blight, spot, and mosaic. The pre-trained ImageNet model was fine-tuned with the leaf dataset using transfer learning. The input images were rescaled to 224 × 224 pixels to correspond to the input requirements of the model. Strong data augmentation was also applied in random rotation, flipping, and colour jittering to improve generalization. The model performed well in all plant types, with a high classification accuracy rate and many false positives or negatives effectively reduced. The architecture of the samples of EfficientNetV2 is shown in Fig. 3.

ConvNeXt

ConvNeXt is a modernized ConvNet that incorporates architectural innovations inspired by vision transformers while retaining the efficiency and inductive biases of traditional ConvNets. This study used ConvNeXt to classify plant leaf diseases across banana, cherry, and tomato crops³³. The model’s hierarchical structure, large kernel sizes, and deep-stage design enable it to effectively capture complex texture patterns and local irregularities associated with various leaf diseases such as bacterial spots, early blight, and mosaic virus. Unlike standard ConvNets, ConvNeXt benefits from Layer Norm instead of BatchNorm, GELU activation, and pacified stem layers, which help improve stability and generalization. Input images were resized to 224 × 224 pixels, normalized, and augmented using standard techniques to improve robustness against background noise, lighting variations, and leaf orientation. The model was pretrained on ImageNet and fine-tuned on the plant disease dataset using transfer learning. ConvNeXt demonstrated strong performance in extracting high-level spatial features from plant leaves and achieved high classification accuracy with efficient convergence during training. Figure 4 depicts the ConvNeXt architecture.

Swin transformer

The Swin Transformer, a hierarchical vision transformer with shifted windows, has recently shown promise in plant leaf disease detection due to its ability to efficiently capture local and global features, as shown in Fig. 5. The Swin Transformer is fine-tuned in this application using a plant disease dataset comprising high-resolution leaf images from multiple plant species affected by various diseases³⁴. The model architecture typically includes four stages of transformer blocks with window sizes 7 × 7, patch size 4 × 4, and embedding dimensions starting from 96 and doubling at each stage (96, 192, 384, 768). It utilizes multi-head self-attention with 3, 6, 12, and 24 heads across the stages, ensuring deep feature extraction. During training, the model is optimized using the AdamW optimizer with a learning rate 0.0001 and weight decay of 0.05 over 100 epochs. Evaluation metrics include accuracy, precision, recall, and F1-score, with reported classification accuracy reaching up to 98.7%, outperforming traditional CONVNETs and ResNet-based methods. This performance highlights the Swin Transformer’s effectiveness in addressing complex background interference, varied lighting, and inter-class similarity in plant disease images.

ViT

Another practical deep learning approach to plant leaf disease detection is the ViT model, which exploits the self-attention mechanism to capture long-range dependencies in the leaf images³⁵. The ViT model usually takes input images and divides them into non-overlapping patches, followed by flattening these patches to 1D token embeddings. For a basic ViT-B model, the input images are partitioned into 16 × 16 patches for a sequence of 196 tokens (given an input image of dimension 224 × 224 and a dimension D = 768 for both the input sequence and the embedding. These token embeddings are combined with position embeddings to retain spatial information and processed through multiple transformer encoder blocks, often configured with 12 layers (L), 12 self-attention heads (H), and a feed-forward dimension of 3072. The model also includes a classification token (CLS) aggregating global information for final disease prediction. ViT is commonly optimized during training using the AdamW optimizer with a learning rate around 1e-4, weight decay of 0.01, and a cosine learning rate scheduler. Fine-tuning on plant disease datasets like PlantVillage, with aggressive data augmentation strategies such as random crop, flip, and colour jitter, can significantly boost its ability to distinguish subtle disease symptoms, resulting in state-of-the-art accuracy for plant pathology classification. Figure 6 represents the sample architecture of ViT for plant leaf disease detection.

Hybrid ConvNet-ViT

In this study, we propose a hybrid architecture that integrates ConvNets with Vision Transformers (ViT) to enhance plant leaf disease detection for crops such as banana, cherry, and tomato. The motivation for this hybrid model arises from the complementary strengths of ConvNets and ViTs ConvNets excel at capturing local spatial hierarchies and textures. At the same time, ViTs are proficient at modelling long-range dependencies and global relationships across the image. The proposed model begins with a convolutional stem, consisting of several Conv-BatchNorm-ReLU blocks, which extract low-level and mid-level spatial features from input images. These features are then flattened and embedded into token sequences suitable for processing by the transformer encoder. A positional embedding is added to preserve spatial structure, and the ViT encoder applies multi-head self-attention (MHSA) to model global interactions across the entire leaf surface. A classification head, comprising fully connected layers with dropout regularization, is used to predict the presence and type of disease. The hybrid architecture leverages the locality bias and inductive priors of ConvNets in early layers while benefiting from the global contextual understanding provided by transformers in deeper layers.

This synergistic design allows the model to detect subtle disease symptoms, such as blight spots, leaf curls, and colour degradation, even under complex background conditions, as shown in Fig. 7. The core novelty of the proposed Hybrid ConvNet-ViT model stems from its architectural fusion of ConvNets and ViTs, explicitly tailored for fine-grained visual classification in plant leaf disease detection. Most existing models fall into two distinct categories: ConvNet-based models, which are excellent at learning local patterns such as edges and textures but struggle to capture long-range dependencies; and transformer-based models, which capture global interactions well but require large-scale datasets and lack inductive biases for spatial hierarchies. The proposed model fills in this gap by using a two-stage processing pipeline, which uses convolutional layers to extract features of localized leaf symptoms, and the transformer encoder further improves this representation by carrying out long-range interactions between spatially separate regions of the leaf surface. The mechanism for transitioning between the convolution and transformer layers is a critical contribution. Rather than directly feeding in image patches to the transformer, the model derives dense convolutional feature maps that preserve the spatial arrangement of disease patterns. These maps are subsequently tokenized into an embedding sequence enriched with semantic information so that the transformer can self-attend to more semantically meaningful representations. This design makes the model more applicable to detecting distributed and irregular signals, which are prevalent in practical plant pathology but poorly addressed by the isolated ConvNet or ViT. Furthermore, the hybrid model is developed with lightweight structures and modularity, which can be easily transferred to other datasets and scales. Transfer learning can also be applied where the ConvNet backbone is fine-tuned from a pre-trained model on an extensive general image dataset and then refined and re-trained with plant disease images using a small or imbalanced plant disease dataset. By combining the two, the model can retain the inductive bias induced by ConvNet and the global attention of ViT, thereby striking a balance among effectiveness, efficiency, and transferability. The new architecture demonstrates utility across multiple crops: banana, cherry, and tomato, without requiring species-specific retraining aspect in real-world applications. It also shows better robustness to background noise, leaf distortion, and light variation, meeting a significant challenge in agricultural images. This hybrid model brings a new direction for plant disease detection systems by leveraging the advantages of two dominant vision architectures and achieving efficiency and scalability with a unified solution.

The proposed algorithm describes the structure and logic of a ConvNet-Transformer Hybrid Model for classifying plant leaf images from banana, cherry, and tomato crops. First, a ConvNet backbone takes an image and outputs rich spatial feature maps. These feature maps are then flattened and projected linearly to patch embeddings for the transformer module. Positional encodings are added to capture spatial information between patches. The output feature representations are then fed through a stack of transformer encoder layers, each consisting of a multi-head self-attention (MHSA) mechanism and a feedforward network (FFN) with normalization and residual connections after each sub-layer. A global average pooling operation is then performed over the transformer layers’ output to create a single vector representation. Such a pooled vector is then sent to a fully connected layer, and SoftMax is applied to generate the final class probabilities (the possibility of the image being one of the three classes). This hybrid model uses the strengths of ConvNets’ local feature extraction and transformers’ global dependency modelling.

The designed Hybrid ConvNet-ViT architecture is intended to use both ConvNets and transformer-based architectures to leverage the plant leaf disease detection performance, as shown in Table 3. Although models like EfficientNetV2 and ConvNeXt are particularly good at extracting local patterns and texture by the convolutional operation, they may not be good at modelling global contextual relationships necessary to find complex or spread-out disease symptoms. Conversely, transformer-based architectures like ViT and Swin Transformer are strong at modelling long-range dependencies and international patterns. However, they perform poorly in searching fine-grained local patterns because they lack strong inductive biases. The Hybrid model addresses these limitations by incorporating ConvNet layers for spatial feature extraction, and then introducing transformer encoder layers to model the spatial features using self-attention to learn the contextual dependencies across the leaf surface. This cascaded fusion mechanism brings advantages in feature representation from different scales. It improves the model’s generalization, ensuring robustness in various complex visual situations commonly existing in real plant disease classification applications.

Experimental setup

The experiments were conducted in a Windows 11 operating environment, using a system with 16 GB of RAM and 1 TB SSD. The model was prototyped and evaluated in Python with PyTorch, NumPy, scikit-learn, OpenCV, and Matplotlib for implementation, data preprocessing, and visualization. AdamW optimizer was employed during training to add weight decay to help with generalization. A learning rate of 0.001 was adopted, with a batch size of 64, and the model was trained over 50 epochs. Data augmentation methods were used to improve model resilience and generalisation where necessary. The same fixed random seed was used to make sure results are reproducible. The experimental design of the proposed study is shown in Table 4.