Detecting common coccinellids found in sorghum using deep learning models

Deep learning approaches for object detection

The generic architecture of deep neural networks for object detection consists of two main components: a backbone, which is commonly a pre-trained CNN network used to generate feature maps, and a head, which is used to detect objects as bounding boxes defined by their coordinates (bounding box prediction) and to classify objects into one of several categories of interest²⁵, in our case, different types of coccinellids. One-stage detectors, including the YOLO family of detectors, have a dense prediction head that achieves the object detection and classification tasks simultaneously. Two-stage detectors, including the popular Faster R-CNN detector, decouple the object detection and classification tasks and achieve them in two stages. In the first stage, they use a dense prediction head to generate RoIs that may contain objects. In the second stage, a sparse detection head is used to classify the RoIs according to different object categories and to refine their bounding boxes. In recent years, it has become standard practice to insert a neck in between the backbone and the head of the network, to collect and mix features from different layers. The FPN network³¹ is one example of a neck that is commonly used in object detection networks. FPN uses a top-down path with lateral connections to extract semantic feature maps at different scales²⁵. The resulting feature maps enable the model to find objects at different scales. Path aggregation network (PANet)⁷⁵ is another example of a neck used in object detectors. It enhances FPN with a bottom-up path which helps propagate the low-level features. Equipped also with an adaptive feature pooling, PANet has been shown to improve object localization²⁵. The generic architecture of the one-stage and two-stage detectors is shown in Fig. 1. We study the popular Faster R-CNN as a representative two-stage approach and two YOLO variants, YOLOv5 and YOLOv7, on the task of detecting and classifying common coccinellid found in sorghum. All models studied were trained and evaluated using images annotated with the Labelbox tool (https://labelbox.com).

Faster R-CNN-FPN

Modern Faster R-CNN models use a pre-trained CNN as a backbone for feature extraction combined with an FPN network as a neck to obtain semantic feature maps at different scales. Extracted feature maps are provided as input to a region proposal network (RPN) which can be seen as the dense prediction head of the network. The RPN identifies Regions of Interest or RoIs (i.e., regions that may contain objects of interest – in our case, coccinellids) and their corresponding locations (i.e., rectangular bounding boxes parameterized using the box’s center coordinates, and its height and width). More precisely, the RPN uses a sliding window to generate three anchors with different aspect ratios (1:2, 1:1 and 2:1, respectively) at each grid cell in each input feature map. The anchors are labeled (as object/positive or background/negative) based on their overlap with ground truth bounding boxes and used to train the RPN network to identify RoIs and their locations. Highly overlapping regions, potentially corresponding to the same object, can be filtered using a non-maximum suppression (NMS) threshold. Subsequently, the resulting RoIs together with the feature maps are provided as input to a sparse prediction head which is trained to classify RoIs into several categories of interest (e.g., different coccinellid types) and refines their locations. All parameters of the network are trained together using a multi-task loss, which combines the cross-entropy classification loss with a linear regression L2 loss⁷⁶. We experiment with two CNN networks pre-trained on ImageNet⁷⁷ as the CNN backbone, specifically, ResNet-50 and ResNet-101 networks, given that they provide a good trade-off between accuracy and speed⁷⁴. The FPN produces features maps at 5 different scales, and consequently 5×3 anchors are generated at each location. Rezatofighi et al.⁷⁸ suggested that the standard L2 loss used to regress the parameters of the bounding box corresponding to an object is not strongly correlated with the IoU (Intersection over Union) metric generally used to evaluate object detection approaches. Instead of the L2 loss, they proposed to use a loss based on the IoU metric. Specifically, they experimented with a IoU loss and a loss based on a generalized IoU (GIoU), and showed that optimizing the GIoU loss helps improve the performance measured either using the GIoU itself or the standard IoU. Given this result, we experiment with the IoU and GIoU as the regression loss for the bounding box regression task in Faster R-CNN.

YOLOv5

As described above, two-stage detectors, such as Faster R-CNN-FPN network, re-purpose image classification to perform object detection by using the RPN to identify anchors that contain objects of interest as RoIs, and subsequently classifying the RoIs into specific categories. As opposed to that, one-stage detectors use directly the input image and to identify bounding box coordinates and class probabilities for objects of interest. YOLOv5 was released in 2020 by a company called Ultralytics²⁶ and has evolved over time. We used the latest YOLOv5 (v6.0/v6.1) architecture⁷⁹. The backbone for YOLOv5 is a New CSP-Darknet53 which combines the original Darknet53 network used in YOLOv3²⁴ with the CSPNet network⁸⁰. Darknet53 was inspired by the ResNet architecture and it was specifically designed for object detection. CSPNet addresses the issue of duplicate gradient information in large backbone networks by truncating the gradient flow to speed up computation. The current neck used in the YOLOv5 architecture consists of two components, SPPF and New CSP-PAN. SPPF is a variant of the Spatial Pyramid Pooling (SPP)⁸¹, which helps identify small objects and also objects at different scales. SPPF was designed to improve the computation speed of SPP. Similar to the Darknet53 backbone, the PAN network (PANet)⁷⁵ is also combined with CSP to improve computation speed. YOLOv5 uses a dense prediction head which is inherited from YOLOv3²⁴.

In addition to components that improve efficiency, YOLOv5 makes use of a variety of augmentation techniques on the input image. Among others, mosaic augmentation²⁵ is used to stitch together four images with the goal of training the model to find objects in places other than the center of the image, where a large majority of objects are generally located. Furthermore, YOLOv5 uses automatically generated anchors (with different scales and aspect ratios) to predict bounding boxes (and confidence scores) for each cell in a grid directly from the input image. The anchors are generated using k-means clustering based on the bounding boxes in the training set²³ and a genetic evolution algorithm that optimizes the initial k-means centroids based on the complete IoU (CIoU) loss⁸². The CIoU loss aggregates the overlap area, distance between center points, and aspect ratio consistency of two bounding boxes. The YOLOv5 head has 3 detection layers corresponding to three different scales and predicts bounding boxes with 3 different aspect ratios for each scale, resulting in a total of 9 anchors. The bounding boxes are predicted as deviations from the anchor dimensions. As in Faster R-CNN-FPN, the NMS technique is used to filter bounding boxes representing the same object. The whole network is trained using a multi-task loss, which combines classification loss (binary cross-entropy), objectness loss (binary cross-entropy) and location loss (CIoU). YOLOv5 uses an exponential moving average (EMA) of the model checkpoints as final detector. YOLOv5 itself represents a series of object detection models (compound-scaled variants of the same architecture) that have been pre-trained on the MS COCO dataset⁴⁹. Models in the YOLOv5 series have different sizes as applications have different needs in terms of the trade-off between accuracy and speed. In this study, we experiment with five YOLOv5 variants that vary in size from nano (YOLOv5n) to small (YOLOv5s), medium (YOLOv5m), large (YOLOv5l) and extra-large (YOLOv5x), whose specific architectures are available from the official GitHub repository²⁶ as .yaml files in the models directory.

YOLOv7

The YOLOv7 architecture has been designed based on the “bag-of-freebies” idea introduced by Bochkovskiy et al.²⁵, which refers to the fact that while it’s important for a detector to be fast at inference time, the training can be more expensive if it helps to improve the overall accuracy of the model (this is acceptable as the training is done offline). With this idea in mind, YOLOv7 introduced several innovations in network architecture and training strategies. One important innovation, the main component of YOLOv7’s architecture (used both in the backbone and neck/head networks), is a block called extended efficient layer aggregation network (E-ELAN). ELAN⁸³ uses a “stack in computational block” structure combined with CSP to optimize the shortest gradient path and ensure that scaling up the network does not result in deterioration of performance. In addition, E-ELAN uses “expand, shuffle, merge cardinality” which enables the model to learn more diverse features.

The neck of the network is structured based on a PAFPN network (a combination of PAN and FPN) which uses E-ELAN for feature extraction and fusion. In addition to the “lead head”, YOLOv7 introduces an “auxiliary head”, somewhere in the middle of the network, meant to assist the “lead head” (which may be too far down the network). Soft labels are assigned to the auxiliary and lead heads in a coarse-to-fine manner based on the predictions of the lead head and the ground truth. The coarse soft labels used by the auxiliary head represent a relaxed version of the fine soft labels used by the lead head, as it is expected that the auxiliary head is less precise. In terms of anchors, YOLOv7 leverages the automated anchor selection approach proposed in YOLOv5²⁶ and uses 3 aspect ratios for each of the 3 features maps representing three different scales (for a total of 9 anchors). Furthermore, data augmentation techniques similar to those used in YOLOv5 are also used in YOLOv7 (including mosaic augmentation), and the commonly-used NMS technique is employed to filter out the predicted bounding boxes. To achieve robustness through module-level re-parameterization (i.e., aggregating the weights of a multi-branch module during inference), YOLOv7 uses gradient flow propagation paths to “plan” what modules can benefit from re-parameterization.

YOLOv7 is trained from scratch on the COCO dataset using a multi-task learning loss consisting of the classification loss (binary cross-entropy), objectness loss (binary cross-entropy) and location loss (CIoU). If both the lead and the auxiliary heads are used, the loss includes similar components corresponding to the two heads, with different weights. The final model used for inference is based on an EMA of the model parameters at different checkpoints during training. Together, the innovations introduced in YOLOv7 and the components reused from prior works have led to state-of-the-art results on standard benchmark datasets for object detection²⁸, and at the same time, smaller inference time as compared with other YOLO models. YOLOv7 also consist of a series of pre-trained models of various sizes. In this study, we experiment with four YOLOv7 variants: 1) the standard YOLOv7 model designed for standard GPUs and 2) its compound scaling variant, YOLOv7-x; 3) the smallest model, YOLOv7-tiny deigned for edge GPU; and 4) a larger model, YOLOv7-d6, a cloud GPU architecture.

Dataset

Coccinellid imagery downloaded from the iNaturalist web portal was used (inaturalist.org). iNaturalist is a citizen science project that allows naturalists to upload and share observations (i.e., images) of biodiversity worldwide through a web platform and mobile app for free. Submission by observers include the actual images, their locations, observed time, and group identifications. Agreements on the taxa in the observations create a “research-grade” label that is assigned to the observation. iNaturalist makes an archive of research-grade observation data available to the environmental science community via the Global Biodiversity Information Facility (GBIF)⁸⁴. We used GBIF to assemble a dataset for training and testing deep learning models for the detection, localization and classification of coccinellids. Only research-grade labels at family, genus, and species level were considered in the dataset that we assembled. The dataset includes seven distinct categories of coccinellids corresponding to the most important coccinellids found in sorghum plants, specifically: Coccinella septempunctata, Coleomegilla maculata, Cycloneda sanguinea, Harmonia axyridis, Hippodamia convergens, Olla v-nigrum and the subfamily Scymninae¹⁷. Three sample images in each of these seven categories are shown in Fig. 2 (where each row corresponds to one coccinellid type).

We aimed to select approximately 700 images per category, and assembled a dataset with a total number of 4,865 images. Each image contains one or more instances of coccinellids of the same type (i.e., corresponding to a particular category). The dataset was split into train (3,053 images), development or dev (1,113) and test (699) subsets. Table 1 shows the distribution of the images/coccinellid instances over the seven categories in each of the train/dev/test subsets and also in the whole dataset. Supplementary Table S1 shows the distribution of the images with respect to the number of instances per image (1, 2, 3, 4, 5, 6, 7, or 8) for each category and for each of the the train/dev/test subsets. As can be seen, most images have only one or two coccinellid instances, although there are some images that have up to 8 coccinellid instances.

We also classified the instances in our dataset according to their size, as this information is frequently used when evaluating object detection approaches⁸⁵. Specifically, instances are classified based on the area that they occupy in an image as small (\(area \le 32^2\)), medium (\(32^2 < area\le 96^2\)) or large (\(area>96^2\))⁸⁵. Supplementary Table S2 shows the distribution of the small, medium and large instances in the train/dev/test subsets and in the whole dataset. As can be seen, the number of small instances is just 5 in the total dataset and they are all included in the training subset. The number of medium instance is 142, with 18 of those instances being in the test subset. The remaining 4995 instances are large and represent the majority in our dataset. Given this observation, our evaluation metrics will be generic (representing mostly the large category) as opposed to being specifically focused on small, medium and large categories, respectively.

Implementation details

We trained and evaluated Faster R-CNN-FPN, YOLOv5 and YOLOv7 models. The data flow diagram for our whole process is depicted in Fig. 3. Training images are used to train the model, while the development images are used to evaluate and select hyperparameters. The performance of the final models is estimated on the test data.

Faster R-CNN-FPN

We used the Detectron2 library for object detection and segmentation⁸⁵ (developed by Facebook Research) to train and evaluate our Faster R-CNN-FPN models for coccinellids detection. Detectron2 uses PyTorch as a deep learning framework and is the successor of Detectron⁸⁶, originally “designed to be flexible in order to support rapid implementation and evaluation of novel research.” The default configuration of the Base-R-CNN-FPN is defined in the “Base-R-CNN-FPN.yaml” file at https://github.com/facebookresearch/detectron2/blob/main/configs/Base-RCNN-FPN.yaml. Hyper-parameters for Detectron2 are available at: https://github.com/facebookresearch/detectron2/blob/main/detectron2/config/defaults.py. The train/test files that we adapted are available https://github.com/cwang16/Detecting-Coccinellids. When training the models, we used the default values for most of the hyper-parameters of Detectron2’s Faster R-CNN-FPN object detection models and experimented with ResNet50/ResNet101 as backbone CNNs and IoU/GIoU as the regression losses, respectively. In terms of number of iterations, we started by training the Faster R-CNN-FPN model with the default 40,000 iterations. However, the learning curves shown in Supplementary Fig. S1

suggested that both training and development losses were still decreasing, while the AP was still increasing after 40,000 iterations. Thus, to enable the network to learn further and give better performance, we ran all experiments for 400,000 iterations, and identified the best iteration for each model based on learning curves as shown in Supplementary Fig. S2. Specifically, the best iteration is identified as the point where the validation loss starts increasing while the training loss is still decreasing. Using this criterion, for Faster R-CNN-FPN with ResNet-50 the best validation/training point was observed around iteration 80,000 for both IoU and GIoU losses, while for Faster R-CNN-FPN with ResNet-101 the best point was observed around iteration 220,000 for both losses.

YOLOv5

We used the official PyTorch YOLOv5 implementation provided by Ultralytics²⁶, which is available at https://github.com/ultralytics/yolov5, to train and evaluate the YOLOv5 models. We used the pre-defined model configurations for the YOLOv5 variants used in the experiments, where the number of classes was changed to 7. The pre-defined configurations are available at https://github.com/ultralytics/yolov5/tree/master/models, where the files “yolov5n.yaml”, “yolov5s.yaml”, “yolov5m.yaml”, “yolov5l.yaml”, “yolov5x.yaml”, correspond to the five YOLOv5 variants used in our study, respectively. In terms of hyper-parameters, YOLOv5 provides three different hyper-parameter settings, specifically, “hyp.scratch-low.yaml”, “hyp.scratch-med.yaml” “hyp.scratch-high.yaml”, to train smaller, medium and larger size models, respectively. These three hyper-parameter settings can be found at https://github.com/ultralytics/yolov5/tree/master/data/hyps. When training the models for YOLOv5, we used the default values for most of the hyperparameters, except for using 500 epochs and a batch size of 4. The best model identified by the YOLOv5 framework using the validation data was used for evaluation. As an example, the specific command line used to train the YOLOv5s model is shown below:

where the modified files and non-default hyper-parameters are highlighted in red font. Similarly, the command lined used to evaluate the YOLOv5s model is:

YOLOv7

We also used the official YOLOv7 PyTorch implementation²⁸ at https://github.com/WongKinYiu/yolov7 to train and evaluate the YOLOv7 models. As for YOLOv5, we used the pre-defined model configurations for the YOLOv7 variants used in the experiments, where the number of classes was changed to 7. The pre-defined configurations are available at https://github.com/WongKinYiu/yolov7/tree/main/cfg/training, where the files “yolov7.yaml”, “yolov7-tiny.yaml”, “yolov7x.yaml”, “yolov7-d6.yaml”, correspond to the four YOLOv7 variants used in our study, respectively. In terms of hyper-parameters, YOLOv7 provides three different hyper-parameter settings for: 1) edge GPU architectures (YOLOv7-tiny); 2) standard GPU architectures (YOLOv7 and YOLOv7-x); and 3) cloud GPU architectures (including YOLOv7-d6). These three hyper-parameter settings can be found at https://github.com/WongKinYiu/yolov7/tree/main/data, where “hyp.scratch.tiny.yaml” is the setting for YOLOv7-tiny, “hyp.scratch.p5.yaml” is the setting for YOLOv7 and YOLOv7-x, and “hyp.scratch.p6.yaml” is the setting for YOLOv7-d6.

When training the models for YOLOv7, we used the default values for most of the hyperparameters, except for using 500 epochs and a batch size of 2. The best model identified by the YOLOv7 framework using the validation data was used for evaluation. As an example, the specific command line used to train the YOLOv7 model is shown below:

where the modified files and non-default hyper-parameters are highlighted in red font. Similarly, the command lined used to evaluate the YOLOv7 model is:

To ensure reproducibility, we make available our train/test/dev data (in the form of iNaturalist image IDs), annotation, model configurations, and best trained models at https://github.com/cwang16/Detecting-Coccinellids.

All the models were trained on Amazon Web Services (AWS) p2.xlarge instances. According to AWS, the configuration of the p2.xlarge instance is as follows: 1 GPU, 4 vCPUs, 61 GiB of memory, and high network bandwidth. Training of the models for the specified number of iterations/epochs took between 8 and 14 days.

Evaluation metrics

We used three standard average precision metrics⁸⁷ to evaluate the results of our models. The three metrics are defined using the Intersection-over-Union (IoU) measure, which captures the overlap between the predicted bounding box of an instance and the ground truth bounding box of that instance. Specifically, the IoU is defined as the area of overlap (i.e., intersection) divided by the area represented by the union. The metrics used to evaluate the ability of the models to correctly identify the type of coccinellid are: 1) average precision at \(IoU=0.50\), denoted by AP@0.50; 2) average precision at \(IoU=0.75\), denoted by AP@0.75; and 3) average precision at \(IoU=.50:.05:.95\), which represents the average precision across ten IoU thresholds varying from 0.5 to 0.95 with a step size of 0.05, denoted by AP@.50 : 0.05 : .95 or simply AP. AP@n considers a prediction to be correct if the IoU between the detected instance and the ground truth instance annotation is greater or equal than n. For example, AP@0.50 considers a prediction to be correct correct if the corresponding IoU is greater or equal to 0.50.

In addition to comparing the models in terms of average precision metrics, we also compared the models in terms of number of layers that the specific model architecture used includes, number of parameters of the network, size of the model (MB) and inference time per image (ms). These characteristics can be used to identify small models that are accurate and fast and can be embedded in mobile devices for automated field scouting.

Ethics statement

The results presented are based solely on experiments with image data. The experiments do not involve live vertebrates and/or higher invertebrates. All experiments were carried out in accordance with relevant guidelines and regulations.