Images dataset and CNN model training
Convolutional Neural Networks (CNNs) were trained using the 6766 images dataset collected for the fruit maturity stage study38. The image data were collected from 54 points of each four fields to cover the whole area with significant clonal variation in each wild blueberry field46,47. This was done to ensure that the dataset captured the range of variation in the wild blueberries across the fields to train a more robust model. Empirical studies have demonstrated that the optimal approach involves allocating 20–30% of the data for testing purposes while utilizing the remaining 70–80% for training the model48. In this study, 70% of the images (4736) were used for model training, 15% (1015) for validation, and 15% (1015) for testing to prevent the model from overfitting and accurately evaluating.
The collected raw image data set size was 1280 × 720. To find the best resolution size to get maximum model accuracy, the YOLOv4-tiny model was trained at three different image resolution sizes (i) 416 × 416, (ii) 608 × 608, and (iii) 832 × 832 to figure out at what images size model provide the maximum mean average precision (mAP). The model was trained to a batch size of 64 with subdivisions of 16. The maximum batch taken was 4000. Steps taken for the training were 3200 and 3600, respectively (80% and 90% of the maximum batch). The YOLOv4-tiny model was trained on Google Colab GPU, and VRAM usage was about 1.87 GB at image size 832 × 832. Ultimately, the model weight file was converted into the MyriadX Binary Large Object format (BLOB) to optimize the best interference with the visual processing unit and allow the depth AI module to use a customized trained model.
Experiment field
The developed system was evaluated in the McCully field (45.42468827° N, 63.473859° W), Debert, Nova Scotia, shown in Fig. 1. Among the total of one hundred twenty location points, 60 points were used for static measurement and the other 60 for the dynamic auto header height adjustment system scenario. The study field area was rectangular, and the wild blueberry harvester header was attached to the left side of the tractor. For optimum picking performance, harvesting each side of the field is preferred while traveling in a straight line. The data point locations were marked using a true random number technique to determine field plots49 using an online random number generator. At each field-side after the first random point from the corner, the next points were marked at a fixed 20-m distance until reaching the next field corner. This procedure was repeated at each corner to ensure that the sample represented the overall field to avoid bias. The automatic header height positioning system was evaluated under two different operating conditions (i) Static mode: the automatic system was used to adjust the header height while the harvester was stationary at a marked field location by switching from manual to auto system (ii) Dynamic mode, the automatic system was used continuously during harvesting without any manual intervention, and measurements were taken when the harvester reached each marked location in the field.
Boundary map of the experimental field and data collection points distribution, Image source: Esri, Maxar, Earthstar geographics, and the GIS use community.
In Fig. 1, the black dots are the locations where the operator’s manual harvester header height adjustment was recorded, and the static mode of the automatic height adjustment system was evaluated. The red flags show locations, where the dynamic scenario of harvesting the automated header height adjustment system was assessed.
Development of a control system
For the automatic header picking height adjustment based on the berry height, a machine vision system using the OAK-D camera module and deep learning was integrated into the developed harvester head controller14. After modification, the already-developed controller worked as a central controller in the new system. The new system featured a closed loop feed control system, as explained in block diagram Fig. 2, consisting of one Luxonis Spatial DepthAI OpenCV module OAK-D, a host computer (ThinkPad L14 with 16 GB RAM), a Linear potentiometer (LPPS-22 series), header height positioning central controller, and one Arduino UNO for serial communication as shown in Fig. 3.
Communication block diagram of automatic harvester header height positioning system.
Control functions of custom build harvester header control box.
The central control takes the continuous values in integer format from the host computer through serial communication and from the linear potentiometer to match both readings. The three-position momentary rocker switch moves the actuator up and down to compensate for the difference. Suppose the current height is less than the recommended height by the host computer; in that case, the controller sends the signal to the mechanical relay to change the header position and moves up to the actuator stroke using the hydraulic power control and vice versa. A customized spatial detection Python library (https://github.com/luxonis/depthai) of DepthAI was continuously run in the host computer to calculate the wild blueberry height, and a Pyfirmata library was included in the Python code to send the calculated header-picking height signal to the controller through the Arduino UNO to adjust the header position.
The OAK-D Module is driven by the Intel Myriad X VPU (vision processing unit) and powered by USB type-C from the host computer. Its baseboard comprises the two 720 pixels/120 Hertz global shutters in synchronized stereo pairs and one autofocus color 12 MP/4 K rolling shutter camera. The two mono cameras are used to create a disparity map which can be used to calculate the distance, and the color camera is for digital zoom, better color representation, and texture information. The Intel Movidius Myriad X technology makes it possible to run the deep neural network on the device and output data to the host device via USB 3.1.
The OAK-D was set up in front of the harvester header by attaching an extra assembly bar with the height adjustable mount as shown in Fig. 4. The camera position is set downward to detect the wild blueberries from the plants. The Lenovo ThinkPad L14 with 16 GB RAM and Windows 10 was set up as a host computer within the tractor cabin. The laptop directly powered the camera via a USB port and connected using a specific 5-m-long USB 3.1 cable.
The left side is the wild blueberry harvester with a camera mount in front of the header, and the right side of the images shows how the camera to berry height (y) is used to calculate berry height from the ground (z) using Python code.
To get the position status of the actuator in real-time, a 25.4 cm linear potentiometer (LPPS-22) was installed with the actuator on the back side of the harvester header as a feedback sensor. The potentiometer is precise and has self-aligning swivel rod eyes on both ends, which makes it very easy to connect with the actuator without any significant modification. This sensor inner rod was 5.08 cm longer than the original actuator stroke length of the harvester, which was taken as a base value and considered in all the calculations. This feedback sensor was directly attached to the central control to give the value of the header position.
Auto berry height measurement
Measuring the accurate blueberry height from the ground is essential for positioning the header. For this purpose, the spatial detection Python code was customized by adding mathematical Eqs. (1) and (2) to filter the height data while removing the noise in data, the berry height calculation formula, and the header height corrections pipelines. In the field on the go, the camera module detects many blueberries in the camera frame of view (FOV) range, but the header positioning system needs a single value of the berry distance from the ground. The rolling average of the nine berries’ distance from the camera was used to calculate the final average berry height from the ground using the following Eq. (1), as explained in Fig. 4.
$${\text{z}} = {\text{ x}} + {\text{ y}} – {\text{h,}}$$
(1)
where ‘h’ is the height correction value obtained from Eq. (2).
When the tractor turns or due to sudden bumps in the field, the camera detects berries that are not directly under the camera. These values are high to the distance value fixed for the camera to the ground, considered an outlier, and not included in the calculation. The initial plant and berry height collected data helped to set the filter range. Lines of codes were added to the Python code to filter out the values 45 ≤ Z-value ≥ 75 cm, so by this filtering, the code only includes those values within the range and takes the average when it detects the nine berries.
While harvesting the height of the header picking unit change according to berry height, with that, the camera height from the ground is also changed, which we initially fixed as a base height. To find the real-time difference in camera height for precise measurement OAK-D camera built-in Inertial Measurement Unit (IMU) BNO085 sensor values were used. This IMU sensor is 9-axis which can record the accelerometer, gyroscope, and magnetometer values. The accelerometer can measure the rotation around all three (X, Y, Z) axes and calculate pitch, yaw, and roll values. The IMU sensor’s built-in position on the module is like when the camera is fixed and pointed downward; the rotation only comes around the y-axis, called pitch. The IMU pipeline was included in the Python coding to correct the change in camera height in real-time.
A real-time relationship was developed using linear regression to calculate the correct camera height from the pitch value. The workshop’s initial trial was done by installing the camera on the mount. The initial camera height was 75 cm from the flat ground surface. The camera was mounted in a way that as the header moves up, the distance between header teeth and the ground increases, but the camera moves downward, and its height decreases. The pitch value changed between 5° to 16° values at the minimum and maximum header height positions, and in the field, it usually varies from 9° to 14°. The camera heights from the ground were measured using the scale at every pitch change. The difference between the initial height and height by changing the header position was calculated, and the relationship was developed using a linear regression method.
$${\text{h }} = \, – 0.{\text{76d }} + { 11}.{68}{\text{.}}$$
(2)
The code used the developed equation to find the change in the camera to ground distance. Here d is the pitch value in degree, and h is the height correction value (cm) which is the change in initial camera height from the ground. The correction value was subtracted from the initial fixed camera height for accurate berry height measurement, as illustrated in Eq. 1. The camera height was noted using a one-meter stick to compare the generated auto-correction value and the difference in the camera’s initial and present height at each data point location.
Berry fruit height measurement
The plant’s lowest and highest berry heights were measured at each marked location using a ruler to evaluate the performance of the auto system. For this purpose, a specific quadrant (41 cm × 41 cm) shown in Fig. 5a was used to follow the same procedure to measure the berry height throughout the field. The lowest and highest berry height at plants were measured at the intersection of wires or from the plant close to the intersection in case any plant is not at the intersection. This measurement helped to compare and understand the variation in the fruit zone and the setting of the harvester header picking teeth.
(a) Fruit height data collection quadrant. (b) Wild blueberry harvester header.
Manual stroke height measurement
The open stroke length was measured using a ruler to find the harvester header picking teeth height from the ground, as shown in Fig. 5b. The driver sets the stroke height based on the plant height, weeds, and slope by visualization from the tractor cabin, while the auto height adjustment system takes the nine berries height and pitch value for camera height correction to set the header height. For comparison, the stroke readings were noted separately at the marked points for both operator and auto systems.
Statement: data collection from wild blueberry fields
Necessary permissions were obtained from the grower to collect the plant height data and to check the performance of the harvester header positioning system in field. In this study, we comply with the IUCN policy statement on research involving species at risk of extinction and the convention on the trade in endangered species of wild fauna and flora.
