CORE PERCEPTION BASELINES
We introduce six core perception tasks that serve as the foundation for any perception system in broadacre farming. For each task, we try to establish preliminary baselines either using our custom models or state-of-the-art models. For each task, we describe its importance, the challenges associated with it in broadacre farming and some of our preliminary results. All models are evaluated on images at 800 × 640 resolution on NVIDIA Jetson Thor device. The inference time only considers the model run time and excludes the data loading and pre/post processing time. The models are converted to ONNX and then to TensorRT for inference. The FPS numbers are rounded to the nearest integer and are the average over 10 runs on a test set. In model conversion to TensorRT, we use FP16 precision where possible.
1. Field and Crop Boundary Mapping
Field boundaries include all the internal and external boundaries defining the area the farming equipment is allowed to drive in and operate. These boundaries often are represented by a set of 2D polygons or 3D surfaces and meshes. The boundaries are used to plan coverage paths for the equipment to follow and complete a task such as seeding or spraying the field. The accuracy of this mapping process is very important for safety and efficiency of the operation. These boundaries usually are constant over a long period of time (e.g., years). However, in certain regions such as western Canada, these boundaries could change more frequently (e.g., weekly) due to weather conditions (e.g., rain or snow melting), natural events such as downed trees on a boundary due to high winds or human activity (e.g., the farmer clearing a corner of the field from bushes and trees). When boundaries change, the remapping process is quite timely and expensive (in many aspects) for the farmer and having an autonomous solution that can take care of this mapping operation safely and accurately would help tremendously. Ideally, the autonomous perception system should be able to detect the boundaries, map them, compare them with previous versions and update them as needed without human intervention.
Similar to field boundaries, during harvest, crop boundary mapping becomes very important for autonomous operation of the combine and harvest cart. For example combine harvesters must know where the crop exactly is to be able to do their task. The harvest cart needs to know which portion of the field is harvested because it is only allowed to drive in that area and not over the crop. A crop boundary segmentation and mapping is a main component of any harvest related perception system. Generally, the scope of this perception task is very broad and includes any boundary estimation and mapping task related to the crop, crop rows, stubble rows, field boundaries, swath boundaries, harvested crop boundary, etc. We have a dedicated dataset for this task with accurate segmentation annotations for common crops from western Canada including wheat, canola, peas and lentils. Table 1 shows some of our preliminary results by evaluating our custom real-time segmentation models on a test set with 5000 frames.
Table 1. Preliminary field and crop boundary segmentation results on FarmScapes using our custom architecture segmentation models (small and large variants).
Model
Ours-S
Ours-L
mIoU (%)
88.7
95.1
Params (M)
8.3
38.5
FPS
552
220
2. Rural Road Detection and Navigation
Autonomous machines must transition between fields via rural roads, which are often unpaved and lack lane markings. We treat this as a semantic segmentation task, focusing on the “drivable surface” category. As mentioned in the broadacre farming lifecycle, road driving is heavily involved in the process. The challenging roads are the dirt roads and gravel roads with no clear boundaries. Such roads are very common in western Canada and are non-existent in open source datasets which are mostly focused on urban or highway driving scenarios. Also, existing trained models often fail to properly detect the drivable surface on these roads. We have a dedicated dataset for this task with accurate segmentation annotations for rural scenes from western Canada including dirt, gravel and paved roads. Table 2 shows some of our preliminary results by evaluating our custom real time segmentation models on a test set with 5000 frames.
Table 2. Preliminary rural road segmentation results on FarmScapes using our custom architecture segmentation models (small and large variants).
Model
Ours-S
Ours-L
mIoU (%)
90.1
96.1
Params (M)
8.3
38.5
FPS
550
221
 |  |
|---|---|
 |  |
 |  |
 |  |
 |
3. Outlier Detection: Rocks and Dust
Operating in the farm field usually involves driving over uniform surfaces such as soil, crop, rows, stubble, etc. with repetitive patterns. Anything other than the known patterns is considered an outlier (e.g., a rock). Because of the vastness of broadacre farms and low occurrence of outliers, collecting real data from these scenarios is extremely time consuming and expensive. That’s why we treat this problem as anomaly detection.
One of the most common anomalies in the fields are rocks. Rocks scattered in the fields are extremely dangerous for combine harvesters during harvest season. If a rock is picked up by a combine, it can damage it significantly and more importantly it will cause downtime for farmers. Downtime is the most critical factor during harvest for Canadian farmers due to the extremely short season and the limited time available to harvest the crop before the first frost in the fall season. We have created a large rock dataset from real fields and have developed a rock detection and mapping system that can be used during spring season at seeding time to map out the rocks in the fields. This gives the farmers time to use our rock maps to clean their fields before harvest. Our dedicated rock dataset contains both bounding box and segmentation annotations. Dust is another common challenge in broadacre farming. It can significantly reduce the visibility of the sensors and make it difficult for any perception system. We also have created a dedicated dust dataset that contains both bounding box and segmentation annotations. We also have created a general anomaly detection dataset that contains a wide range of anomalies that can be found in the fields. Anything that is NOT a crop, crop row, stubble row, field boundary, swath boundary, harvested crop boundary, etc. is considered an anomaly. This dataset is used to train anomaly detection models. More details about our anamoly detection will be released in our future work. Table 3 shows some of our preliminary results by evaluating our custom real-time rock detection model on a test set with 2000 frames.
Table 3. Rock detection inference results from our single- class detection model on FarmScapes.
Model
Ours
AP@50 (%)
88.3
Params (M)
40.2
FPS
174
AP@50:95 (%)
71.7
4. Depth Estimation in Unstructured Scenes
Depth estimation is another important perception task in broadacre farming. LiDARs are used on many mobile robots, however they are expensive and lack the resolution, color and texture information of RGB images. In order to map everything accurately around the machine, we need accurate depth information. In broadacre farming, the machine and implement size can be up to 160 feet (50 meters) wide which introduces a new challenge for depth sensors. Even for LiDARs, at 100 feet range the point clouds become so sparse that makes it almost useless in some of the common autonomy applications in broadacre farming. Vision based depth estimation is a promising alternative or complement to LiDAR in these scenarios which helps create dense depth maps of the environment around the machine. The depth models help us complement other task specific segmentation and object detection models and allow us to build high resolution 3D maps of the environment around the machine used for autonomous navigation purposes. We have created a dedicated dataset for depth estimation task that contains accurate depth annotations for all frames. The dataset includes training data for both monocular, multi-view and stereo depth estimation models with millions of frames.
Table 4. Stereo depth estimation results on FarmScapes. EPE: End Point Error, D1-all: Percentage of pixels with disparity error greater than 1 pixel.
Model
Ours
EPE (px) ↓
0.74
Params (M)
6.8
FPS
46
D1-all (%) ↓
1.71
Table 4 shows some of our preliminary results by evaluating our custom real-time stereo depth estimation model on a test set with 200 frames. More proper analysis of the 461
results and establishing a baseline for depth esitmation task will be published in our future work.
5. Human Detection in Agriculture
Humans are the most important asset at the farm and are the top priority in any perception system and safety systems. The majority of the available datasets (open/closed source) involve human data from urban or indoor environments and they lack the context and diversity of broadacre farming related scenes. For example, almost every open source model struggles with detecting humans lying down in farming fields. We are creating one of the largest human datasets with farming context to help overcome the challenges of detecting humans in farming environments. We already have a dataset with tens of thousands of human annotations for this purpose. We also have developed real time human detection models for farming environments and have evaluated them on our dataset. The preliminary results for establishing a baseline will be published in our future work.
6. Implement Pose and State Estimation
The attached implement to the machine (e.g., land roller, heavy harrow, seeder, sprayer, combine header, etc) does the actual farming job. As part of our perception system for autonomously completing a task, we have created large datasets of the implements. Most of the implements in broadacre farming are foldable because of their large size (e.g., 30ft+ and up to 160ft wide). Using our datasets we have already developed perception systems that help us autonomously fold/unfold implements. While operating autonomously with no supervision in the field, it is important for the machine to know the full state of the implement (e.g., folded, unfolded, partially folded, plugged, detached hydraulics, etc.) and the pose of the implement (e.g., position and orientation of the implement relative to the machine). Another important state estimation task for the implement is to estimate the quality of work and reporting that back to the farmer. The preliminary results for establishing a baseline will be published in our future work.