MACHINE LEARNING
Introduction
Machine learning (ML) and computer vision (CV) are two of the main focuses of our team at Mojow. We use ML/CV to solve different perception problems that help our machines see and understand the environment at the farm and interpret the situations that happen in different farming operations. There are far too many tasks and operations happening at the farm across a season that makes it impossible to develop a single monolithic perception application that solves all perception tasks without compromising the quality of outcomes. Instead, at Mojow, we focus on individual perception problems, and develop dedicated solutions that deliver outstanding outputs with the same or better quality than an expert human operator. Our goal is to solve perception problems the same way expert experienced farmers solve them.
Thanks to our flexible and robust software infrastructure of EYEBOX™, we are able to deploy one or more perception applications for different tasks and have them work in unison with the highest accuracy on our machines in the field. Each perception application is an expert at what it is doing, whether it be road perception, field boundary perception, obstacle detection, crop row detection and mapping, implement perception, or any other task that an expert operator cares about while doing the farming operation. These perception applications are robust, real-time, and safely satisfying the requirements for a particular task the farmer cares about.
You can think of sensors on EYEBOX™ as the eyes (LiDARs, cameras, radars) and ears (microphones) of the expert human operator multiplied by 10 (because we can add as many sensors to the machine as we want), seeing and hearing everything around the machine and building a dense 3D representation of the machine’s surroundings. That’s literally why we call our product EYEBOX™; it's a box full of eyes. These sensors pass all the data at multiple gigabytes per second rate to the perception applications. The perception applications act as the expert human operator brain and turn the dense 3D representations of the machine’s surroundings into meaningful semantic representations such as field boundaries, road boundaries, power poles, obstacles, sloughs, crop rows, humans, hay bales, fields, tractors, etc. The dense semantic information along with dense 3D geometric information gives EYEBOX™ navigation system all it needs to drive the machine autonomously and complete a task.
Our perception applications turn EYEBOX™ into an expert autonomous operator fully understanding the context of the farming environment, and able to make decisions according to farmer instructions set in the mobile application. This is a powerful assistant for the farmer to help them get the job done autonomously while maintaining total control of the operation.
This page provides insights into how machine learning is done at Mojow, and how we turn high quality data into robust and safe perception applications deployed in the field.
Perception Recipe
At Mojow, we have prepared a recipe for successfully solving a perception problem at the farm. Here are the main ingredients:
-
High quality data from real operations
-
Farmer input and feedback for the design of the system to make sure we build something the farmers want
-
Mojow’s proprietary machine learning models that are robust, safe and run in real-time
-
Robust distributed software and hardware infrastructure to deploy the perception applications. This is EYEBOX™ infrastructure already developed by the Mojow software team
-
Simulation to validate the performance in thousands of scenarios in a short period of time
-
Deployment and field trials to validate the performance in real field conditions
-
Final packaging and release
OUR DATA-DRIVEN APPROACH
Annotated frames for training
3+ Million
Targeted frames for
testing and evaluation
50+ Million
Field Boundaries
Farm Implements
Harvested/Unharvested
Crop Boundary
Humans in Farming
Environments
Multi-View
Depth
Rural
Roadways
Field Wetness &
Open Water
Rocks, Ruts &
Anomalies in the Field
Farmyards
Hay Bales
and Hay Stacks
Power Poles
Many more...
All the applications developed at Mojow are data-driven, meaning that each project starts by brainstorming the design by studying real data collected from real field operation. This helps our engineering team to stop hallucinating problems in our heads and coming up with solutions that nobody wants. It helps us make realistic decisions about design and operating domains for each project using real data points, not just based on abstract assumptions in our heads. Our philosophy is that if an expert human operator can do the job, we should be able to collect their sample data from expert human operations and observations and develop a perception application that is at least as good as the human operator and in most cases better than the human operator.
Since day one at Mojow, we started collecting quality data from farming operations at numerous farms across Western Canada. We considered the complete farming lifecycle: preparing the equipment before spring operation starts, field preparation, seeding/planting, spraying, harvesting, and all post harvest field operations (until all field work is complete and machinery is returned to within the farm yard). After months of brainstorming with many farmers, we designed a Yard-to-Yard Autonomy concept. In simple terms, Yard-to-Yard Autonomy means that all the work starts in the farmyard at the beginning of the season, and ends back in the farmyard after the work is completed. Therefore, our autonomy software and hardware stack must consider the complete farming lifecycle, otherwise it won’t be addressing all the challenges we will face in our implementation.
Considering the yard-to-yard autonomy tasks, we start collecting high quality datasets from every single step of the lifecycle such as:
-
Navigation through farm yards
-
Attaching or detaching implements to tractors
-
Folding/unfolding implements
-
Navigating through all kinds of entrances for yards and fields
-
Driving on all kinds of roads (paved, gravel, dirt, trails, etc.)
-
All general farming operations such as seeding, rolling, spraying, rock picking, harrowing, harvest, grain cart, etc.
-
Day and night operations
-
Different crop types
One of the key aspects of our software stack is its ability to develop and deploy machine learning models for perception applications. We had to make sure we have high quality training-ready datasets for these applications. So we developed a framework for collecting high quality datasets. We came up with the concept of training-ready datasets, which means that using our framework, we have defined and automated all the steps necessary to:
-
Collect data from the field
-
Filter, clean-up, and index all the collected data in our database so the engineers can easily access it through the Mojow data query engine
-
Annotate the data using the Mojow annotation tools
-
Use the Mojow data query engine to package training ready datasets in any format required by the machine learning models
-
Start training on our Mojow servers
Data collection is an expensive process. We didn’t want to risk compromising the quality of the data during the collection phase, so we prepared completely custom hardware and software to guarantee the collection of high quality data. Our data collection software and hardware system has the following design constraints:
-
Tight time synchronization among all sensors at the microsecond level
-
Our accurate calibration (extrinsics for all sensors) done using our proprietary software allows for smooth transformations between 2D and 3D
-
Different modalities of data (camera, LiDAR, GNSS, IMU, Radar, CAN Bus)
-
All possible viewpoints
-
Distribution and variety of scenes
-
Variety in lighting conditions
-
Variety of objects in the scene
-
Usage in various locations
-
Usage in all farming operations, and with all the different equipment
-
Accurate and consistent annotations using our proprietary annotation tools
-
Density of annotations from interesting subjects relevant to farming operations
We have a dedicated full-time team responsible for collecting and creating training-ready datasets for all of our projects that are relevant to our yard-to-yard autonomy objectives. Our team has access to the following resources to create such high quality datasets:
-
Custom software for data collection
-
Custom software and tools for calibration
-
Easy process for quickly iterating over sensor configurations, and collecting data from all various viewpoints on the machine
-
Can quickly retrofit a machine for data collection
Using our method for creating training-ready datasets, we have been able to create the world’s largest training-ready dataset for broadacre farming applications.
MACHINE LEARNING APPLICATION LIFECYCLE
Data Collection
Annotation
Model Training
Evaluation
Deployment
-
5 years of data collection
-
Clean data for machine learning
-
All modalities of data (camera, lidar, radar, IMU, GPS, CAN, etc.)
-
Event-triggered data recording
-
Lots of targeted datasets already created from a large variety of operations
-
Dedicated team of 10 full-time annotators
-
High quality annotations
-
2 levels of reviews to ensure quality
-
Already have many training-ready datasets
-
Dedicated datasets for training, testing, and validation
-
Database of high performing models for different tasks
-
Multi-task models
-
Real-time and fast models
-
Training infrastructure for type of model
-
Model repository and version control
-
Dedicated software for evaluating trained models on test sets
-
Infrastructure for quick field testing
-
3D simulation software for testing perception applications before testing in the field (early-stage verification)
-
Model optimization and compression
-
Deployment to any environment with x86 and ARM processors, and other deep learning accelerator modules
-
Inference server onboard for serving many models at once
-
Infrastructure for updating models in the field
We use the following steps to develop our perception applications :
Model
Development
Because quality is one of the most important requirements in our design for perception applications, most of our machine learning models are designed by the Mojow team to satisfy the strict requirements. At Mojow, we are looking for solutions that solve the problem completely. We’re not looking for 80% of the solution. To deliver a solution at this level of quality, we are forced to develop custom proprietary models that target our problems directly and solve them 100%. The biggest trap most other teams fall into is using generic open source models that work 80% of the time and fail miserably for the rest of the 20%. This is not acceptable at Mojow. That’s why we have designed a framework for developing custom machine learning models with the following key factors taken into account:
-
Every application is different
-
We specialize in designing, picking, and tweaking the best model for the application
-
Optimizing trained models for inference speed and performance
-
Real-time and fast models for real-time applications (e.g., navigation)
-
Supervised and self-supervised learning models
-
Testing and evaluation software infrastructure
-
Database of high performing models
-
Model tracking and version control is paramount for completely solving the problem and tracking issues
This is very critical in designing a custom model, and without it it would be impossible to track and fix all the issues that will come up in the field at deployment time. It helps us iterate quickly and efficiently through different versions of the models and update it until we have solved the problem.
-
We do extensive testing on our test datasets
-
Test datasets are never used in training to avoid leaking information (we don’t want our models to memorize them)
-
Test sets are handcrafted by expert reviewers from our data team
-
Without these test sets it would be almost impossible, or extremely time consuming, to test models in real fields
-
Real field testing comes after running our models and detecting failures on our test sets
The great news is that we have the biggest test sets in the world to validate the models before deploying them to the field, which saves us months of field testing and evaluations.
