Inside the EYEBOX: How We Turn Raw Sensor Data Into Accurate 3D Models of the Farm

A point cloud is easy to generate and hard to trust. Here's what actually has to go right — from calibration to annotation — before a 3D reconstruction is accurate enough to steer a machine through a field.

A point cloud captured by the EYEBOX sensor suite along a field boundary— color mapped to elevation to reveal terrain, ruts, and field boundaries.

Every autonomous farming function — obstacle detection, field boundary tracking, rock and rut detection, monocular depth estimation — depends on one underlying thing: a 3D model of the world around the machine that can be trusted down to the centimeter. Building that model sounds simple in a slide deck. In practice, it's one of the hardest engineering problems in the whole perception stack, and almost none of the difficulty is in the "3D" part people usually picture.

We wanted to pull back the curtain on how a 3D model actually comes together, from raw sensor data to an annotated point cloud a machine learning model can train on — and where things typically go wrong.

What a "3D model" actually is

When people say "3D model" in the context of perception, they usually mean a point cloud: a dense collection of individual points in space, each with an (x, y, z) position and often a color or intensity value. There are two dominant ways to produce one, and they work in almost opposite ways.

LiDAR: measuring geometry directly

LiDAR (Light Detection and Ranging) sensors fire laser pulses outward and time how long each pulse takes to bounce back off a surface. That round-trip time, combined with the speed of light, gives a precise distance to that point. Millions of these measurements per second build up a point cloud directly from physics — no guessing involved. Because it's an active sensor rather than a passive one, LiDAR keeps working in the dark, in dust, and in flat, textureless light where cameras struggle.

Vision based: inferring geometry from images

The vision based approach, instead of measuring distance directly, it captures overlapping 2D images from multiple viewpoints and uses triangulation — comparing how the same physical point shifts across different camera positions — to calculate depth. It's a clever indirect trick, and modern algorithms are very good at it, but the result is only as good as the images: shadows, glare, low texture, and poor overlap between shots all degrade accuracy in ways that don't affect LiDAR.

The short version: LiDAR measures geometry. The vision based approach infers it. That's why almost every serious autonomous-vehicle and off-road-machine perception stack — EYEBOX included — fuses both, along with radar, GNSS, IMU, and CAN bus data, rather than betting everything on one sensor.

The EYEBOX sensor pod: a LiDAR unit paired with multiple cameras in a single housing, built to be retrofitted onto large farming equipment.

The unglamorous part that determines everything: calibration

Here's the part most explanations of "how 3D models work" skip entirely, and it's the part that actually determines whether your point cloud is usable: calibration.

A sensor pod full of cameras, LiDAR, radar, GNSS, and IMU produces six or seven independent streams of data, each describing the world from its own physical position and its own internal quirks. To fuse them into one coherent 3D model, you need to know two things with high precision for every sensor:

• Intrinsic calibration — the internal properties of a sensor: for a camera, this means focal length, optical center, and lens distortion. Get this wrong and straight lines in the world bend in your image.

• Extrinsic calibration — the precise physical position and orientation of each sensor relative to every other sensor and to the vehicle itself. Get this wrong and a LiDAR point that's actually 3 centimeters to the left of where a camera pixel says it is will silently corrupt every downstream fusion step.

This is genuinely difficult on a passenger car. It's substantially harder on a combine harvester, sprayer, or tractor: these machines are enormous, flex and vibrate under load, and don't sit still on a factory calibration rig. A calibration routine built for a sedan doesn't transfer.

A calibration point cloud used to verify that LiDAR returns and camera imagery agree on where objects actually are in space, across a large piece of equipment.

That's why Mojow built a proprietary calibration toolbox specifically for large agricultural machines — custom software and hardware fixtures that let us calibrate a full sensor suite quickly, verify it, and re-run the process any time a machine is retrofitted with a new sensor configuration. Without tight, repeatable calibration, none of the rest of the pipeline matters: you can have the best LiDAR on the market and still end up with a 3D model that's geometrically wrong.

Terrain and obstacle detail captured once calibration is verified — note how cleanly discrete objects and ground plane separate when extrinsics are accurate.

Time synchronization: the other invisible requirement

Calibration tells you where each sensor is. You also need to know when each measurement was taken, to the millisecond. A LiDAR sweep, a camera frame, and a GNSS fix that are even a few tens of milliseconds out of sync will describe slightly different moments in time — which, on a machine moving through a field, means they're describing slightly different physical scenes. Tight time synchronization across every sensor modality is a prerequisite for fusion, not an optimization you add later.

From point cloud to training data: annotation

A calibrated, synchronized, fused point cloud is a big step forward — but it's still just geometry. To train a machine learning model to recognize field boundaries, rocks, ruts, standing water, or unharvested crop from that geometry, every relevant object in the scene has to be labeled. That's 3D annotation, and it's meaningfully harder than labeling a 2D photo.

3D bounding-box annotations on EYEBOX point cloud data. Each box captures an object's position, dimensions, and orientation in three dimensions — not just a rectangle on an image.

A few things make 3D annotation its own discipline:

• 3D bounding boxes and cuboids capture not just where an object is, but its size and orientation in space — essential for anything a machine needs to navigate around or avoid.

• Semantic segmentation assigns a class label to every individual point, which is how a model learns to tell drivable ground from a field boundary or an unharvested strip of crop.

• Occlusion and point density make this harder than 2D: objects close to the sensor are dense and easy to label, distant objects thin out into a sparse scatter of points, and anything outside the sensor's line of sight simply isn't there at all. Annotators have to reason in three dimensions, rotating and zooming through a scene rather than reading it off a flat image.

Mojow runs this with a dedicated team of trained annotators who work exclusively with farming data and Mojow's sensor modalities, backed by a two-step review process before any annotation is used in a model. That team has produced over three million annotated data frames spanning field boundaries, farm implements, multi-view depth, field wetness, rocks and ruts, and harvested-versus-unharvested crop boundaries.

3M+

ANNOTATED DATA FRAMES

10

FULL-TIME DEDICATED ANNOTATORS

2-STEP

QUALITY REVIEW BEFORE USE

Multiple viewpoints, one consistent model

A single sensor pod, however well calibrated, only sees part of the machine's surroundings at any moment. Large equipment needs 3D coverage from multiple positions around the machine simultaneously — front, sides, rear — all resolving to one geometrically consistent model of the world. That means every viewpoint has to be calibrated into the same shared coordinate frame, so a rock detected by a side-facing sensor and a boundary detected by a forward-facing one line up correctly in the same 3D space.

Multiple synchronized viewpoints around the machine, fused into a single 3D reference frame rather than treated as independent feeds.

Why accuracy compounds

The reason all of this matters together, rather than in isolation, is that error compounds through the pipeline. A small intrinsic calibration error distorts every pixel from that camera. A small extrinsic error misaligns that camera's data from every other sensor's data. A timing mismatch misaligns the whole scene by a few centimeters at the machine's operating speed. And an inconsistent annotation on top of already-imperfect geometry teaches a model the wrong thing with high confidence. None of these failures are individually dramatic — they're the kind of thing that looks "close enough" in a demo and then causes a missed rock detection or a wrong field boundary call in the field, which is a very different kind of consequence on a moving farm implement than on a phone app.

That's the real answer to "what does it take to accurately create a 3D model": not a single clever algorithm, but a disciplined pipeline where calibration, synchronization, multi-modal sensor fusion, and careful human annotation each have to be right, consistently, at scale, before the geometry is trustworthy enough to build a machine learning model on top of it.