Landing-navigation benchmark · v0.1.0

Can your lander find itself when the shadows move?

Descents onto real lunar south-pole terrain, rendered frame by frame with the exact camera position and pointing. Match each frame to the map, estimate the pose, flag the hazards. At the pole the Sun skims the horizon, so the map you carry was lit from one side and the landing may be lit from another.

Our reference map matcher gets 46% of frames (11 of 24) within 10 m horizontally and vertically and 0.2° in attitude when the descent is lit like the map, and 0% (0 of 24) when the Sun has moved about 90° (withheld test site). That gap is the open problem.

south-polar sites on LOLA 5 m terrain

descent sequences, two lighting cases each

144

frames with ground-truth pose

16°

glide, 1000 m to 150 m, Apollo-style approach

Frame 1, 1223 m above ground — 1223 m · T+0 s

Frame 6, 994 m above ground — 994 m · T+12 s

Frame 10, 799 m above ground — 799 m · T+24 s

Frame 15, 547 m above ground — 547 m · T+42 s

Frame 19, 347 m above ground — 347 m · T+61 s

Frame 24, 138 m above ground — 138 m · T+96 s

Shackleton rim at the south pole · 1.48° Sun · one descent from 1223 m to low gate

Watch a descent

Left: what the camera sees. Right: the reference map the lander carries, with the true path, the inertial guess it starts from, and where SIFT + PnP map matching put it. Switch the lighting to see the matcher lose its grip.

Descent frame 1 of 24, Shackleton–de Gerlache connecting ridge, Lit like the map

T+0.0 s · frame 1/241206 m above ground

Sun 1.07° up · 20° from the map Sun2024-01-23

Hazard label

Horizontal position error, per frame (log scale)20/24 frames within limits

dashed grey: navigation priordots: SIFT + PnP map matching× no fixdashed green: 10 m limit

Reference map of Shackleton–de Gerlache connecting ridge — true tracknavigation priormap-matching fix within limitsfix outside limitsmap: 2.5 m/px render, Sun 1.12° up (dashed in the compass)

Leaderboard

Ranked on the withheld test site (de Gerlache crater, pole-facing rim). A frame counts only if the position is within 10 m horizontally and 10 m vertically and the attitude within 0.2°; a frame with no estimate counts as a miss. The first rows are our own reference methods, scored by the same open scorer.

test split

Pose track

Method	Success	Lit like map	Cross-light	Fix rate	Median error	95th pct	Attitude
1 SIFT + PnP map matching Hevelius baseline Predict the view from the prior pose by casting rays onto the DEM and sampling the reference map, match SIFT features to the frame, solve the pose with RANSAC PnP (two passes).	23%11 of 48	46%	0%	33%	1.74 m	3.43 m	0.15°
2 Navigation prior only Hevelius baseline The navigation prior, unchanged: where the lander is without terrain-relative navigation.	0%0 of 48	0%	0%	100%	53.7 m	59.5 m	0.33°

Hazard track

Method	IoU	Lit like map	Cross-light	Precision	Recall	F1
1 Dark pixels are hazards Hevelius baseline Pixels darker than 12% of the frame's 99th-percentile brightness are hazards.	0.752	0.865	0.641	0.873	0.845	0.858
2 Everything is a hazard Hevelius baseline Every pixel is a hazard (trivial reference, recall 1).	0.441	0.447	0.435	0.441	1.000	0.612

Median and 95th-percentile horizontal error and median attitude error are over frames with a fix. Hazard scores pool pixels where the label is defined (on the 5 m terrain window). Scorer: score.py.

Test site: de Gerlache crater, pole-facing rim

ground truth withheld

Frames, intrinsics, navigation prior, terrain and reference map are public; the true poses and hazard masks are read only by the scorer. Near 89.00°S 96.35°W.

The task

For every frame, estimate where the camera is (map metres) and how it is pointed, using the frames, the camera model, the navigation prior, the site terrain and the reference map. Leave a frame empty when you have no fix. Optionally return a hazard mask per frame.

Prior error: 30 m per axis horizontally and 10 m vertically at the start, drifting 0.4 m/√s; attitude 0.2° per axis.

In the package

16-bit frames (12-bit camera with shot and read noise), 640 × 480, timestamps
Pinhole intrinsics and OpenCV axes; poses as map position + quaternion
LOLA 5 m terrain window per site and a 2.5 m/px reference map
Dev split: true poses and hazard masks; test split: withheld
Scorer and the reference methods, open source

Get it, submit

The v0.1.0 package is shared on request while public hosting is set up. Send a poses.json (and optional hazard masks) through the form; we score it with the same code and add your row.

Test-split scores are published for one submission per team per week, best entry shown, so the board cannot be used to probe the hidden poses. Score the dev split yourself as often as you like.

Request the package or submit

What this is not

The reference map is a render, not an orbital photo mosaic, so results under matching light are optimistic. Render-vs-photo agreement is measured on one site; see the numbers.
The terrain is LOLA at 5 m per pixel with synthetic sub-pixel shading: no real rocks or small craters. Late frames are smoother than the real surface.
The approach and the navigation prior are simplified, assumed models (Apollo-like glide, stated noise), not any vehicle's flight plan. Aim points are flat spots we picked, not mission landing sites.
The camera is an ideal pinhole: no lens distortion, motion blur or rolling shutter.

Full method, equations, thresholds and sources: docs/science/landing-nav-benchmark.md.