Localize over MCP

localize answers from memory: the colour frames the module has embedded, lifted to 3D through depth or lidar. It does not look at the current frame. Every answer is a verified 3D object with a score and a view count, and an empty answer is an answer.

Before the first call

nvidia-smi --query-gpu=name,memory.used,memory.total --format=csv,noheader
uv run dimos mcp call state

• Probe the GPU first. How many labels a call can carry depends on it, see "Labels and GPU".
• state() returns ready: N frames embedded, localize will answer or not ready: <stage>. Until it reads ready, localize returns its stage instead of an answer.
• Error: no running MCP server (is DimOS running?) means the stack is down. Check uv run dimos status and report it. Do not stop or restart the stack.

Calling localize

uv run dimos mcp call localize -j '{"objects": "chair,door", "start": -30, "duration": 30}' --timeout 300

• localize(objects, start=-10, duration=10, params="") returns once the answer is built. Send the next call after it returns.
• objects is one label, or a few separated by commas. The labels of one call share one detector pass per frame.
• The client waits 30 s. A window with many unread frames takes longer, so pass --timeout 300 or set MCP_TIMEOUT=300.

Building the window

start and duration are seconds over the embedded frames the module holds.

• A negative start counts back from the newest embedded frame, a positive one forward from the oldest frame still held. The module holds horizon_s of frames, 600 s by default, so once memory is full the oldest frame moves.
• Frames are embedded at 1 per second, 3 on a walking robot (mobile=True), each the sharpest of its slot.
• The first answer line is the window as resolved: window A s to B s of C s of feed, N frames. With N = 0 nothing was read and the answer is memory only.
• A frame is read once per label and params, accept_score and min_views aside. A window of frames never read for a label costs detector passes; asking the same window again is nearly free.

Reading the answer

window 0.0s to 30.0s of 28.3s of feed, 76 frames
'chair' at (-0.97, 7.31, 0.40) in world score=0.49 views=6 id=ed6e5cde seen 24s ago
no verified detection of 'bag'

• One line per verified object, newest sighting first. The position is the centroid of the object's newest sighting, in the frame the line names.
• id names the same object for as long as the module runs.
• score is the detector's best score for the object. Compare scores within one label only; the scale changes with the wording.
• views counts the distinct camera positions, 1 cm apart, that saw the object. An object needs 2, and a camera that does not move adds none. A camera that never moves, such as a fixed top view, stays silent until min_views is 1: pass '{"min_views": 1}', which reads no frame again.
• Objects found by earlier calls are answered again, also outside this window. seen Ns ago is the age of the newest sighting.
• no verified detection means nothing passed the gates at these params. It is not proof that the object is absent.
• Labels are not compared: one object can be answered under two labels.

Watching labels on a schedule

Four labels on a rolling 5 s window, one call after another:

uv run dimos mcp call localize -j '{"objects": "chair,door,bag,plant", "start": -5, "duration": 5}' --timeout 60

• A laptop RTX 5070 with 8 GB sustained this, calling continuously.
• A call reads only the frames not yet read for its labels, so each call costs the new seconds, not its whole window.
• The window must span the time between the starts of two calls: the call's own run time plus any wait. If calls take longer than the window, widen it. The overlap is free, because read frames are skipped; a gap loses frames.
• Keep the label strings and params the same on every call. Memory is per exact label string, and frames read under other params are read again, unless only accept_score or min_views changed.
• An object that left view stays in the answer at its last position, with a growing seen Ns ago. An object that moved farther than cluster_radius_m gets a new id at its new position, and the old id stays at the old one.

Narrowing a position

The position is the newest sighting's, so a closer sighting refines it.

1. Bring the camera close to the object and move it a little: every new camera position adds a view.
2. Ask a short recent window, {"objects": "red mug", "start": -5, "duration": 5}. A sighting within cluster_radius_m of the object's centre joins it under the same id, and the position moves to the new sighting.
3. Check that views grew and seen is near 0 s.

At a table, a walking robot's room-scale params are coarse for small objects:

Pass desk-scale field overrides on every call for that label while working at the table. Objects the label already has from room-scale calls keep how they were built; a label first asked at the table is built at desk scale.

Changing params per call

Each blueprint sets its rig's LocalizeParams (types.py) at construction (desk-scale defaults in LocalizeParams(), room-scale GO2_PARAMS in go2_localize.py).

params is a JSON object of LocalizeParams field overrides for that call only. The next call without it is back on the module's params, the ones its blueprint set.

uv run dimos mcp call localize -j '{"objects": "apple", "params": "{\"accept_score\": 0.45}"}' --timeout 300
uv run dimos mcp call localize -j '{"objects": "apple", "params": "{\"min_views\": 3}"}' --timeout 300

• accept_score and min_views act only when the answer is built: changing them reads no frame again and applies at once to everything remembered for the label. Try them first.
• The other fields shape only what happens from now on. Objects already built keep how they were built.
• A misspelled field fails the call with unexpected keyword argument.
• Every field and its desk-scale default is defined on LocalizeParams in types.py.

Labels and GPU

• Do not pass many labels at once. Every detector box is segmented on the GPU, so each label adds memory and time.
• Reference: a laptop RTX 5070 with 8 GB sustained 4 labels per call on a rolling 5 s window, calling continuously. With less free GPU memory, or with Rerun or other models on the same GPU, start with fewer.
• Use concrete nouns with one attribute that sets the object apart: red mug, wooden crate. Broad or overlapping labels make many boxes.
• On Remote torch.OutOfMemoryError, halve the labels per call.

Where a failure comes from

Embedding, detection and segmentation work from the image and find almost any object. Lifting a detection to 3D needs depth at the object, and that depends on the rig. Most failures to get a position come from the lift or from the rig, not from the image models.

Room: walking robots

• No depth at the object. The stack lifts through the rig's depth camera or lidar, so a detected object gets no 3D answer until depth reaches it. Go2's L1 lidar is sparse and covers a small radius: when the robot starts moving, localize can lift only what is close. Bring the robot nearer and ask a recent window again.
• Odometry drift. A static object seen at two times with drifted poses lands in two places, as two ids. The one seen most recently matches the current odometry.

Desk: arms

• Depth is rarely the problem.
• Look-alike objects. A group of objects of the same colour and shape, with few unique features, can be missed by the detector from some angles.

Any rig

• Calibration. A poorly calibrated rig shows every object twice, the copies shifted by the same offset for every object. That points at the rig, not at the params.
• Light and environment. Every rig depends heavily on them. Tune params on the fly with the call's params, the score fields candidate_floor and accept_score first.
• Clouds for grasping. fuse_voxel_m is 0.01 in LocalizeParams() and 0.03 in go2. For GraspGenX, go as low as 0.003, because it can reject clouds fused at larger voxels. The voxel applies every time a sighting joins an object, so pass it on every call for that label, for example '{"fuse_voxel_m": 0.003}'. A cloud already fused coarser does not regain detail.
• Simulation. Sim images and MuJoCo workflows need their own params; the presets were fit on images from real rigs.

Tuning params offline on a recording, before deploying them, is in README.md.