feat(agents): replay-backed spatial goal-selection evals - #3307
Open
mc856 wants to merge 18 commits into
Open
Conversation
…scorer Data contracts (QuestionSpec/AnswerRecord/ScoreResult, six-state outcome), recording harness modules (goal + query capture in worker processes), a per-question fixture with configurable double timeouts, a pure scorer, and a shard-to-figure renderer bounded by the 75KB large-file limit.
One-time SpatialMemory ingest driven on recorded timestamps, an offline teacher that derives qualified reference locations from synchronized RGB, LiDAR and pose (gated, cross-view-verified, location-linked), and a question generator that applies an auditable human-review overlay. Ships the go2_bigoffice reference table: 26 qualified labels on 19 locations, 6 questions after review.
…vals Synthetic default-lane units for contracts, scorer, questions, teacher helpers and renderer; a self_hosted 2x2 live sweep (2 models x 2 system prompts, one shard per case, harness failures assert rather than score); and a keyless self_hosted full-chain smoke driving a recorded transcript through ingest, the shipping navigation skill, scoring and the figure. Harden read_shard to reject duplicated questions in one shard.
Threshold now equals the teacher's observation envelope plus margin (the shipping goal is a viewpoint by construction) with an enforced passability gate; pass/no-prediction rates divide by agent-attributable outcomes only, broken measurements reported separately; cross-file shard duplicate guard; XY location linkage; rename-stable question ids; the sweep baseline is the shipping system prompt; projection regression tests; review crops committed as the audit trail.
Per-question thresholds with an enforced passability gate; a confusability gate that refuses questions whose pass radius contains another question's teacher viewpoints (question set: 4, conflict-free); crops overlay projected inliers and the reference position; run_id on every record with cross-file duplicate detection; post-hoc retrieval observability (deterministic re-query recorded per question, diagnostic only); rates divide by agent-attributable outcomes; store-identity guard before coordinator builds; funnel identity holds on the no-lidar path; view independence requires a spatial baseline.
Multi-frame context verification showed the 'elevator door' reference sitting on the blank wall beside a shutter surface (bbox contamination that single-crop review missed twice) — question dropped, set is 3. Review crops now pick the sharpest LiDAR-supported frame within ±1s of a contributing view (blur tracks yaw rate, not speed) and ship in pairs: identification frame plus the unchanged measurement frame. Crop rendering split into crops.py (file-size limit). Smoke question re-recorded against the storage-boxes question.
README for the committed artifacts (what each file is, how the review was done, how to reproduce); record the cleaning-robot identification from the full-video review pass.
One reference sub-directory per recording, discovered by globbing questions.jsonl — adding a recording is adding a directory, with no harness or test edits. Single DIMOS_EVAL_STORE_ROOT convention (the --dataset argument, the reference directory and the chroma collection share the same dataset name; the store path derives from it); per-dataset skip independence in the sweep (now dataset x prompt x model). Ships go2_short (repo default recording): 14 qualified labels, 3 questions after review; one question's review images are omitted because every candidate frame shows an identifiable person — reproducible locally instead.
… provenance Two guards were missing. A store root that exists but holds no ingested dataset now fails instead of producing a sweep of clean skips that reads as green; the renderer refuses shards whose question ids span two recordings, which previously pooled two rooms' errors into one row with nothing on the figure saying so. Documentation corrected where it over-claimed. The smoke's model fixture is a hand-maintained minimal transcript, not a capture of a live turn, and now says so; the crop LiDAR gate is one-sided (a surface behind the marker passes) and certifies geometry, never object identity; the shard duplicate guard does not catch a crashed-and-restarted sweep, which gets a fresh run_id; the retrieval read-back swallows exceptions but is bounded by the module RPC timeout rather than unbounded; a harness_error is a BROKEN outcome, not an attributable one. Reference README now states what manifest.json records and separates mechanically reproducible positions from recorded human review verdicts. Three location-group ids in go2_short's review notes were wrong against its own manifest. Crops are now inventoried by a test, cv2 is imported inside the functions that use it, and the section-marker comments that failed codebase_checks are gone.
DIMOS_EVAL_MODEL_IDS names the arms, comma-separated; unset keeps the committed default pair. The default is fixed so that "the eval" names one measurement, not so that nobody else can measure anything: _init_model resolves any LangChain provider prefix, so a machine with no OpenAI credentials can now sweep its own models instead of editing a test file. Nothing downstream needs telling. Model identity is already carried end-to-end — model_id on every shard line, in every shard filename, and the key the figure groups by — so an overridden run cannot be read as, or appended to, a default one. Set-but-empty raises rather than falling back to the default, the same rule resolve_ingested_store applies to a set-but-absent store root: a run that quietly measured something other than what was asked for is the one failure a shard cannot expose, because it looks exactly like a default run.
dimensionalOS#3275 moved dimos/perception/spatial_perception.py under perception/experimental/; update the two lazy imports in the eval ingest and store fixture to the new path.
…tree *.jpg is LFS-routed and the project's LFS endpoint does not take anonymous uploads from outside contributors; the crop pairs remain one documented --crops command away, and nothing in the eval reads them.
mc856
requested review from
Dreamsorcerer,
leshy,
mustafab0,
paul-nechifor and
spomichter
as code owners
July 31, 2026 10:05
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #3307 +/- ##
==========================================
- Coverage 75.03% 75.01% -0.02%
==========================================
Files 1135 1154 +19
Lines 108631 111749 +3118
Branches 9787 10060 +273
==========================================
+ Hits 81507 83828 +2321
- Misses 24334 25116 +782
- Partials 2790 2805 +15
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 9 files with indirect coverage changes 🚀 New features to boost your workflow:
|
The all_blueprints registry scan excludes /testing/ paths (the same convention as dimos/agents/testing/); the eval's observation-only modules are test scaffolding and do not belong in the module registry.
dimos/codebase_checks/test_no_init_files forbids __init__.py below the root; dimos/agents/testing/ has none either.
…penAI The skip was a static skipif_no_openai marker, so a machine sweeping only local models (DIMOS_EVAL_MODEL_IDS=ollama:...) was skipped for a key it would never send -- contradicting the module's own docs. The condition now mirrors mcp_client._init_model's routing: bare ids and openai:-prefixed ids need the key, other provider prefixes do not.
Rows are keyed by (model_id, prompt_id), but only prompt_sha256 proves which prompt text ran. Two sweeps taken across a system-prompt edit share the id and differ in the hash; pooled, they rendered as repeated measurement of one configuration -- hiding exactly the change this eval exists to surface. render_figure now fails loudly, like the existing mixed-dataset guard.
write_crops re-built default GateParams instead of taking the caller's, so a teacher run with a non-default --front-z or --lidar-tolerance would review crops rendered under different projection and sweep tolerance than the measurement. The gates are now a required parameter; the shipped reference sets were generated with defaults and are unaffected.
render.dataset_of recovers a dataset from a question id as its first two hyphen tokens, so a one-token dataset name splits every id a token early and a nested name (go2_short_v2) collides with its own prefix -- either way the mixed-input guard fails silently. dataset_names now refuses names that do not slug to exactly two tokens, at the single place a recording is added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contribution path
Problem
Behavior-level spatial evaluation today is an e2e test with hand-measured hardcoded coordinates (
dimos/e2e_tests/test_dimsim_spatial_memory.py). Adding a question means measuring by hand, and after a prompt or model change there's no repeatable way to tell whether object-finding got better or worse.Solution
An offline "teacher" derives object reference locations from a recording the robot already made — RGB semantics from an open-vocabulary detector, geometry from time-aligned LiDAR and pose. A pytest harness then scores the goal the shipping
navigate_with_textpath selects, with navigation execution stubbed. Everything lives in one new directory (dimos/agents/evals/, no existing file changes), following the landed precedent of evals colocated with the subsystem they test (dimos/navigation/nav_3d/evaluator/,dimos/control/benchmarking/).passedmeans "the robot ends up where the object is observable."navigate_with_textbuilds its goal from the retrieved frame's robot pose (_get_goal_pose_from_result,dimos/agents/skills/navigation.py:267) — a viewpoint, not an object position — so each question's threshold is its own nearest teacher-viewpoint distance + 0.5 m, capped at 3.5 m, and the fine-grained signal lives in the continuouserror_m. Every record also carries the retrieved frame's pose and match distance (post-hoc, diagnostic only) — the observable that instance-level scoring needs next.go2_bigoffice; 214 → 14 → 3 ongo2_short. The generator refuses any question unpassable on the teacher's own evidence and any pair whose pass radius contains another question's viewpoints — in a dense aisle a wrong-object retrieval would otherwise score as a pass. Human review cuts the rest. Full funnels and location-group tables ship in the manifests; per-label verdicts and confusability distances live in the review overlay.reference/<dataset>/review.json: verified / renamed / dropped, with reasons), reviewed on rendered crop pairs — the measurement frame plus a sharper neighboring frame for identification. This layer caught real failures: a moving object, and one reference whose point-cloud median had been pulled onto a blank wall by bbox contamination — it survived single-crop review twice and only fell to multi-frame context, which is why the crop tooling renders both frames.reference/<dataset>/holds refs, manifest, review, questions and crops; the sweep discovers datasets by globbing*/questions.jsonl, so adding a recording is adding a directory. The second one (go2_short, the repo's default recording) went through the identical protocol, and an independent rerun reproduced itsrefs.jsonlbyte-identically.The eval can't be passed without the memory actually working: goals only come from the shipping retrieval path, "no prediction" is a scored outcome, and broken measurements (harness/timeout/tool errors) are excluded from the rates and reported separately.
Review evidence
The five publishable identification/measurement pairs, one column per question — top row: the sharpest supported neighboring frame with the derived reference (magenta) and LiDAR inliers (yellow); bottom row: the actual detection frame with the detection box (green) and in-box inliers. Rendered by
teacher.py --cropsfrom the recordings and composited into one sheet: program output, not a hand- or AI-drawn diagram. The pairs live in this description rather than the tree —*.jpgis LFS-routed and the project's LFS endpoint doesn't take anonymous uploads from outside contributors. The sixth question (go2_short/ office-desk) has no publishable frames: every candidate contains an identifiable person.Relationship to existing eval work in the tree
misc/DimSim/evals/is the repo's documented eval authoring surface for authored sim scenes, and #3249 extends it with closed-loop agent runs. This PR is the complementary boundary: recorded real-sensor data instead of authored scenes, derived references instead of author-placed objects, goal selection instead of closed-loop execution. Result records mirror DimSim'sEvalSuccessshape ({passed, reason, score}) with one semantic difference: DimSim'sscoreis a continuous lower-is-better distance, here it's binary and the distance lives inerror_m— the two shouldn't be aggregated blindly. #2989 (video spatial-relation QA) sits on a different data plane and subject. Conceptually this is the position-question base layer that #1913's "point placement" line would sit on.What's in the diff
Offline tools (CLI, run once per recording):
teacher.py— samples RGB on recorded timestamps (skipping near-black frames), runs open-vocabulary detection, projects the nearest LiDAR sweep withpose ∘ BASE_TO_OPTICAL+ fisheye intrinsics, gates the points (ground rejection, point count, depth IQR, observation envelope), clusters per label, and qualifies a cluster only on views with ≥ 0.3 m of real spatial baseline between them. A label survives only if exactly one of its clusters qualifies; surviving labels on one spot are linked into alocation_group. Every threshold is a flag echoed intomanifest.jsonalongside the funnel, location-group table, camera/extrinsics, detector settings and timing; the same recording and flags reproducerefs.jsonlbyte-identically on one machine.questions.py— applies the human-review overlay (silence means dropped); refuses duplicate names, two questions on one location group, unpassable questions, confusable pairs.ingest.py— fills the SpatialMemory store the eval answers from, backs the saved pickle up read-only.Harness and scoring:
contracts.py—QuestionSpec/AnswerRecord/ScoreResult, the six-state outcome,RetrievalRecord(the post-hoc retrieval observable).testing/modules.py— observation-only modules: goal-recording navigation stub, a skill container that records the exactquerystrings before delegating to the unwrapped shipping body (a parity test pins its skill metadata to the shipping decorator), fake sensor publishers, a test runner with configurable waits.conftest.py— one question, one fresh coordinator, always torn down; the shippingSpatialMemoryattached read-only with per-run disposable artifacts; a store-identity guard fails fast on a missing or empty collection. Deployed tools are navigation + server admin only (navigate_with_text,stop_navigation,tag_location,agent_send,list_modules,server_status) — narrower than a full robot blueprint, which matters when reading the tool-routing numbers.scorer.py— pure functions; rates divide by agent-attributable outcomes only, broken counts reported alongside; every record carriesrun_id(duplicates within one run are rejected).render.py— shard directory → one figure (aggregating across runs), hard-failing over the 75 KB limit.Committed artifacts, one directory per recording under
reference/:refs.jsonl(26 + 14 qualified labels on 19 + 12 locations),manifest.json,review.json,questions.jsonl(3 + 3). The review crops (an identification/measurement pair per question) ship out-of-band:*.jpgis LFS-routed and anonymous upload to the project's LFS endpoint returns 401 for outside contributors, so they regenerate locally instead (one--cropscommand, documented inreference/README.md) — happy to push the originals as soon as there's a flow for it; the five publishable pairs are attached as one sheet under Review evidence above. Onego2_shortquestion has no publishable frames at all: every candidate contains an identifiable person.How to Test
CI (this PR): the synthetic unit layer runs by default — 195 tests, ~1 s, no data, no network; two skip while the review crops ship out-of-band. Both self-hosted lanes skip fork PRs, so the recording-backed layers show as skipped here; commands to run them locally:
Full chain, no API key (ingests a thin replay slice, replays a frozen transcript, scores and renders):
Live sweep (needs
OPENAI_API_KEYfor the default arms; 8 cases, ~21 min observed). The arms default togpt-5.6-luna+openai:gpt-5.6-sol, which is what any reported numbers should be measured on;DIMOS_EVAL_MODEL_IDSnames your own instead (comma-separated, anythingmcp_client._init_modelresolves — so a machine with no OpenAI credentials can sweep e.g.ollama:qwen3:8b).OPENAI_BASE_URLis honoured too, but point it somewhere serving a different model and the arm label stops matching what answered — name the real model inDIMOS_EVAL_MODEL_IDSso the shards stay attributable:Sample results from one live sweep of the default arms (run via an OpenAI-compatible endpoint, after verifying it serves both request surfaces and rejects nonexistent model ids): all 8 configurations routed correctly — n_pred 3/3, no_prediction 0, broken 0 — and the four arms produced identical goals question-for-question, the Responses+reasoning arm included. Per recording, 2/3 questions land inside their thresholds; the two misses (4.54 m against a 2.00 m threshold, 3.99 m against 3.05 m) are identical across arms — a retrieval-layer floor, not a model difference. One case hit the fixture's 180 s answer timeout on its first run and passed a single-case rerun; the broken shard is excluded from the rates and kept aside, which is the scorer's contract doing its job. The dashed line in both panels is the 3.5 m threshold cap — each question's own threshold lives in its record. The figure is
render.pyoutput rendered from the shard data, composited side by side — program output, not a hand- or AI-drawn diagram.One shard line, as written (the
go2_bigofficeorganization miss —retrievalsis the post-hoc observable saying where the goal came from):{"kind": "answer", "data": {"error": null, "goal_x": -3.359025, "goal_y": -4.088011, "goal_yaw": -0.15469104442073522, "model_id": "openai:gpt-5.6-sol", "n_goals": 1, "outcome": "predicted", "prompt_id": "shipping", "prompt_sha256": "f146c8e3e4f7d87d217ce01a80553435ad4804d1bdb32df7df0066a70201a446", "question_id": "go2-bigoffice-organization", "retrievals": [{"distance": 0.7343969941139221, "query": "the stack of storage boxes", "x": -3.359025, "y": -4.088011}], "run_id": "20260801T010525-25561", "tool_invoked": true, "tool_queries": ["the stack of storage boxes"], "wall_time_s": 13.826804374984931}}Non-goals
presence/count questions (needs student-side aggregation tools — the recorded retrieval observable is the first rail for this) · room-level questions (#1913, needs a room representation) · instance-identity pass criteria (needs validation of the retrieval observable shipped here) · LLM judging (needs free-text answer space) · sweep runner / result store / dashboard (needs multi-round trend tracking) · DimSim/sim variant (documented next step) · upstream fixes split out as follow-ups (runner timeout config;
SpatialMemory.stop()double-save; detector mask letterbox offset).Limitations
SpatialMemory's deterministic CLIP top-1, so a model's influence is confined to whether it calls the tool, what query text it sends, and how often — read the sweep as a tool-routing and query-formulation comparison, not a ranking of spatial ability. In the one live sweep so far (n = 1 per configuration) the four arms produced identical goals question-for-question — the Responses+reasoning arm included — consistent with routing being saturated on these questions and the deterministic retrieval fixing everything downstream; repeated runs underrun_idare how that reading gets tested.run_id). Of the labels review dropped, over half are validity exclusions (dynamic objects, identical instances, confusable neighbors — no unique answer exists), most of the rest are unverifiable (dim or motion-blurred frames), and exactly one was a caught reference error. Lighting is the dominant yield factor across these two recordings — the bright 60 sgo2_shortyielded proportionally more. A larger detector variant (yoloe-11l) was measured and rejected: 26 → 17 / 14 → 8 qualified labels, no meaningful new coverage, 2.2× the detector runtime — the 11s default stands.go2_bigoffice's elevator lobby at t+4–8 s: detected, zero supported measurements).navigate_with_textcall triggers ChromaDB's default-EF download (~80 MB) for the tagged-location lookup; self-hosted runners need that cached or egress to chroma-onnx-models.s3.amazonaws.com.AI assistance
Implemented with Claude Code (Fable 5). The review overlay verdicts are mine, from the review evidence (committed crops plus the multi-frame and full-video checks documented in
reference/).Checklist