Skip to content

Evaluate image goals with Jev against native PiPER physical state - #23

Open
Tarzelf wants to merge 5 commits into
feat/graspability-v03from
feat/image-goal-benchmark-v01
Open

Tarzelf wants to merge 5 commits into
feat/graspability-v03from
feat/image-goal-benchmark-v01

Conversation

@Tarzelf

@Tarzelf Tarzelf commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

This adds a reproducible image-goal diagnostic for the native PiPER workstation and publishes the first measured Jev 1.14.0 run. The fixed cohort has 32 authored states evaluated with RGB alone and RGB plus native telemetry; four settled/moving pairs have identical image-only inputs. Every request, failure and abstention remains in the record.

All 64 frozen requests returned valid responses without retries. Observability-aware verdict accuracy was 24/32 for RGB and 25/32 with telemetry; the reference labels differ between conditions. Both had zero false completion on 24 physically incomplete states, but telemetry recognized only 1/8 actual complete states. Overall latency was 1.776 s p50 and 3.287 s p95. These are passive hosted judgments, not a learned robot controller or real-time actuator loop.

The implementation includes native fixtures, a frozen evaluation plan, credential-safe client, independent evidence verifier, scorer, case viewer and append-only publication tooling. The viewer prominently reports positive-case recall alongside false completion. The original protocol remains at image-goals/v01; the measured follow-up is published separately at image-goals/v02.

Validation: the original 51 targeted checks passed; the changed viewer and publisher pass 24 tests (11 + 13). An independent audit reconstructed all 64 requests, matched every ledger entry/returned model/usage field, and recomputed the metrics. All three frozen core files and all fixture hashes remain unchanged. The exact credential scan found zero matches in 1,677 tracked and run files; all six committed files passed the publication credential-pattern scan. Credentials and raw service responses are excluded from public artifacts.

The shoe is a narrowed 30 g rigid proxy in four correlated layouts. Privileged telemetry is not a demonstrated visual estimator, and this diagnostic establishes no hardware transfer or production reliability.

@Tarzelf Tarzelf changed the title Add image-goal verification fixtures and frozen Jev benchmark Evaluate image goals with Jev against native PiPER physical state Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant