From 19acf0734d63ecebc07140c8d84302a54baee4c4 Mon Sep 17 00:00:00 2001 From: DVIDIA Date: Sun, 18 Oct 2026 23:31:57 -0400 Subject: [PATCH 1/5] Add frozen PiPER image-goal fixtures and Jev evaluation protocol --- .github/workflows/image-goal.yml | 24 ++ docs/image-goal-benchmark.md | 112 ++++++ pyproject.toml | 1 + src/dvidia_training/image_goal_benchmark.py | 322 ++++++++++++++++ src/dvidia_training/image_goal_scenes.py | 394 ++++++++++++++++++++ src/dvidia_training/jev_client.py | 375 +++++++++++++++++++ tests/test_image_goal_benchmark.py | 92 +++++ tests/test_image_goal_freeze.py | 180 +++++++++ tests/test_image_goal_scenes.py | 127 +++++++ tests/test_jev_client.py | 209 +++++++++++ 10 files changed, 1836 insertions(+) create mode 100644 .github/workflows/image-goal.yml create mode 100644 docs/image-goal-benchmark.md create mode 100644 src/dvidia_training/image_goal_benchmark.py create mode 100644 src/dvidia_training/image_goal_scenes.py create mode 100644 src/dvidia_training/jev_client.py create mode 100644 tests/test_image_goal_benchmark.py create mode 100644 tests/test_image_goal_freeze.py create mode 100644 tests/test_image_goal_scenes.py create mode 100644 tests/test_jev_client.py diff --git a/.github/workflows/image-goal.yml b/.github/workflows/image-goal.yml new file mode 100644 index 0000000..1400bbe --- /dev/null +++ b/.github/workflows/image-goal.yml @@ -0,0 +1,24 @@ +name: Image-goal verification +on: + push: + pull_request: + workflow_dispatch: +permissions: + contents: read +jobs: + offline-contracts: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 + with: + persist-credentials: false + - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 + with: + python-version: '3.12' + - run: python -m pip install . + - name: No-key protocol, client, scoring and report checks + run: >- + python -m unittest tests.test_jev_client tests.test_image_goal_benchmark + tests.test_image_goal_freeze tests.test_image_goal_scenes + tests.test_image_goal_report -v diff --git a/docs/image-goal-benchmark.md b/docs/image-goal-benchmark.md new file mode 100644 index 0000000..1406a2c --- /dev/null +++ b/docs/image-goal-benchmark.md @@ -0,0 +1,112 @@ +# Image-goal verification with Jev + +DVIDIA research · protocol v0.1 · 18 October 2026 + +This benchmark asks whether an image-goal evaluator can distinguish a finished +placement from a scene that merely looks finished. It runs in our native PiPER +workstation and uses **Jev 1.14.0 by TypeSafe** as an optional hosted evaluator. +It does not train a robot, produce actuator commands, or install Jev locally. + +## What the experiment measures + +Four authored layouts each contain eight conditions: completed placement, +missing target, wrong object, wrong position, wrong orientation, contact without +release, motion with an identical rendered pose, and an occluded target. +The resulting 32 cases are evaluated twice: current/goal RGB images alone, +and the same images with privileged simulator measurements. These are correlated +diagnostic cases, not 32 independent real-world tasks or a held-out deployment +success estimate. + +The physical rubric requires the intended green shoe within 25 mm in the table +plane and 15 degrees in yaw, supported by the table, released, and stationary +for 200 ms. Exact force and speed thresholds are in the frozen protocol. The +ground truth is computed from native state and contact observations every 2 ms. +The telemetry condition supplies numeric observations and extrema over the +complete native window, without a completion label. Object identity and pose +errors are privileged simulator information, not a validated vision estimator. + +The shoe is an explicitly declared 30 g rigid proxy with half the catalog shoe +width. The PiPER drives are unchanged. This enables a real finger-contact +negative control; it is not a demonstration of full-size shoe grasping, +deformation or hardware transfer. State fixtures are initialized deliberately; +the moving control changes native velocity instantaneously without claiming a +physically applied impulse or successful manipulation trajectory. + +The completed and moving fixtures have byte-identical images in each layout. +RGB alone therefore cannot distinguish their physical completion labels. +The correct evidence-aware response is **unknown** when hidden velocity, +support, release or dwell cannot be established. We separately report physical +false completion, observability-aware accuracy, justified abstention, decision +coverage, presence errors on absent targets, and probability Brier scores. +An always-abstaining system has zero false completion but zero completion +coverage; neither number should be presented alone. + +## Reproduce locally + +Use Python 3.12 or newer, MuJoCo 3.15.0 and the repository's pinned dependencies. +Native fixture rendering needs a working OpenGL context. On a headless Linux +server, configure EGL or OSMesa for your environment. A discrete GPU is not a +requirement for the simulator physics, but rendering support is required. +Measured time on one host is not a minimum hardware specification. + +```sh +python -m pip install '.[arm]' +dvidia-robot-sim fetch piper --output robot-assets +dvidia-image-goal generate --assets robot-assets --output runs/image-goal-fixtures +dvidia-image-goal freeze --fixtures runs/image-goal-fixtures --source-commit "$(git rev-parse HEAD)" +dvidia-image-goal evaluate --fixtures runs/image-goal-fixtures --output runs/image-goal-evaluation +``` + +Configure `TYPESAFE_API_KEY` securely in the process environment before the final +command. Do not commit a key, put it in a shell argument, or include it in a +benchmark artifact. On macOS, a Keychain item can be selected by nonsecret names: + +```sh +dvidia-image-goal evaluate --fixtures runs/image-goal-fixtures \ + --output runs/image-goal-evaluation \ + --keychain-service dvidia-typesafe --keychain-account dvidia-research +``` + +Without credentials, the runner makes no network requests and writes explicit +unrun trials. Accuracy, latency and cost remain unmeasured. Use a fresh output +directory for each evaluation; existing evidence cannot be overwritten. + +The evaluation seals images, labels, prompts, implementation hashes and the full +paired trial inventory before calls. It prevalidates every request, executes +at most 64 requests serially with no automatic retries, and retains all results +and errors. Each request uses a 20-second socket-operation timeout; this is not +a hard wall-clock or robotics deadline. A checkpoint always preserves the full +scheduled inventory, including unattempted trials after interruption. + +Latency includes request preparation, image upload, network/service delay and +response parsing from preloaded PNGs. It excludes fixture loading and rendering, +camera capture, actuation and physical task execution. Report successful-request +latency separately from all attempts. p99 from a small cohort is descriptive, +not a reliable tail-latency guarantee. Missing responses never become correct +negative predictions, and unknown billed usage remains explicitly unknown. + +## Jev integration and licensing + +The client uses the documented multipart API, the experimental image header, +two Noul probabilities and one three-way Choice verdict. Presence thresholds +are fixed at 0.2 and 0.8 before evaluation. There is no bounding-box grasp path: +TypeSafe's Box primitive alone has no absent-object answer. + +Our code is MIT under the repository license. PiPER assets retain their pinned +upstream notices. Jev is a separately operated hosted service; no Jev weights +are included. Its current service terms restrict model imitation/distillation. +This experiment evaluates decisions and does not use service outputs to train +a replacement model. Only authored simulator images and observations are sent. + +Primary references, checked 18 October 2026: + +- [Jev 1.14 release](https://typesafe.ai/blog/jev-1-14) +- [Experimental image API and limits](https://docs.typesafe.ai/images) +- [Noul probabilities](https://docs.typesafe.ai/primitives/noul) +- [Choice decisions](https://docs.typesafe.ai/primitives/choice) +- [Models and pricing](https://docs.typesafe.ai/models) +- [Service terms](https://typesafe.ai/legal/mca) + +Published model results must identify the frozen source, case manifest, protocol, +request inventory and returned model version. A valid local fixture run or an +offline client test is not a measured Jev result. diff --git a/pyproject.toml b/pyproject.toml index 7e2264e..a6b6014 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -51,6 +51,7 @@ dvidia-piper-reliability = "dvidia_training.piper_reliability_benchmark:main" dvidia-assets = "dvidia_training.asset_cli:main" dvidia-workstation = "dvidia_training.workstation:main" dvidia-workstation-benchmark = "dvidia_training.workstation_benchmark:main" +dvidia-image-goal = "dvidia_training.image_goal_benchmark:main" dvidia-prerequisites = "dvidia_training.prerequisite_benchmark:main" dvidia-grip-benchmark = "dvidia_training.grip_benchmark:main" dvidia-graspability-benchmark = "dvidia_training.graspability_benchmark:main" diff --git a/src/dvidia_training/image_goal_benchmark.py b/src/dvidia_training/image_goal_benchmark.py new file mode 100644 index 0000000..edda412 --- /dev/null +++ b/src/dvidia_training/image_goal_benchmark.py @@ -0,0 +1,322 @@ +"""Frozen, bounded hosted-model evaluation against independent simulator labels. + +This evaluates observation decisions. It does not command a robot or train a +policy. Prerecorded rendering time and hosted request time remain separate. +""" +from __future__ import annotations + +import argparse +from collections import Counter +from datetime import datetime, timezone +from hashlib import sha256 +import json +import math +from pathlib import Path +import platform +import random +import time + +CONDITIONS = ("rgb", "rgb_telemetry") +MODEL = "jev-1.14.0" +MAX_REQUESTS = 64 + + +def json_bytes(value): + return (json.dumps(value, indent=2, sort_keys=True, allow_nan=False) + "\n").encode() + + +def read_json(path): + return json.loads(Path(path).read_text()) + + +def digest(path): + raw = Path(path).read_bytes() + return {"sha256": sha256(raw).hexdigest(), "bytes": len(raw)} + + +def relative_file(root, name): + if not isinstance(name, str) or not name or Path(name).is_absolute(): + raise ValueError("Expected a relative artifact path") + root = Path(root).resolve() + path = (root / name).resolve() + if not path.is_relative_to(root) or not path.is_file(): + raise ValueError("Artifact must be a file inside the fixture directory") + return path + + +def validate_cases(document): + cases = document.get("cases") + if not isinstance(cases, list) or not 1 <= len(cases) <= MAX_REQUESTS // 2: + raise ValueError("Expected 1–32 cases") + ids = set() + for case in cases: + identity = case.get("id") + if not isinstance(identity, str) or not identity or identity in ids: + raise ValueError("Case identities must be unique strings") + ids.add(identity) + truth = case.get("physical_truth", {}) + if any(type(truth.get(key)) is not bool for key in ("present", "complete")): + raise ValueError("Independent presence and completion labels are required") + if truth["complete"] and not truth["present"]: + raise ValueError("An absent target cannot be complete") + if not isinstance(case.get("measured_inputs"), dict): + raise ValueError("Measured inputs must be explicit") + for condition in CONDITIONS: + if case.get("expected_verdict", {}).get(condition) not in {"complete", "incomplete", "unknown"}: + raise ValueError("Observability-aware verdict labels are required") + if case.get("expected_presence", {}).get(condition) not in {"present", "absent", "unknown"}: + raise ValueError("Observability-aware presence labels are required") + return cases + + +def freeze(root, *, task, source_commit): + """Seal images, labels, semantic requests and evaluation rules before calls.""" + from .jev_client import build_goal_request + root = Path(root) + target = root / "evaluation-plan.json" + if target.exists(): + raise FileExistsError("A frozen evaluation plan cannot be overwritten") + cases = validate_cases(read_json(root / "cases.json")) + files = {name: digest(root / name) for name in ("cases.json", "protocol.json")} + for name in ("integrity.json", "protocol.sha256"): + if (root / name).is_file(): + files[name] = digest(root / name) + trials = [] + for case in cases: + current = relative_file(root, case["current_image"]) + goal = relative_file(root, case["goal_image"]) + files[case["current_image"]] = digest(current) + files[case["goal_image"]] = digest(goal) + if case.get("state_file"): + files[case["state_file"]] = digest(relative_file(root, case["state_file"])) + for condition in CONDITIONS: + prepared = build_goal_request(current.read_bytes(), goal.read_bytes(), task=task, + measured_state=case["measured_inputs"] if condition == "rgb_telemetry" else None) + trials.append({"case_id": case["id"], "condition": condition, + "request_sha256": prepared.request_sha256}) + random.Random(20261018).shuffle(trials) + source_files = {} + for name in ("image_goal_benchmark.py", "image_goal_scenes.py", "jev_client.py"): + source_files[name] = digest(Path(__file__).with_name(name)) + plan = {"schema": "dvidia.image-goal-evaluation-plan/v1", "model": MODEL, + "frozen_at": datetime.now(timezone.utc).isoformat(), "source_commit": source_commit, + "task": task, "conditions": list(CONDITIONS), "trials": trials, "files": files, + "source_files": source_files, "max_requests": MAX_REQUESTS, + "timeout_s": 20., "timeout_scope": "Per socket operation; not a hard wall-clock deadline.", + "automatic_retries": 0, "presence_thresholds": [.2, .8], + "completion_decision": "Unmodified Jev Choice verdict; unknown is abstention.", + "documented_input_price_usd_per_million_tokens": .042, + "price_checked_date": "2026-10-18", + "planning_allowance_per_request_usd": .003, + "cost_scope": "Planning allowance assumes documented roughly 64k default context. Not a billing guarantee; receipts retain known usage and unknown-cost attempts.", + "cohort": "Authored diagnostic fixtures; four scene layouts are correlated clusters, not 32 independent real-world tasks.", + "latency_scope": "Preloaded image bytes + request construction + image upload + hosted response + parsing; excludes fixture file loading, prerecorded rendering, live camera, actuation and physical task duration.", + "rules": ["Keep every scheduled trial, including errors and unrun trials.", + "No retries, prompt tuning, threshold tuning or case selection after evaluation begins.", + "No direct actuator commands and no model training or distillation.", + "Report physical false completion separately from observability-aware accuracy.", + "No remote calls when credentials are absent; missing results are never scored as accurate."]} + target.write_bytes(json_bytes(plan)) + return plan + + +def verify_plan(root): + root = Path(root) + plan = read_json(root / "evaluation-plan.json") + if plan.get("model") != MODEL or len(plan.get("trials", [])) > MAX_REQUESTS: + raise ValueError("Invalid frozen model or request budget") + for name, expected in plan["files"].items(): + if digest(relative_file(root, name)) != expected: + raise ValueError("Frozen fixture changed: " + name) + for name, expected in plan["source_files"].items(): + if digest(Path(__file__).with_name(name)) != expected: + raise ValueError("Frozen implementation changed: " + name) + cases = validate_cases(read_json(root / "cases.json")) + expected = {(case["id"], condition) for case in cases for condition in CONDITIONS} + actual = [(row["case_id"], row["condition"]) for row in plan["trials"]] + if len(actual) != len(set(actual)) or set(actual) != expected: + raise ValueError("Frozen trial inventory differs from complete paired cohort") + return plan, cases + + +def quantiles(values): + values = sorted(float(value) for value in values if type(value) in (int, float) and math.isfinite(value)) + if not values: + return {"count": 0, "p50": None, "p95": None, "p99": None, "max": None} + def percentile(q): + position = (len(values) - 1) * q + lo, hi = math.floor(position), math.ceil(position) + return values[lo] + (values[hi] - values[lo]) * (position - lo) + return {"count": len(values), "p50": percentile(.5), "p95": percentile(.95), + "p99": percentile(.99), "max": values[-1]} + + +def fraction(numerator, denominator): + return {"numerator": numerator, "denominator": denominator, + "rate": numerator / denominator if denominator else None} + + +def score(rows, cases): + """Errors stay in scheduled denominators; abstention is not binary success.""" + by_id = {case["id"]: case for case in cases} + if len({(r["case_id"], r["condition"]) for r in rows}) != len(rows): + raise ValueError("Duplicate trial result") + for row in rows: + if row.get("case_id") not in by_id or row.get("condition") not in CONDITIONS: + raise ValueError("Unknown trial identity") + if row.get("status") not in {"ok", "error", "not_run"}: + raise ValueError("Invalid trial status") + if row["status"] == "ok": + if row.get("verdict") not in {"complete", "incomplete", "unknown"} or row.get("presence") not in {"present", "absent", "unknown"}: + raise ValueError("Invalid typed decision") + for name in ("p_present", "p_complete"): + value = row.get(name) + if type(value) not in (int, float) or not math.isfinite(value) or not 0 <= value <= 1: + raise ValueError("Invalid probability") + expected = {(case["id"], condition) for case in cases for condition in CONDITIONS} + if {(r["case_id"], r["condition"]) for r in rows} != expected: + raise ValueError("Scoring requires every scheduled case and condition, including explicit unrun rows") + def group(selected): + valid = [r for r in selected if r["status"] == "ok"] + answered = [r for r in valid if r["verdict"] != "unknown"] + incomplete = [r for r in selected if not by_id[r["case_id"]]["physical_truth"]["complete"]] + absent = [r for r in selected if not by_id[r["case_id"]]["physical_truth"]["present"]] + complete = [r for r in selected if by_id[r["case_id"]]["physical_truth"]["complete"]] + known = [r for r in selected if by_id[r["case_id"]]["expected_verdict"][r["condition"]] != "unknown"] + ambiguous = [r for r in selected if by_id[r["case_id"]]["expected_verdict"][r["condition"]] == "unknown"] + false_complete = sum(r["status"] == "ok" and r["verdict"] == "complete" for r in incomplete) + true_complete = sum(r["status"] == "ok" and r["verdict"] == "complete" for r in complete) + probability_rows = [r for r in valid if type(r.get("p_complete")) in (float, int)] + brier = (sum((r["p_complete"] - by_id[r["case_id"]]["physical_truth"]["complete"]) ** 2 + for r in probability_rows) / len(probability_rows) if probability_rows else None) + return {"scheduled": len(selected), "completed_requests": len(valid), + "statuses": dict(Counter(r["status"] for r in selected)), + "verdicts": dict(Counter(r["verdict"] for r in valid)), + "observability_aware_accuracy": fraction(sum(r["status"] == "ok" and r["verdict"] == by_id[r["case_id"]]["expected_verdict"][r["condition"]] for r in selected), len(selected)), + "known_verdict_accuracy": fraction(sum(r["status"] == "ok" and r["verdict"] == by_id[r["case_id"]]["expected_verdict"][r["condition"]] for r in known), len(known)), + "ambiguous_abstention": fraction(sum(r["status"] == "ok" and r["verdict"] == "unknown" for r in ambiguous), len(ambiguous)), + "physical_false_completion": fraction(false_complete, len(incomplete)), + "physical_false_completion_valid_requests": fraction(false_complete, sum(r["status"] == "ok" for r in incomplete)), + "physical_true_completion": fraction(true_complete, len(complete)), + "missing_object_false_presence": fraction(sum(r["status"] == "ok" and r["presence"] == "present" for r in absent), len(absent)), + "presence_accuracy": fraction(sum(r["status"] == "ok" and r["presence"] == by_id[r["case_id"]]["expected_presence"][r["condition"]] for r in selected), len(selected)), + "completion_coverage": fraction(len(answered), len(selected)), + "selective_physical_accuracy": fraction(sum((r["verdict"] == "complete") == by_id[r["case_id"]]["physical_truth"]["complete"] for r in answered), len(answered)), + "physical_completion_brier": {"count": len(probability_rows), "mean": brier}, + "online_ms_all_attempts": quantiles([r.get("online_total_ms") for r in selected]), + "online_ms_successful_requests": quantiles([r.get("online_total_ms") for r in valid]), + "request_ms_successful_requests": quantiles([r.get("latency_ms", {}).get("http") for r in valid]), + "known_usage_estimated_input_cost_usd": sum(r["estimated_input_cost_usd"] for r in selected if r.get("estimated_input_cost_usd") is not None) if any(r.get("estimated_input_cost_usd") is not None for r in selected) else None, + "unknown_cost_attempts": sum(r.get("attempts", int(r["status"] != "not_run")) > 0 and r.get("estimated_input_cost_usd") is None for r in selected)} + output = {"overall": group(rows), "by_condition": {}, "by_category": {}} + for condition in CONDITIONS: + output["by_condition"][condition] = group([r for r in rows if r["condition"] == condition]) + for category in sorted({case.get("category", "unspecified") for case in cases}): + output["by_category"][category] = group([r for r in rows if by_id[r["case_id"]].get("category", "unspecified") == category]) + return output + + +def evaluate(root, output, *, api_key=None): + from .jev_client import JevClient, build_goal_request + root, output = Path(root), Path(output) + plan, cases = verify_plan(root) + if output.exists(): + raise FileExistsError("Use a new directory; existing attempts are immutable") + by_id = {case["id"]: case for case in cases} + # Check the entire request cohort before contacting the provider. Cache the + # verified bytes so a later fixture mutation cannot alter a paid trial. + verified = {} + for trial in plan["trials"]: + case = by_id[trial["case_id"]] + current = relative_file(root, case["current_image"]).read_bytes() + goal = relative_file(root, case["goal_image"]).read_bytes() + measured = case["measured_inputs"] if trial["condition"] == "rgb_telemetry" else None + prepared = build_goal_request(current, goal, task=plan["task"], measured_state=measured) + if prepared.request_sha256 != trial["request_sha256"]: + raise ValueError("Frozen semantic request changed") + verified[(case["id"], trial["condition"])] = (current, goal, measured) + client = JevClient(api_key=api_key, timeout_s=plan["timeout_s"]) + output.mkdir(parents=True) + (output / "evaluation-plan.json").write_bytes((root / "evaluation-plan.json").read_bytes()) + rows = [{"case_id": trial["case_id"], "condition": trial["condition"], + "status": "not_run", "attempts": 0, "presence": "unknown", "verdict": "unknown", + "p_present": None, "p_complete": None, "online_total_ms": None, + "error": {"kind": "not_attempted", "message": "Scheduled trial has not run"}} + for trial in plan["trials"]] + base_report = {"schema": "dvidia.image-goal-results/v1", "model": MODEL, + "planned_conditions": list(CONDITIONS), + "evaluation_plan_sha256": digest(root / "evaluation-plan.json")["sha256"], + "environment": {"platform": platform.platform(), "python": platform.python_version()}, + "limitations": [plan["cohort"], plan["latency_scope"], + "Telemetry is privileged simulator state, not validated real sensor readings.", + "Jev is a hosted experimental image evaluator; no robot learning or local Jev inference is demonstrated."]} + def checkpoint(status, active_trial=None): + report = {**base_report, "status": status, "results": rows, + "active_trial": active_trial, + "metrics": None if any(row["status"] == "not_run" for row in rows) else score(rows, cases), + "updated_at": datetime.now(timezone.utc).isoformat()} + temporary = output / ".results.json.tmp" + temporary.write_bytes(json_bytes(report)) + temporary.replace(output / "results.json") + return report + checkpoint("running") + ledger = output / "attempts.jsonl" + for index, trial in enumerate(plan["trials"]): + case = by_id[trial["case_id"]] + start = time.perf_counter() + current, goal, measured = verified[(case["id"], trial["condition"])] + # If the process stops inside the request, this marker prevents an + # unreturned, potentially billed attempt from being called "not sent". + checkpoint("running", {"case_id": case["id"], "condition": trial["condition"], + "outcome": "Request may have reached provider; result and charge unknown until receipt is recorded."}) + start = time.perf_counter() + result = client.evaluate(current, goal, task=plan["task"], measured_state=measured) + total = (time.perf_counter() - start) * 1000 + row = {**result, "case_id": case["id"], "condition": trial["condition"], + "online_total_ms": total if result["status"] != "missing_credentials" else None, + "decision": {"present": {"present": True, "absent": False}.get(result.get("presence")), + "complete": {"complete": True, "incomplete": False}.get(result.get("verdict"))}} + if row["status"] == "missing_credentials": + row["status"] = "not_run" + row["timing_ms"] = {"total_ms": row["online_total_ms"], + "prepare_ms": row.get("latency_ms", {}).get("prepare"), + "request_ms": row.get("latency_ms", {}).get("http")} + rows[index] = row + with ledger.open("ab") as stream: + stream.write(json.dumps(row, sort_keys=True, allow_nan=False).encode() + b"\n") + stream.flush() + checkpoint("running") + print(json.dumps({"trial": index + 1, "total": len(plan["trials"]), "case_id": row["case_id"], + "condition": row["condition"], "status": row["status"]}), flush=True) + pending = any(row["status"] == "not_run" for row in rows) + return checkpoint("not_run" if pending else "completed") + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="action", required=True) + make = sub.add_parser("generate") + make.add_argument("--assets", required=True); make.add_argument("--output", required=True) + seal = sub.add_parser("freeze") + seal.add_argument("--fixtures", required=True); seal.add_argument("--source-commit", required=True) + run = sub.add_parser("evaluate") + run.add_argument("--fixtures", required=True); run.add_argument("--output", required=True) + run.add_argument("--keychain-service"); run.add_argument("--keychain-account") + args = parser.parse_args(argv) + if args.action == "generate": + from .image_goal_scenes import generate + generate(args.assets, args.output) + elif args.action == "freeze": + from .image_goal_scenes import TASK + plan = freeze(args.fixtures, task=TASK, source_commit=args.source_commit) + print(json.dumps({"frozen_trials": len(plan["trials"]), "model": plan["model"]})) + else: + if bool(args.keychain_service) != bool(args.keychain_account): + parser.error("Specify both Keychain service and account") + from .jev_client import load_api_key + key = load_api_key(keychain_service=args.keychain_service, keychain_account=args.keychain_account) + evaluate(args.fixtures, args.output, api_key=key) + + +if __name__ == "__main__": + main() diff --git a/src/dvidia_training/image_goal_scenes.py b/src/dvidia_training/image_goal_scenes.py new file mode 100644 index 0000000..101464d --- /dev/null +++ b/src/dvidia_training/image_goal_scenes.py @@ -0,0 +1,394 @@ +"""Frozen native PiPER image-goal verification fixtures. + +These are authored *state fixtures*, not robot-policy rollouts or evidence of +sim-to-real transfer. RGB cannot reveal an instantaneous velocity or prove a +contact dwell. The matched-pixel moving control makes that limitation explicit. +""" +from __future__ import annotations + +import argparse +from copy import deepcopy +from hashlib import sha256 +import json +import math +from pathlib import Path +import struct +import time +import zlib + +from . import reference_sim as sim, robot_profiles, workstation as ws + +SCHEMA = "dvidia.image-goal-scenes/v1" +TASK = ("Place the green shoe inside the outlined goal zone, matching the goal " + "image's position and orientation. Completion requires the target shoe " + "to have table support force greater than 0.01 N, robot contact force " + "at most 0.01 N, XY position error at most 25 mm, yaw error at most " + "15 degrees, linear speed at most 0.01 m/s and angular speed at most " + "0.05 rad/s, all continuously for at least 200 ms. " + "The goal picture is a visual reference, not proof that these physical " + "conditions hold in the current picture. Answer unknown when the inputs " + "cannot establish the required physical conditions. An orange shoe is " + "a different object. The opaque inspection cover can hide the target.") +CATEGORIES = ("complete", "missing_target", "wrong_object", "wrong_position", + "wrong_orientation", "unreleased", "moving_alias", "occluded") +LAYOUTS = ( + {"id": "layout-01", "xy_m": [.40, -.10], "yaw_rad": .0, "azimuth_deg": 130.}, + {"id": "layout-02", "xy_m": [.43, -.16], "yaw_rad": .08, "azimuth_deg": 145.}, + {"id": "layout-03", "xy_m": [.38, -.07], "yaw_rad": -.08, "azimuth_deg": 115.}, + {"id": "layout-04", "xy_m": [.42, -.13], "yaw_rad": .15, "azimuth_deg": 155.}, +) +CRITERIA = {"position_tolerance_m": .025, "yaw_tolerance_rad": math.radians(15), + "linear_speed_max_m_s": .01, "angular_speed_max_rad_s": .05, + "support_force_min_n": .01, "robot_force_max_n": .01, + "continuous_dwell_s": .20} +WIDTH, HEIGHT = 640, 480 + + +def _json(value): + return (json.dumps(value, indent=2, allow_nan=False) + "\n").encode() + + +def protocol(): + """Return the predeclared cases and rubric, without native imports/network.""" + return {"schema": SCHEMA, "task": TASK, "criteria": deepcopy(CRITERIA), + "layouts": deepcopy(LAYOUTS), "categories": list(CATEGORIES), + "case_ids": [f"{x['id']}-{c}" for x in LAYOUTS for c in CATEGORIES], + "image_size": [WIDTH, HEIGHT], "physics_timestep_s": ws.TIMESTEP, + "settling_s": .60, "native_observation_window_s": .20, + "unreleased_controller": {"stages": ["approach above heel", "descend", "close"], + "stage_duration_s": [3.,2.,1.2], "heel_offset_m": -.085, + "tcp_height_m": [.20,.055,.055], "jaw_command_m": [.035,.035,0.], + "native_dt_s": .002, "control_dt_s": .020, + "scope": "Authored approach then native finger contact; not a lift or learned manipulation."}, + "moving_alias_velocity_m_s": [.12, 0., 0.], + "synthetic_shoe": {"source": "packaged trainer-left collision primitives", + "width_scale": .50, "mass_kg": .030, + "visual_colors": "Green upper for target, orange upper for decoy, cream sole and toe.", + "reason": "Declared narrow rigid proxy permits unchanged PiPER finger contact; not a full-size measured shoe."}, + "scope": "Authored native physical-state verification. Four deterministic layouts, not statistical independent real-world scenes or a trained policy benchmark.", + "dwell_scope": "Every 2 ms native sample satisfies all thresholds for 100 consecutive ticks; no continuous-time or hardware certification claim.", + "reset_scope": "Each fixture starts from an authored state. The moving alias changes native velocity at the snapshot without advancing time; this is a declared state intervention, not a measured physical impulse.", + "telemetry_scope": "Privileged native state and contact history; not an implemented camera or hardware estimator.", + "rgb_observability": "Single RGB cannot prove velocity, support force, release force or 200 ms dwell. Complete, moving, occluded and unreleased states require unknown. Even visibly closed fingers do not establish the 0.01 N contact-force threshold. Missing targets, wrong objects and visible pose violations permit incomplete.", + "freeze": "Generate and hash this protocol before any model requests; never choose cases or thresholds from model results."} + + +def _validate(layout, category): + if layout not in LAYOUTS or category not in CATEGORIES: + raise ValueError("Expected a frozen image-goal layout and category") + + +def build_model(assets, layout, category="complete"): + """Build one frozen condition with pinned arm mechanics and no attachments.""" + _validate(layout, category) + mj, _ = sim._native_imports() + spec = ws._spec(assets) + shoe = spec.body(ws.SHOE) + xy = list(layout["xy_m"]) + yaw = layout["yaw_rad"] + if category == "missing_target": + spec.delete(shoe) + else: + if category == "wrong_position": + xy[0] += .17 + if category == "wrong_orientation": + yaw += math.pi / 2 + if category == "wrong_object": + shoe.name = "orange_decoy_shoe" + shoe.pos = [*xy, .012] + shoe.quat = [math.cos(yaw/2), 0, 0, math.sin(yaw/2)] + for geom in shoe.geoms: + geom.pos[1] *= .5 + geom.size[1] *= .5 + geom.mass *= .1 + if "upper" in geom.name: + geom.rgba = ([.79, .26, .07, 1.] if category == "wrong_object" + else [.12, .43, .27, 1.]) + else: + geom.rgba = [.84,.84,.76,1.] + # The zone is an explicitly non-colliding visual reference on the table. + for i, (dx, dy, sx, sy) in enumerate(((0, -.058, .16, .002), + (0, .058, .16, .002), (-.16, 0, .002, .06), (.16, 0, .002, .06))): + yaw = layout["yaw_rad"] + x = layout["xy_m"][0] + dx*math.cos(yaw)-dy*math.sin(yaw) + y = layout["xy_m"][1] + dx*math.sin(yaw)+dy*math.cos(yaw) + spec.worldbody.add_geom(name=f"goal_outline_{i}", type=mj.mjtGeom.mjGEOM_BOX, + pos=[x,y,.0006], size=[sx,sy,.0004], + quat=[math.cos(yaw/2),0,0,math.sin(yaw/2)], rgba=[.10,.27,.23,1.], + contype=0, conaffinity=0) + if category == "occluded": + spec.worldbody.add_geom(name="opaque_inspection_cover", + type=mj.mjtGeom.mjGEOM_BOX, pos=[*layout["xy_m"], .14], + size=[.24,.20,.006], rgba=[.62,.65,.62,1.], contype=1, conaffinity=1) + spec.visual.global_.offwidth = WIDTH + spec.visual.global_.offheight = HEIGHT + spec.visual.headlight.ambient = [.4,.4,.4] + spec.visual.headlight.diffuse = [.6,.6,.6] + return spec.compile() + + +def _initial_arm(assets): + model = ws.build_model(assets) + return ws.initialize(model).qpos[:8].copy() + + +def _measure(model, data, layout): + mj, np = sim._native_imports() + obj = mj.mj_name2id(model, mj.mjtObj.mjOBJ_BODY, ws.SHOE) + support = robot = 0. + contacts = [] + if obj < 0: + return {"target_present": False, "target_position_m": None, + "position_error_m": None, "yaw_error_rad": None, + "linear_speed_m_s": None, "angular_speed_rad_s": None, + "table_support_force_n": 0., "robot_contact_force_n": 0., + "contacts": [], "instantaneous_conditions_met": False} + table = mj.mj_name2id(model, mj.mjtObj.mjOBJ_GEOM, "workstation_table_top") + for i in range(data.ncon): + c = data.contact[i] + ga, gb = int(c.geom1), int(c.geom2) + a, b = int(model.geom_bodyid[ga]), int(model.geom_bodyid[gb]) + if obj not in (a,b): + continue + other_geom, other_body = (gb,b) if a == obj else (ga,a) + wrench = np.zeros(6) + mj.mj_contactForce(model, data, i, wrench) + normal = max(0., float(wrench[0])) + if other_geom == table: + support += normal + if other_body > 0 and other_body != obj: + robot += normal + if normal > .0001: + contacts.append({"other_geom": mj.mj_id2name(model,mj.mjtObj.mjOBJ_GEOM,other_geom), + "normal_force_n": normal}) + joint = int(model.body_jntadr[obj]) + dof = int(model.jnt_dofadr[joint]) + linear = float(np.linalg.norm(data.qvel[dof:dof+3])) + angular = float(np.linalg.norm(data.qvel[dof+3:dof+6])) + pos = data.xpos[obj].tolist() + rotation = data.xmat[obj].reshape(3,3) + yaw = math.atan2(float(rotation[1,0]),float(rotation[0,0])) + yaw_error = abs(math.atan2(math.sin(yaw-layout["yaw_rad"]), math.cos(yaw-layout["yaw_rad"]))) + position_error = float(np.linalg.norm(np.asarray(pos[:2])-layout["xy_m"])) + c = CRITERIA + passed = (position_error <= c["position_tolerance_m"] and yaw_error <= c["yaw_tolerance_rad"] + and linear <= c["linear_speed_max_m_s"] and angular <= c["angular_speed_max_rad_s"] + and support > c["support_force_min_n"] and robot <= c["robot_force_max_n"]) + return {"target_present": True, "target_position_m": pos, + "position_error_m": position_error, "yaw_error_rad": yaw_error, + "linear_speed_m_s": linear, "angular_speed_rad_s": angular, + "table_support_force_n": support, "robot_contact_force_n": robot, + "contacts": contacts, "instantaneous_conditions_met": bool(passed)} + + +def observe(model, data, layout, history): + """Independent physical rubric; no image/model output participates in truth.""" + measured = _measure(model,data,layout) + count = 0 + for row in reversed(history): + if not row["instantaneous_conditions_met"]: + break + count += 1 + # Current snapshot must itself satisfy the rubric, including alias velocity. + dwell = count*float(model.opt.timestep) if measured["instantaneous_conditions_met"] else 0. + truth = {"present": measured["target_present"], + "complete": bool(measured["instantaneous_conditions_met"] and dwell+1e-12 >= CRITERIA["continuous_dwell_s"]), + "physical_conditions_dwell_s": dwell, + "position_error_m": measured["position_error_m"], "yaw_error_rad": measured["yaw_error_rad"], + "supported": measured["table_support_force_n"] > CRITERIA["support_force_min_n"], + "released": measured["robot_contact_force_n"] <= CRITERIA["robot_force_max_n"], + "linear_speed_m_s": measured["linear_speed_m_s"], "angular_speed_rad_s": measured["angular_speed_rad_s"]} + # No completion label or verdict leaks into the measured-input condition. + inputs = {key: measured[key] for key in ("target_present", "target_position_m", + "position_error_m", "yaw_error_rad", "linear_speed_m_s", "angular_speed_rad_s", + "table_support_force_n", "robot_contact_force_n")} + inputs["observation_window_s"] = len(history)*float(model.opt.timestep) + inputs["native_timestep_s"] = float(model.opt.timestep) + inputs["native_observation_count"] = len(history) + inputs["target_present_observation_count"] = sum(row["target_present"] for row in history) + inputs["native_window_extrema"] = {} + for key in ("position_error_m", "yaw_error_rad", "linear_speed_m_s", + "angular_speed_rad_s", "table_support_force_n", "robot_contact_force_n"): + values = [row[key] for row in history if row[key] is not None] + inputs["native_window_extrema"][key] = {"min": min(values) if values else None, + "max": max(values) if values else None} + return truth, inputs, measured + + +def _native_state(assets, layout, category, arm_qpos): + mj, np = sim._native_imports() + model = build_model(assets, layout, category) + data = mj.MjData(model) + data.qpos[:8] = arm_qpos + data.ctrl[:6] = data.qpos[:6] + data.ctrl[6] = .035 + mj.mj_forward(model,data) + history = [] + def tick(): + mj.mj_step(model,data) + mj.mj_forward(model,data) + if any(w.number for w in data.warning) or not np.isfinite(data.qpos).all(): + raise RuntimeError("Native fixture emitted a solver warning or non-finite state") + history.append(_measure(model,data,layout)) + del history[:-100] + for _ in range(300): + tick() + if category == "unreleased": + # A real approach and close through unchanged position actuators. The + # shoe stays supported; this fixture tests incomplete release, not lift. + yaw = layout["yaw_rad"] + target = [layout["xy_m"][0]-.085*math.cos(yaw), + layout["xy_m"][1]-.085*math.sin(yaw), .055] + stages = (([*target[:2], .20], .035, 1500), + (target, .035, 1000), (target, 0., 600)) + for stage_target, jaw, count in stages: + for step in range(count): + if step%10 == 0: + desired = ws.controller_action(model,data,stage_target,jaw) + data.ctrl[:6] += np.clip(desired[:6]-data.ctrl[:6],-.003,.003) + data.ctrl[6] += float(np.clip(desired[6]-data.ctrl[6],-.0001,.0001)) + tick() + if category == "moving_alias": + obj = mj.mj_name2id(model,mj.mjtObj.mjOBJ_BODY,ws.SHOE) + dof = int(model.jnt_dofadr[int(model.body_jntadr[obj])]) + data.qvel[dof:dof+3] = [.12,0,0] + mj.mj_forward(model,data) + # Instantaneous post-impulse reset fixture: no time has passed since the + # velocity intervention. Preserve exact pose/pixels, invalidate dwell. + history = [] + truth, inputs, measured = observe(model,data,layout,history) + return model,data,truth,inputs,measured,history + + +def _png(rgb): + """Lossless RGB PNG writer; avoids a mandatory image-library dependency.""" + h,w,_ = rgb.shape + def chunk(kind,payload): + return struct.pack(">I",len(payload))+kind+payload+struct.pack(">I",zlib.crc32(kind+payload)&0xffffffff) + raw=b"".join(b"\0"+row.tobytes() for row in rgb) + return (b"\x89PNG\r\n\x1a\n"+chunk(b"IHDR",struct.pack(">IIBBBBB",w,h,8,2,0,0,0)) + +chunk(b"IDAT",zlib.compress(raw,6))+chunk(b"IEND",b"")) + + +def render(model,data,layout): + mj, np = sim._native_imports() + camera = mj.MjvCamera() + camera.lookat[:] = [.31,-.06,.08] + camera.distance = 1.0 + camera.azimuth = layout["azimuth_deg"] + camera.elevation = -55. + renderer = mj.Renderer(model,HEIGHT,WIDTH) + try: + renderer.update_scene(data,camera=camera) + rgb=renderer.render().copy() + renderer.enable_segmentation_rendering() + renderer.update_scene(data,camera=camera) + segmentation=renderer.render().copy() + finally: + renderer.close() + obj=mj.mj_name2id(model,mj.mjtObj.mjOBJ_BODY,ws.SHOE) + target_geoms=np.flatnonzero(model.geom_bodyid==obj) if obj>=0 else np.asarray([],dtype=int) + mask=np.isin(segmentation[:,:,0],target_geoms)&(segmentation[:,:,1]==int(mj.mjtObj.mjOBJ_GEOM)) + ys,xs=np.nonzero(mask) + bounds=None if not len(xs) else [float(xs.min()/WIDTH),float(ys.min()/HEIGHT), + float((xs.max()+1)/WIDTH),float((ys.max()+1)/HEIGHT)] + return _png(rgb), {"visible_target_pixels":int(mask.sum()), "target_box_xyxy_normalized":bounds, + "box_scope":"Visible target pixels only, from native geom-ID segmentation; not an amodal box."} + + +def _check_category(category,truth,measured,visibility): + expected = category in ("complete","occluded") + if truth["complete"] != expected: + raise RuntimeError(f"Fixture {category} violates its declared completion label: {truth}") + if category == "unreleased" and (truth["released"] or not truth["supported"] + or truth["position_error_m"]>CRITERIA["position_tolerance_m"]): + raise RuntimeError("Unreleased fixture must retain real robot contact at the supported goal") + if category == "occluded" and visibility["visible_target_pixels"] != 0: + raise RuntimeError("Occluded fixture leaks visible target pixels") + if category not in ("missing_target","wrong_object","occluded") and not visibility["visible_target_pixels"]: + raise RuntimeError("Expected visible target has no native segmentation pixels") + + +def expected_labels(category, truth): + """Separate physical truth from conclusions supported by each input mode.""" + if category not in CATEGORIES: + raise ValueError("Expected a frozen image-goal category") + visible_violation = category in ("missing_target", "wrong_object", + "wrong_position", "wrong_orientation") + presence = ("unknown" if category == "occluded" else + "present" if truth["present"] else "absent") + return { + "expected_verdict": {"rgb": "incomplete" if visible_violation else "unknown", + "rgb_telemetry": "complete" if truth["complete"] else "incomplete"}, + "expected_presence": {"rgb": presence, + "rgb_telemetry": "present" if truth["present"] else "absent"}} + + +def generate(assets,output): + """Write fresh benchmark inputs/truth. Never contacts any model service.""" + destination=Path(output) + if destination.exists(): + raise ValueError("Image-goal output must be a fresh directory") + destination.mkdir(parents=True) + frozen=_json(protocol()) + (destination/"protocol.json").write_bytes(frozen) + (destination/"protocol.sha256").write_text(sha256(frozen).hexdigest()+"\n") + assets_info=robot_profiles.verify_assets("piper",assets) + arm=_initial_arm(assets) + rows=[]; aliases=[]; start=time.perf_counter() + (destination/"images").mkdir() + (destination/"states").mkdir() + for layout in LAYOUTS: + goal_png=None + for category in CATEGORIES: + model,data,truth,inputs,measured,history=_native_state(assets,layout,category,arm) + png,visibility=render(model,data,layout) + _check_category(category,truth,measured,visibility) + identifier=f"{layout['id']}-{category}" + image_path=f"images/{identifier}.png" + (destination/image_path).write_bytes(png) + if category=="complete": + goal_png=png + if category=="moving_alias": + if png!=goal_png: + raise RuntimeError("Matched-pixel velocity intervention changed the rendered image") + aliases.append({"case_id":identifier,"pixel_identical_to":f"{layout['id']}-complete"}) + state={"time_s":float(data.time),"qpos":data.qpos.tolist(),"qvel":data.qvel.tolist(), + "ctrl":data.ctrl.tolist(),"physical_truth":truth,"current_measurement":measured, + "history":history,"solver_warnings":sum(int(w.number) for w in data.warning), + "object_attachment_constraints_added":0,"model_equalities":int(model.neq), + "external_forces_applied":False} + state_path=f"states/{identifier}.json" + (destination/state_path).write_bytes(_json(state)) + rows.append({"id":identifier,"category":category,"layout_id":layout["id"], + "task":TASK,"current_image":image_path,"goal_image":f"images/{layout['id']}-complete.png", + "physical_truth":truth,"measured_inputs":inputs,"visibility":visibility, + "state_file":state_path, **expected_labels(category,truth)}) + manifest={"schema":SCHEMA,"task":TASK,"criteria":deepcopy(CRITERIA),"cases":rows, + "aliases":aliases,"assets":assets_info,"protocol_sha256":sha256(frozen).hexdigest(), + "mujoco_version":sim._native_imports()[0].__version__, + "rendering":"Native MuJoCo RGB and geom segmentation; no image-generation model.", + "generation_wall_seconds":time.perf_counter()-start, + "scope":protocol()["scope"],"telemetry_scope":protocol()["telemetry_scope"]} + (destination/"cases.json").write_bytes(_json(manifest)) + files=[] + for path in sorted(destination.rglob("*")): + if path.is_file(): + content=path.read_bytes() + files.append({"path":path.relative_to(destination).as_posix(),"bytes":len(content),"sha256":sha256(content).hexdigest()}) + (destination/"integrity.json").write_bytes(_json({"schema":SCHEMA+"-integrity","files":files})) + return manifest + + +def main(argv=None): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument("--assets",required=True) + parser.add_argument("--output",required=True) + args=parser.parse_args(argv) + result=generate(args.assets,args.output) + print(json.dumps({"schema":result["schema"],"cases":len(result["cases"]), + "protocol_sha256":result["protocol_sha256"],"output":args.output})) + + +if __name__=="__main__": + main() diff --git a/src/dvidia_training/jev_client.py b/src/dvidia_training/jev_client.py new file mode 100644 index 0000000..125f228 --- /dev/null +++ b/src/dvidia_training/jev_client.py @@ -0,0 +1,375 @@ +"""Small, auditable Jev image-goal evaluation client (no training or control). + +Uses TypeSafe's documented HTTP multipart API without an SDK or automatic +retries. Images and typed decisions are experimental as of 2026-10-18. +Official contracts: https://docs.typesafe.ai/images, +https://docs.typesafe.ai/api, https://docs.typesafe.ai/primitives/noul, +https://docs.typesafe.ai/primitives/choice and https://docs.typesafe.ai/models. + +The receipt never contains authentication headers. The caller must provide +only images and permitted observations: benchmark labels are not inputs. +""" +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timezone +from hashlib import sha256 +import json +import math +import os +import socket +import struct +import subprocess +import time +from typing import Any +from urllib.error import HTTPError, URLError +from urllib.request import HTTPRedirectHandler, Request, build_opener +import zlib + +MODEL = "jev-1.14.0" +ENDPOINT = "https://api.typesafe.ai/v1/systemone" +INPUT_USD_PER_MILLION = 0.042 +PRICE_CHECKED_DATE = "2026-10-18" +PRESENCE_NO = 0.2 +PRESENCE_YES = 0.8 +MAX_IMAGE_BYTES = 16 * 1024 * 1024 +MAX_REQUEST_BYTES = 32 * 1024 * 1024 +MAX_RESPONSE_BYTES = 1024 * 1024 + +# Keep prompts unchanged across evaluated cases and modalities. Different task +# contracts should be frozen as separate benchmark protocols before API calls. +QUESTIONS = { + "presence": { + "type": "noul", + "instructions": ( + "Is the target object specified by `task` and shown in `goal_image` " + "present in the current scene? Use `current_image` and any supplied " + "`measured_state`. Do not confuse a different object with the target. " + "Occlusion or missing measurements can leave presence uncertain." + ), + "criteria": { + "true": "The target object is present in the current scene.", + "false": "The target object is absent from the current scene.", + }, + }, + "completion": { + "type": "noul", + "instructions": ( + "Has the physical goal in `task` actually been achieved now? " + "Compare `current_image` with `goal_image` and use any supplied " + "`measured_state`. Matching appearance alone does not prove hidden " + "support, release, stability, identity or dwell requirements. " + "Return the probability of physical completion; express uncertainty " + "when the provided evidence cannot establish those requirements." + ), + "criteria": { + "true": "All physical requirements explicitly defined in `task` are satisfied.", + "false": "At least one physical requirement explicitly defined in `task` is not satisfied.", + }, + }, + "verdict": { + "type": "choice", + "instructions": ( + "What conclusion about completion of `task` is justified by " + "`current_image`, `goal_image` and any supplied `measured_state`? " + "Assess only the supplied evidence. Do not invent unseen telemetry " + "or assume that a single image establishes stability over time." + ), + "criteria": { + "complete": "Available evidence establishes every physical requirement in `task`.", + "incomplete": "Available evidence establishes that at least one physical requirement in `task` is not satisfied.", + "unknown": "No observed failure is established, but the available evidence cannot establish every physical requirement in `task`.", + }, + }, +} + + +def _json_bytes(value: Any) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":"), + ensure_ascii=False, allow_nan=False).encode("utf-8") + + +@dataclass(frozen=True) +class PreparedRequest: + request: dict[str, Any] + body: bytes + content_type: str + images: dict[str, dict[str, Any]] + request_sha256: str + + +def _png_metadata(data: bytes) -> dict[str, Any]: + """Check still-PNG framing, CRCs and service size limits before uploading.""" + if not isinstance(data, bytes) or len(data) > MAX_IMAGE_BYTES: + raise ValueError("Image must be PNG bytes no larger than 16 MiB") + if not data.startswith(b"\x89PNG\r\n\x1a\n"): + raise ValueError("Only PNG inputs are accepted by this benchmark client") + position, width, height, saw_idat, saw_end = 8, 0, 0, False, False + while position + 12 <= len(data): + length = struct.unpack_from(">I", data, position)[0] + end = position + 12 + length + if end > len(data): + raise ValueError("Truncated PNG chunk") + kind = data[position + 4:position + 8] + chunk = data[position + 8:position + 8 + length] + actual_crc = struct.unpack_from(">I", data, position + 8 + length)[0] + if zlib.crc32(kind + chunk) & 0xffffffff != actual_crc: + raise ValueError("Invalid PNG checksum") + if position == 8: + if kind != b"IHDR" or length != 13: + raise ValueError("PNG must start with a valid IHDR chunk") + width, height = struct.unpack_from(">II", chunk) + elif kind == b"IHDR": + raise ValueError("Duplicate PNG image header") + if kind in (b"acTL", b"fcTL", b"fdAT"): + raise ValueError("Animated PNG is not supported by Jev images") + if kind == b"IDAT": + saw_idat = True + if kind == b"IEND": + if length or end != len(data): + raise ValueError("Invalid PNG end marker") + saw_end = True + break + position = end + if not saw_idat or not saw_end: + raise ValueError("Incomplete PNG image") + if not (280 <= width <= 1568 and 280 <= height <= 1568): + raise ValueError("Jev image width and height must each be 280 through 1568 pixels") + return {"sha256": sha256(data).hexdigest(), "bytes": len(data), + "width": width, "height": height, "format": "png"} + + +def build_goal_request(current_png: bytes, goal_png: bytes, *, task: str, + measured_state: dict[str, Any] | None = None) -> PreparedRequest: + """Build a deterministic, credential-free request for freezing and review.""" + if not isinstance(task, str) or not task.strip(): + raise ValueError("A nonempty physical task contract is required") + if measured_state is not None and not isinstance(measured_state, dict): + raise ValueError("measured_state must be an observation dictionary or None") + images = {"current": _png_metadata(current_png), "goal": _png_metadata(goal_png)} + refs = {name: f"dvidia-image-{name}-{value['sha256']}" for name, value in images.items()} + state: dict[str, Any] = {"task": task, "current_image": refs["current"], + "goal_image": refs["goal"]} + if measured_state is not None: + state["measured_state"] = measured_state + payload = {"model": MODEL, "state": state, + "assets": {refs[name]: {"type": "image", "part": name, + "format": "png"} for name in images}, + "questions": QUESTIONS} + # JSON round-trip freezes caller-owned structures and rejects NaN/Infinity. + request_bytes = _json_bytes(payload) + payload = json.loads(request_bytes) + boundary = "dvidia-" + sha256(request_bytes).hexdigest() + pieces = [f"--{boundary}\r\nContent-Disposition: form-data; name=\"request\"\r\n" + "Content-Type: application/json\r\n\r\n".encode(), request_bytes, b"\r\n"] + for name, data in (("current", current_png), ("goal", goal_png)): + if boundary.encode() in data: + raise ValueError("Multipart boundary collides with an image") + pieces.extend([f"--{boundary}\r\nContent-Disposition: form-data; name=\"{name}\"; " + f"filename=\"{name}.png\"\r\nContent-Type: image/png\r\n\r\n".encode(), + data, b"\r\n"]) + pieces.append(f"--{boundary}--\r\n".encode()) + body = b"".join(pieces) + if len(body) > MAX_REQUEST_BYTES: + raise ValueError("Multipart request exceeds 32 MiB") + return PreparedRequest(payload, body, f"multipart/form-data; boundary={boundary}", + images, sha256(body).hexdigest()) + + +def _probability(value: Any, label: str) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise ValueError(f"{label} must be a number") + value = float(value) + if not math.isfinite(value) or not 0 <= value <= 1: + raise ValueError(f"{label} must be finite and between zero and one") + return value + + +def parse_goal_response(response: dict[str, Any]) -> dict[str, Any]: + """Reject malformed responses instead of silently coercing them to negatives.""" + if not isinstance(response, dict) or response.get("model") != MODEL: + raise ValueError("Response does not identify the pinned Jev model") + answers = response.get("answers") + if not isinstance(answers, dict): + raise ValueError("Missing answer map") + probabilities = {} + for name in ("presence", "completion"): + answer = answers.get(name) + if not isinstance(answer, dict) or answer.get("type") != "noul": + raise ValueError(f"Missing {name} Noul answer") + probabilities[name] = _probability(answer.get("noul"), name) + verdict = answers.get("verdict") + if not isinstance(verdict, dict) or verdict.get("type") != "choice": + raise ValueError("Missing verdict Choice answer") + choices = verdict.get("probabilities") + options = {"complete", "incomplete", "unknown"} + if not isinstance(choices, dict) or set(choices) != options: + raise ValueError("Verdict probabilities must cover complete, incomplete and unknown") + choices = {name: _probability(value, name) for name, value in choices.items()} + if not math.isclose(sum(choices.values()), 1.0, abs_tol=0.005): + raise ValueError("Verdict probabilities do not sum to one") + selected = verdict.get("choice") + if selected not in options or choices[selected] < max(choices.values()) - 0.005: + raise ValueError("Verdict selection disagrees with probabilities") + confidence = _probability(verdict.get("confidence"), "verdict confidence") + usage_fields = _usage_fields(response) + p_present = probabilities["presence"] + return {"p_present": p_present, "p_complete": probabilities["completion"], + "presence": ("present" if p_present >= PRESENCE_YES else + "absent" if p_present <= PRESENCE_NO else "unknown"), + "verdict": selected, "verdict_probabilities": choices, + "verdict_confidence": confidence, **usage_fields} + + +def _usage_fields(response: dict[str, Any]) -> dict[str, Any]: + """Usage can remain valid even when a billed response cannot be scored.""" + usage = response.get("usage") if isinstance(response, dict) else None + if not isinstance(usage, dict): + raise ValueError("Missing token usage") + for name in ("input_tokens", "output_tokens"): + if type(usage.get(name)) is not int or usage[name] < 0: + raise ValueError(f"Invalid {name} usage") + return {"usage": {name: usage[name] for name in ("input_tokens", "output_tokens")}, + "estimated_input_cost_usd": usage["input_tokens"] * INPUT_USD_PER_MILLION / 1e6} + + +def _retain_usage(receipt: dict[str, Any], response: Any) -> None: + try: + receipt.update(_usage_fields(response)) + except ValueError: + pass + + +class _NoRedirect(HTTPRedirectHandler): + def redirect_request(self, req, fp, code, msg, headers, newurl): + raise HTTPError(req.full_url, code, "Redirect refused", headers, fp) + + +def _redact(value: Any, secret: str) -> Any: + if isinstance(value, str): + return value.replace(secret, "[REDACTED]") if secret else value + if isinstance(value, dict): + return {_redact(k, secret): _redact(v, secret) for k, v in value.items()} + if isinstance(value, list): + return [_redact(v, secret) for v in value] + return value + + +def load_api_key(*, keychain_service: str | None = None, + keychain_account: str | None = None) -> str | None: + """Read env first, then only an explicitly named macOS Keychain item. + + Never enumerates Keychain entries, writes credentials, or logs output. + Interactive Keychain access denial is reported without revealing stderr. + """ + configured = os.environ.get("TYPESAFE_API_KEY") + if configured: + return configured + if keychain_service is None and keychain_account is None: + return None + if not keychain_service or not keychain_account: + raise ValueError("Both Keychain service and account are required") + try: + result = subprocess.run( + ["/usr/bin/security", "find-generic-password", "-s", keychain_service, + "-a", keychain_account, "-w"], capture_output=True, text=True, + timeout=15, check=False, + ) + except (OSError, subprocess.TimeoutExpired): + return None + if result.returncode: + return None + return result.stdout.strip() or None + + +class JevClient: + """One call per evaluate, never retry or follow redirects. + + timeout_s is the socket-operation timeout (including connect/read), not a + real-time deadline guarantee. The receipt measures actual wall time. DNS, + TLS, upload and network/server delay are all inside the HTTP timing. + """ + def __init__(self, api_key: str | None = None, *, timeout_s: float = 20.0): + if not math.isfinite(timeout_s) or not 0 < timeout_s <= 60: + raise ValueError("timeout_s must be positive and at most 60 seconds") + self._api_key = api_key if api_key is not None else os.environ.get("TYPESAFE_API_KEY", "") + if not isinstance(self._api_key, str) or any(c in self._api_key for c in "\r\n"): + raise ValueError("Invalid API credential") + self.timeout_s = float(timeout_s) + self._opener = build_opener(_NoRedirect()) + + def evaluate(self, current_png: bytes, goal_png: bytes, *, task: str, + measured_state: dict[str, Any] | None = None) -> dict[str, Any]: + start = time.perf_counter() + receipt: dict[str, Any] = { + "status": "error", "model": MODEL, + "started_at": datetime.now(timezone.utc).isoformat(), + "request": None, "request_sha256": None, "images": None, + "response": None, "response_sha256": None, "http_status": None, + "latency_ms": {"prepare": None, "http": None, "total": None}, + "p_present": None, "p_complete": None, "presence": "unknown", + "verdict": "unknown", "verdict_probabilities": None, + "verdict_confidence": None, "usage": None, + "estimated_input_cost_usd": None, "error": None, + "socket_timeout_s": self.timeout_s, "attempts": 0, + } + try: + prepared = build_goal_request(current_png, goal_png, task=task, + measured_state=measured_state) + receipt.update(request=prepared.request, request_sha256=prepared.request_sha256, + images=prepared.images, request_bytes=len(prepared.body)) + receipt["latency_ms"]["prepare"] = (time.perf_counter() - start) * 1000 + if not self._api_key: + receipt["status"] = "missing_credentials" + receipt["error"] = {"kind": "missing_credentials", "message": "TYPESAFE_API_KEY is not configured"} + return receipt + request = Request(ENDPOINT, data=prepared.body, method="POST", headers={ + "Authorization": f"Bearer {self._api_key}", + "TypeSafe-Danger-Zone-Images": "true", + "Content-Type": prepared.content_type, + "Accept": "application/json", "Accept-Encoding": "identity", + "User-Agent": "DVIDIA-Image-Goal-Benchmark/0.1", + }) + receipt["attempts"] = 1 + http_start = time.perf_counter() + try: + with self._opener.open(request, timeout=self.timeout_s) as response: + receipt["http_status"] = response.status + body = response.read(MAX_RESPONSE_BYTES + 1) + finally: + receipt["latency_ms"]["http"] = (time.perf_counter() - http_start) * 1000 + if len(body) > MAX_RESPONSE_BYTES: + raise ValueError("Provider response exceeds the 1 MiB benchmark limit") + receipt["response_sha256"] = sha256(body).hexdigest() + decoded = json.loads(body) + receipt["response"] = _redact(decoded, self._api_key) + _retain_usage(receipt, decoded) + receipt.update(parse_goal_response(decoded)) + receipt["status"] = "ok" + except HTTPError as error: + receipt["http_status"] = error.code + receipt["error"] = {"kind": "http_error", "message": f"TypeSafe returned HTTP {error.code}; no automatic retry"} + # Preserve bounded JSON error evidence, but never server text echoing auth. + try: + body = error.read(MAX_RESPONSE_BYTES + 1) + if len(body) <= MAX_RESPONSE_BYTES: + receipt["response_sha256"] = sha256(body).hexdigest() + decoded = json.loads(body) + receipt["response"] = _redact(decoded, self._api_key) + _retain_usage(receipt, decoded) + except (ValueError, OSError): + pass + receipt["latency_ms"]["http"] = (time.perf_counter() - http_start) * 1000 + except (TimeoutError, socket.timeout): + receipt["error"] = {"kind": "timeout", "message": "TypeSafe socket operation timed out; no automatic retry"} + except URLError as error: + kind = "timeout" if isinstance(error.reason, (TimeoutError, socket.timeout)) else "network_error" + receipt["error"] = {"kind": kind, "message": "TypeSafe request could not complete; no automatic retry"} + except (ValueError, TypeError, UnicodeError) as error: + receipt["error"] = {"kind": "invalid_response" if receipt["attempts"] else "invalid_request", + "message": _redact(str(error), self._api_key)} + except OSError: + receipt["error"] = {"kind": "network_error", "message": "TypeSafe transport failed; no automatic retry"} + finally: + receipt["latency_ms"]["total"] = (time.perf_counter() - start) * 1000 + return receipt diff --git a/tests/test_image_goal_benchmark.py b/tests/test_image_goal_benchmark.py new file mode 100644 index 0000000..e9826fc --- /dev/null +++ b/tests/test_image_goal_benchmark.py @@ -0,0 +1,92 @@ +"""Failure denominators, abstention and immutable paired evaluation contracts.""" +import copy +from pathlib import Path +import tempfile +import unittest + +from dvidia_training import image_goal_benchmark as bench + + +def case(identity, *, present=True, complete=False, expected="incomplete"): + return {"id": identity, "category": identity, "physical_truth": {"present": present, "complete": complete}, + "measured_inputs": {}, "expected_verdict": {condition: expected for condition in bench.CONDITIONS}, + "expected_presence": {condition: "present" if present else "absent" for condition in bench.CONDITIONS}} + + +def row(identity, *, verdict="incomplete", presence="present", status="ok", latency=10.): + return {"case_id": identity, "condition": "rgb", "verdict": verdict, "presence": presence, + "status": status, "online_total_ms": latency, "latency_ms": {"http": latency}, + "p_present": .9, "p_complete": .9 if verdict == "complete" else .1, "estimated_input_cost_usd": .0001} + + +def paired(rows): + return [dict(row, condition=condition) for row in rows for condition in bench.CONDITIONS] + + +class ScoreTests(unittest.TestCase): + def test_errors_and_abstention_cannot_improve_reported_scheduled_accuracy(self): + cases = [case("missing", present=False), case("moving", expected="unknown"), + case("good", complete=True, expected="complete"), case("bad")] + rows = [row("missing", verdict="complete"), row("moving", verdict="unknown"), + row("good", verdict="complete"), row("bad", status="error", verdict="unknown", latency=20000)] + result = bench.score(paired(rows), cases)["overall"] + self.assertEqual(result["physical_false_completion"], {"numerator": 2, "denominator": 6, "rate": 1/3}) + self.assertEqual(result["missing_object_false_presence"]["rate"], 1.) + self.assertEqual(result["observability_aware_accuracy"]["rate"], .5) + self.assertEqual(result["completion_coverage"]["rate"], .5) + self.assertEqual(result["selective_physical_accuracy"]["rate"], .5) + self.assertEqual(result["ambiguous_abstention"]["rate"], 1.) + self.assertEqual(result["online_ms_all_attempts"]["max"], 20000) + self.assertEqual(result["online_ms_successful_requests"]["max"], 10) + self.assertEqual(result["statuses"], {"ok": 6, "error": 2}) + + def test_always_abstain_has_no_selective_accuracy_and_no_success_recall(self): + cases = [case("good", complete=True, expected="complete"), case("bad")] + result = bench.score(paired([row(c["id"], verdict="unknown") for c in cases]), cases)["overall"] + self.assertIsNone(result["selective_physical_accuracy"]["rate"]) + self.assertEqual(result["completion_coverage"]["rate"], 0.) + self.assertEqual(result["physical_true_completion"]["rate"], 0.) + self.assertEqual(result["physical_false_completion"]["rate"], 0.) + + def test_no_data_is_unavailable_not_perfect(self): + result = bench.score([], [])["overall"] + self.assertIsNone(result["physical_false_completion"]["rate"]) + self.assertIsNone(result["online_ms_all_attempts"]["p50"]) + self.assertIsNone(result["known_usage_estimated_input_cost_usd"]) + + def test_duplicate_or_unknown_trials_rejected(self): + with self.assertRaises(ValueError): + bench.score([row("one"), row("one")], [case("one")]) + with self.assertRaises(ValueError): + bench.score([row("two")], [case("one")]) + + def test_missing_scheduled_result_and_nonfinite_probability_are_rejected(self): + with self.assertRaises(ValueError): + bench.score([row("one")], [case("one")]) + rows = paired([row("one")]); rows[0]["p_complete"] = float("nan") + with self.assertRaises(ValueError): + bench.score(rows, [case("one")]) + + def test_presence_and_completion_truth_are_consistent(self): + bad = case("bad", present=False, complete=True) + with self.assertRaises(ValueError): + bench.validate_cases({"cases": [bad]}) + bad = case("bad") + bad["physical_truth"]["complete"] = 0 + with self.assertRaises(ValueError): + bench.validate_cases({"cases": [bad]}) + + def test_artifact_path_cannot_escape_fixtures(self): + with tempfile.TemporaryDirectory() as tmp: + base = Path(tmp); fixtures = base / "fixtures"; fixtures.mkdir() + (base / "secret.txt").write_text("not a fixture") + for name in ("../secret.txt", str(base / "secret.txt")): + with self.assertRaises(ValueError): + bench.relative_file(fixtures, name) + (fixtures / "link").symlink_to(base / "secret.txt") + with self.assertRaises(ValueError): + bench.relative_file(fixtures, "link") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_image_goal_freeze.py b/tests/test_image_goal_freeze.py new file mode 100644 index 0000000..616425c --- /dev/null +++ b/tests/test_image_goal_freeze.py @@ -0,0 +1,180 @@ +"""Freeze and offline execution tests; no native simulator or network required.""" +from contextlib import redirect_stdout +from hashlib import sha256 +import io +import json +from pathlib import Path +import struct +import tempfile +import unittest +from unittest.mock import patch +import zlib + +from dvidia_training import image_goal_benchmark as bench +from dvidia_training import jev_client as jev + + +def png(width=320, height=280): + def chunk(kind, value): + return (struct.pack('>I', len(value)) + kind + value + + struct.pack('>I', zlib.crc32(kind + value) & 0xffffffff)) + return (b'\x89PNG\r\n\x1a\n' + + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0)) + + chunk(b'IDAT', zlib.compress((b'\0' + b'\0' * width * 3) * height)) + + chunk(b'IEND', b'')) + + +def fixtures(directory): + root = Path(directory) / 'fixtures' + root.mkdir() + (root / 'current.png').write_bytes(png()) + (root / 'goal.png').write_bytes(png(321)) + cases = [] + for name, present in [('present-case', True), ('absent-case', False)]: + cases.append({'id': name, 'category': 'unit-fixture', + 'current_image': 'current.png', 'goal_image': 'goal.png', + 'state_file': f'{name}-state.json', + 'physical_truth': {'present': present, 'complete': False}, + 'measured_inputs': {'observed_contact_force_n': .12 if present else 0.}, + 'expected_verdict': {condition: 'incomplete' for condition in bench.CONDITIONS}, + 'expected_presence': {condition: 'present' if present else 'absent' for condition in bench.CONDITIONS}}) + for case in cases: + (root / case['state_file']).write_bytes(bench.json_bytes( + {'scope': 'unit-test-only', 'native_contact_history_n': [.12, .12]})) + (root / 'cases.json').write_bytes(bench.json_bytes({'cases': cases})) + (root / 'protocol.json').write_bytes(bench.json_bytes({'fixture_type': 'unit-test-only'})) + (root / 'protocol.sha256').write_text(sha256((root / 'protocol.json').read_bytes()).hexdigest() + '\n') + entries = [{'path': path.name, **bench.digest(path)} for path in sorted(root.iterdir())] + (root / 'integrity.json').write_bytes(bench.json_bytes({'files': entries})) + return root, cases + + +def seal(root): + return bench.freeze(root, task='Place the target shoe in the goal zone.', source_commit='a' * 40) + + +class ImageGoalFreezeTests(unittest.TestCase): + def test_freeze_covers_every_paired_trial_and_digest_matches_exact_input(self): + with tempfile.TemporaryDirectory() as directory: + root, cases = fixtures(directory) + plan = seal(root) + self.assertEqual(len(plan['trials']), len(cases) * len(bench.CONDITIONS)) + self.assertEqual({(row['case_id'], row['condition']) for row in plan['trials']}, + {(case['id'], c) for case in cases for c in bench.CONDITIONS}) + self.assertEqual(set(plan['files']), {'current.png', 'goal.png', 'cases.json', 'protocol.json', + 'protocol.sha256', 'integrity.json', 'present-case-state.json', 'absent-case-state.json'}) + self.assertEqual(plan['automatic_retries'], 0) + by_id = {case['id']: case for case in cases} + for row in plan['trials']: + case = by_id[row['case_id']] + prepared = jev.build_goal_request( + (root / case['current_image']).read_bytes(), (root / case['goal_image']).read_bytes(), + task=plan['task'], measured_state=case['measured_inputs'] if row['condition'] == 'rgb_telemetry' else None) + self.assertEqual(prepared.request_sha256, row['request_sha256']) + serialized = json.dumps(prepared.request) + self.assertNotIn(case['id'], serialized) + self.assertNotIn('physical_truth', serialized) + self.assertNotIn('expected_verdict', serialized) + verified, verified_cases = bench.verify_plan(root) + self.assertEqual(verified, plan) + self.assertEqual(verified_cases, cases) + + def test_mutating_any_frozen_evidence_file_is_rejected(self): + for name in ('current.png', 'goal.png', 'cases.json', 'protocol.json', + 'protocol.sha256', 'integrity.json', 'present-case-state.json', 'absent-case-state.json'): + with self.subTest(name=name), tempfile.TemporaryDirectory() as directory: + root, _ = fixtures(directory) + seal(root) + with (root / name).open('ab') as stream: + stream.write(b' ') + with self.assertRaises(ValueError): + bench.verify_plan(root) + + def test_duplicate_missing_trial_or_late_request_mutation_rejected_before_calls(self): + for mutation in ('duplicate', 'missing', 'late_digest', 'task'): + with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as directory: + root, _ = fixtures(directory) + plan = seal(root) + if mutation == 'duplicate': + plan['trials'][-1] = plan['trials'][0] + elif mutation == 'missing': + plan['trials'].pop() + elif mutation == 'late_digest': + plan['trials'][-1]['request_sha256'] = '0' * 64 + else: + plan['task'] = 'A modified contract after freezing.' + (root / 'evaluation-plan.json').write_bytes(bench.json_bytes(plan)) + with patch.object(jev.JevClient, 'evaluate') as provider: + with self.assertRaises(ValueError), redirect_stdout(io.StringIO()): + bench.evaluate(root, Path(directory) / 'results', api_key='test-not-real') + provider.assert_not_called() + + def test_freeze_and_results_refuse_overwrite(self): + with tempfile.TemporaryDirectory() as directory: + root, _ = fixtures(directory) + seal(root) + original = (root / 'evaluation-plan.json').read_bytes() + with self.assertRaises(FileExistsError): + seal(root) + self.assertEqual((root / 'evaluation-plan.json').read_bytes(), original) + output = Path(directory) / 'results' + output.mkdir() + (output / 'keep.txt').write_text('existing result') + with self.assertRaises(FileExistsError): + bench.evaluate(root, output, api_key='') + self.assertEqual((output / 'keep.txt').read_text(), 'existing result') + + def test_interruption_preserves_full_scheduled_inventory_and_unscored_checkpoint(self): + with tempfile.TemporaryDirectory() as directory: + root, _ = fixtures(directory) + plan = seal(root) + response = {'status': 'ok', 'model': jev.MODEL, 'attempts': 1, + 'presence': 'present', 'verdict': 'unknown', + 'p_present': .8, 'p_complete': .5, + 'latency_ms': {'prepare': .1, 'http': 5., 'total': 5.1}, + 'estimated_input_cost_usd': .0001} + output = Path(directory) / 'results' + with patch.object(jev.JevClient, 'evaluate', side_effect=[response, KeyboardInterrupt()]): + with self.assertRaises(KeyboardInterrupt), redirect_stdout(io.StringIO()): + bench.evaluate(root, output, api_key='not-a-real-key') + report = json.loads((output / 'results.json').read_text()) + self.assertEqual(len(report['results']), len(plan['trials'])) + self.assertEqual({(row['case_id'], row['condition']) for row in report['results']}, + {(row['case_id'], row['condition']) for row in plan['trials']}) + self.assertIsNone(report['metrics']) + self.assertEqual(sum(row['status'] == 'ok' for row in report['results']), 1) + self.assertNotEqual(report['status'], 'completed') + self.assertEqual(report['active_trial']['case_id'], plan['trials'][1]['case_id']) + self.assertEqual(report['active_trial']['condition'], plan['trials'][1]['condition']) + self.assertIn('charge unknown', report['active_trial']['outcome']) + self.assertNotIn('not-a-real-key', (output / 'results.json').read_text()) + + def test_absent_credentials_keep_all_trials_without_model_scores_or_network(self): + with tempfile.TemporaryDirectory() as directory: + root, cases = fixtures(directory) + plan = seal(root) + with patch.dict('os.environ', {}, clear=True), patch.object(jev, 'build_opener') as opener: + opener.return_value.open.side_effect = AssertionError('Network must not be called') + with redirect_stdout(io.StringIO()): + report = bench.evaluate(root, Path(directory) / 'results') + opener.return_value.open.assert_not_called() + self.assertEqual(report['status'], 'not_run') + self.assertIsNone(report['active_trial']) + self.assertIsNone(report['metrics']) + self.assertEqual(len(report['results']), len(plan['trials'])) + for row in report['results']: + self.assertEqual(row['status'], 'not_run') + self.assertIsNone(row['p_complete']) + self.assertIsNone(row['p_present']) + self.assertIsNone(row['online_total_ms']) + self.assertIsNone(row['estimated_input_cost_usd']) + self.assertEqual(row['attempts'], 0) + output = Path(directory) / 'results' + self.assertEqual(json.loads((output / 'results.json').read_text()), report) + ledger = [json.loads(line) for line in (output / 'attempts.jsonl').read_text().splitlines()] + self.assertEqual(ledger, report['results']) + self.assertEqual((output / 'evaluation-plan.json').read_bytes(), (root / 'evaluation-plan.json').read_bytes()) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/test_image_goal_scenes.py b/tests/test_image_goal_scenes.py new file mode 100644 index 0000000..875434b --- /dev/null +++ b/tests/test_image_goal_scenes.py @@ -0,0 +1,127 @@ +"""Physical and observability controls for the frozen image-goal study.""" +from copy import deepcopy +import os +import unittest + +from dvidia_training import image_goal_scenes as scenes, reference_sim as sim + + +class ProtocolTests(unittest.TestCase): + def test_protocol_is_balanced_explicit_and_independent(self): + protocol = scenes.protocol() + self.assertEqual(len(protocol["case_ids"]), 32) + self.assertEqual(len(set(protocol["case_ids"])), 32) + self.assertEqual(len(protocol["layouts"]), 4) + self.assertEqual(len(protocol["categories"]), 8) + self.assertEqual(protocol["synthetic_shoe"]["width_scale"], .5) + self.assertIn("not statistical independent", protocol["scope"]) + protocol["criteria"]["position_tolerance_m"] = 9 + self.assertEqual(scenes.protocol()["criteria"]["position_tolerance_m"], .025) + + def test_unknown_fixture_is_rejected_before_loading_assets(self): + with self.assertRaises(ValueError): + scenes.build_model("/missing", scenes.LAYOUTS[0], "invented") + with self.assertRaises(ValueError): + scenes.build_model("/missing", dict(scenes.LAYOUTS[0], xy_m=[9.,9.])) + + def test_visible_fingers_do_not_prove_a_contact_force_threshold(self): + labels = scenes.expected_labels("unreleased", {"present": True, "complete": False}) + self.assertEqual(labels["expected_verdict"], {"rgb": "unknown", "rgb_telemetry": "incomplete"}) + self.assertEqual(labels["expected_presence"], {"rgb": "present", "rgb_telemetry": "present"}) + for category in ("missing_target", "wrong_object", "wrong_position", "wrong_orientation"): + labels = scenes.expected_labels(category, {"present": False, "complete": False}) + self.assertEqual(labels["expected_verdict"]["rgb"], "incomplete") + with self.assertRaises(ValueError): + scenes.expected_labels("invented", {"present": True, "complete": True}) + + +@unittest.skipUnless(os.environ.get("DVIDIA_ROBOT_PROFILE_ASSETS"), "Set pinned robot assets") +class NativeFixtureTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.assets = os.environ["DVIDIA_ROBOT_PROFILE_ASSETS"] + cls.mj, cls.np = sim._native_imports() + arm = scenes._initial_arm(cls.assets) + cls.states = {(layout["id"], category): scenes._native_state( + cls.assets, layout, category, arm) + for layout in scenes.LAYOUTS for category in scenes.CATEGORIES} + + def test_all_categories_have_measured_physical_truth(self): + for (layout, category), (model,data,truth,inputs,measured,history) in self.states.items(): + with self.subTest(layout=layout, category=category): + self.assertEqual(truth["complete"], category in ("complete","occluded")) + self.assertEqual(truth["present"], category not in ("missing_target","wrong_object")) + self.assertEqual(sum(w.number for w in data.warning), 0) + self.assertEqual(model.neq, 1) # Upstream gripper coupling only. + self.assertFalse(self.np.any(data.xfrc_applied)) + self.assertFalse(self.np.any(data.qfrc_applied)) + self.assertNotIn("complete", inputs) + self.assertNotIn("instantaneous_conditions_met", inputs) + + def test_unreleased_is_supported_in_goal_with_real_finger_forces(self): + for layout in scenes.LAYOUTS: + model,data,truth,inputs,measured,_ = self.states[(layout["id"],"unreleased")] + with self.subTest(layout=layout["id"]): + self.assertTrue(truth["supported"]) + self.assertFalse(truth["released"]) + self.assertLess(truth["position_error_m"], .025) + self.assertLess(truth["yaw_error_rad"], scenes.CRITERIA["yaw_tolerance_rad"]) + self.assertGreater(measured["robot_contact_force_n"], .01) + self.assertGreater(measured["table_support_force_n"], .01) + self.assertEqual(model.actuator_gainprm[6,0], 40.) + obj=self.mj.mj_name2id(model,self.mj.mjtObj.mjOBJ_BODY,scenes.ws.SHOE) + self.assertAlmostEqual(model.body_mass[obj], .030) + self.assertEqual(model.body_gravcomp[obj], 0.) + self.assertEqual(model.jnt_type[model.body_jntadr[obj]], self.mj.mjtJoint.mjJNT_FREE) + + def test_velocity_alias_keeps_pose_but_changes_truth_and_observation_age(self): + for layout in scenes.LAYOUTS: + complete=self.states[(layout["id"],"complete")] + moving=self.states[(layout["id"],"moving_alias")] + with self.subTest(layout=layout["id"]): + self.np.testing.assert_array_equal(complete[1].qpos,moving[1].qpos) + self.assertTrue(complete[2]["complete"]) + self.assertFalse(moving[2]["complete"]) + self.assertAlmostEqual(moving[2]["linear_speed_m_s"], .12) + self.assertEqual(moving[3]["native_observation_count"], 0) + self.assertEqual(moving[3]["observation_window_s"], 0.) + + def test_one_native_tick_violation_invalidates_dwell_and_is_in_extrema(self): + layout=scenes.LAYOUTS[0] + model,data,truth,inputs,measured,history=self.states[(layout["id"],"complete")] + baseline=self.mj.MjData(model) + baseline.qpos[:]=data.qpos + baseline.qvel[:]=data.qvel + baseline.ctrl[:]=data.ctrl + self.mj.mj_forward(model,baseline) + obj=self.mj.mj_name2id(model,self.mj.mjtObj.mjOBJ_BODY,scenes.ws.SHOE) + dof=int(model.jnt_dofadr[int(model.body_jntadr[obj])]) + baseline.qvel[dof]=.12 + self.mj.mj_forward(model,baseline) + actual_violation=scenes._measure(model,baseline,layout) + changed=deepcopy(history) + changed[-25]=actual_violation + changed_truth,changed_inputs,_=scenes.observe(model,data,layout,changed) + self.assertFalse(changed_truth["complete"]) + self.assertAlmostEqual(changed_truth["physical_conditions_dwell_s"], .048) + self.assertGreater(changed_inputs["native_window_extrema"]["linear_speed_m_s"]["max"], .1) + self.assertEqual(changed_inputs["native_observation_count"], 100) + + @unittest.skipUnless(os.environ.get("DVIDIA_RENDER_MUJOCO"), "Enable native graphics explicitly") + def test_native_images_preserve_alias_and_fully_occlude_target(self): + for layout in scenes.LAYOUTS: + images={} + for category in ("complete","moving_alias","occluded","missing_target","wrong_object"): + model,data,*_=self.states[(layout["id"],category)] + images[category],visibility=scenes.render(model,data,layout) + with self.subTest(layout=layout["id"],category=category): + if category in ("occluded","missing_target","wrong_object"): + self.assertEqual(visibility["visible_target_pixels"], 0) + self.assertIsNone(visibility["target_box_xyxy_normalized"]) + else: + self.assertGreater(visibility["visible_target_pixels"], 500) + self.assertEqual(images["complete"],images["moving_alias"]) + + +if __name__=="__main__": + unittest.main() diff --git a/tests/test_jev_client.py b/tests/test_jev_client.py new file mode 100644 index 0000000..e1827f7 --- /dev/null +++ b/tests/test_jev_client.py @@ -0,0 +1,209 @@ +"""Provider contract tests use fake transports only; never require an API key.""" +from copy import deepcopy +from email.parser import BytesParser +from email.policy import default +from hashlib import sha256 +import io +import json +import socket +import struct +import subprocess +import time +import unittest +from unittest.mock import Mock, patch +from urllib.error import HTTPError, URLError +import zlib + +from dvidia_training import jev_client as jev + + +def png(width=320, height=280): + def chunk(kind, body): + return (struct.pack('>I', len(body)) + kind + body + + struct.pack('>I', zlib.crc32(kind + body) & 0xffffffff)) + return (b'\x89PNG\r\n\x1a\n' + + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0)) + + chunk(b'IDAT', zlib.compress((b'\0' + b'\0' * width * 3) * height)) + + chunk(b'IEND', b'')) + + +def valid_response(present=.99, complete=.03, verdict='incomplete'): + choices = {name: .01 for name in ('complete', 'incomplete', 'unknown')} + choices[verdict] = .98 + return {'model': jev.MODEL, + 'answers': {'presence': {'type': 'noul', 'noul': present}, + 'completion': {'type': 'noul', 'noul': complete}, + 'verdict': {'type': 'choice', 'choice': verdict, + 'probabilities': choices, 'confidence': .97}}, + 'usage': {'input_tokens': 1200, 'output_tokens': 20}} + + +class FakeResponse: + status = 200 + def __init__(self, value, delay=0): + self.body = json.dumps(value).encode() + self.delay = delay + def __enter__(self): + return self + def __exit__(self, *_): + return False + def read(self, limit): + time.sleep(self.delay) + return self.body[:limit] + + +class JevContractTests(unittest.TestCase): + def test_multipart_images_are_uploaded_once_and_referenced_in_state(self): + current, goal = png(), png(321) + state = {'support_force_n': .25} + prepared = jev.build_goal_request(current, goal, task='Place the target shoe.', measured_state=state) + message = BytesParser(policy=default).parsebytes( + b'Content-Type: ' + prepared.content_type.encode() + b'\r\nMIME-Version: 1.0\r\n\r\n' + prepared.body) + parts = list(message.iter_parts()) + self.assertEqual([p.get_param('name', header='Content-Disposition') for p in parts], + ['request', 'current', 'goal']) + payload = json.loads(parts[0].get_payload(decode=True)) + self.assertEqual(payload['model'], 'jev-1.14.0') + self.assertEqual(set(payload['questions']), {'presence', 'completion', 'verdict'}) + self.assertEqual(parts[1].get_payload(decode=True), current) + self.assertEqual(parts[2].get_payload(decode=True), goal) + self.assertEqual(set(payload['assets']), {payload['state']['current_image'], payload['state']['goal_image']}) + self.assertEqual(prepared.images['current']['sha256'], sha256(current).hexdigest()) + state['support_force_n'] = 999 + self.assertEqual(prepared.request['state']['measured_state']['support_force_n'], .25) + self.assertEqual(prepared.request_sha256, sha256(prepared.body).hexdigest()) + repeated = jev.build_goal_request(current, goal, task='Place the target shoe.', measured_state={'support_force_n': .25}) + self.assertEqual(prepared.body, repeated.body) + + def test_rgb_omits_telemetry_and_rejects_invalid_images_and_nonfinite_state(self): + prepared = jev.build_goal_request(png(), png(), task='Place the target shoe.') + self.assertNotIn('measured_state', prepared.request['state']) + for image in (b'not a PNG', png(279), png(1569), png()[:-12], png()[:-13] + b'x' * 13): + with self.subTest(image_bytes=len(image)), self.assertRaises(ValueError): + jev.build_goal_request(image, png(), task='Place shoe.') + with self.assertRaises(ValueError): + jev.build_goal_request(png(), png(), task='Place shoe.', measured_state={'force': float('nan')}) + + def test_noul_uncertainty_is_preserved_independently_of_tristate_choice(self): + result = jev.parse_goal_response(valid_response(.5, .8, 'unknown')) + self.assertEqual(result['presence'], 'unknown') + self.assertEqual(result['verdict'], 'unknown') + self.assertEqual(result['p_complete'], .8) + self.assertAlmostEqual(result['estimated_input_cost_usd'], .0000504) + self.assertEqual(jev.parse_goal_response(valid_response(.2))['presence'], 'absent') + self.assertEqual(jev.parse_goal_response(valid_response(.8))['presence'], 'present') + + def test_invalid_model_probabilities_choice_and_usage_are_not_scored(self): + variants = [] + row = valid_response(); row['model'] = 'jev-latest'; variants.append(row) + row = valid_response(); row['answers']['presence']['noul'] = True; variants.append(row) + row = valid_response(); row['answers']['completion']['noul'] = float('nan'); variants.append(row) + row = valid_response(); row['answers']['verdict']['probabilities'].pop('unknown'); variants.append(row) + row = valid_response(); row['answers']['verdict']['probabilities']['complete'] = .9; variants.append(row) + row = valid_response(); row['answers']['verdict']['choice'] = 'complete'; variants.append(row) + row = valid_response(); row['usage']['input_tokens'] = '1200'; variants.append(row) + for response in variants: + with self.subTest(response=response), self.assertRaises(ValueError): + jev.parse_goal_response(response) + + def test_success_receipt_times_upload_response_and_keeps_credentials_private(self): + client = jev.JevClient('secret-for-test', timeout_s=3) + client._opener = Mock() + client._opener.open.return_value = FakeResponse(valid_response(), delay=.003) + receipt = client.evaluate(png(), png(), task='Place shoe.') + self.assertEqual(receipt['status'], 'ok') + self.assertEqual(receipt['attempts'], 1) + self.assertGreaterEqual(receipt['latency_ms']['http'], 2) + self.assertGreaterEqual(receipt['latency_ms']['total'], receipt['latency_ms']['http']) + request = client._opener.open.call_args.args[0] + self.assertEqual(request.full_url, jev.ENDPOINT) + self.assertEqual(request.get_header('Typesafe-danger-zone-images'), 'true') + self.assertEqual(request.get_header('Authorization'), 'Bearer secret-for-test') + self.assertEqual(client._opener.open.call_args.kwargs['timeout'], 3) + self.assertNotIn('secret-for-test', json.dumps(receipt)) + self.assertNotIn('Authorization', json.dumps(receipt)) + + def test_missing_auth_never_calls_provider_and_is_not_an_absent_prediction(self): + client = jev.JevClient('') + client._opener = Mock() + result = client.evaluate(png(), png(), task='Place shoe.') + client._opener.open.assert_not_called() + self.assertEqual(result['status'], 'missing_credentials') + self.assertEqual(result['attempts'], 0) + self.assertEqual(result['presence'], 'unknown') + self.assertIsNone(result['p_complete']) + self.assertIsNone(result['estimated_input_cost_usd']) + self.assertIsNone(result['latency_ms']['http']) + + def test_timeout_and_rate_limit_are_single_attempts_and_keep_error_evidence(self): + for error in (socket.timeout('timed out'), URLError(socket.timeout('timeout')), + HTTPError(jev.ENDPOINT, 429, 'limit', {}, io.BytesIO(b'{"error":"secret-for-test"}'))): + client = jev.JevClient('secret-for-test') + client._opener = Mock() + client._opener.open.side_effect = error + result = client.evaluate(png(), png(), task='Place shoe.') + self.assertEqual(client._opener.open.call_count, 1) + self.assertEqual(result['attempts'], 1) + self.assertEqual(result['status'], 'error') + self.assertEqual(result['verdict'], 'unknown') + self.assertIsNone(result['p_complete']) + self.assertNotIn('secret-for-test', json.dumps(result)) + if isinstance(error, HTTPError): + self.assertEqual(result['http_status'], 429) + self.assertEqual(result['response']['error'], '[REDACTED]') + else: + self.assertEqual(result['error']['kind'], 'timeout') + + def test_invalid_provider_response_preserved_but_not_scored(self): + client = jev.JevClient('secret-for-test') + client._opener = Mock() + response = valid_response() + response['answers']['presence']['noul'] = 'secret-for-test' + client._opener.open.return_value = FakeResponse(response) + result = client.evaluate(png(), png(), task='Place shoe.') + self.assertEqual(result['status'], 'error') + self.assertEqual(result['error']['kind'], 'invalid_response') + self.assertIsNone(result['p_present']) + self.assertNotIn('secret-for-test', json.dumps(result)) + self.assertEqual(result['response']['answers']['presence']['noul'], '[REDACTED]') + self.assertEqual(result['usage']['input_tokens'], 1200) + self.assertAlmostEqual(result['estimated_input_cost_usd'], .0000504) + + def test_invalid_usage_is_unknown_cost_and_http_error_usage_is_not_lost(self): + client = jev.JevClient('secret-for-test') + client._opener = Mock() + response = valid_response() + response['usage']['input_tokens'] = -1 + client._opener.open.return_value = FakeResponse(response) + result = client.evaluate(png(), png(), task='Place shoe.') + self.assertEqual(result['status'], 'error') + self.assertIsNone(result['usage']) + self.assertIsNone(result['estimated_input_cost_usd']) + body = json.dumps({'error': 'model failed', 'usage': {'input_tokens': 400, 'output_tokens': 0}}).encode() + client._opener.open.side_effect = HTTPError(jev.ENDPOINT, 500, 'failed', {}, io.BytesIO(body)) + result = client.evaluate(png(), png(), task='Place shoe.') + self.assertEqual(result['status'], 'error') + self.assertEqual(result['usage']['input_tokens'], 400) + self.assertAlmostEqual(result['estimated_input_cost_usd'], .0000168) + + def test_redirect_is_refused_before_credential_can_cross_hosts(self): + with self.assertRaises(HTTPError): + jev._NoRedirect().redirect_request(Mock(full_url=jev.ENDPOINT), None, 307, + 'Redirect', {}, 'https://elsewhere.invalid/') + + def test_keychain_only_queried_when_explicitly_named_and_env_takes_precedence(self): + with patch.dict('os.environ', {}, clear=True), patch.object(jev.subprocess, 'run') as run: + self.assertIsNone(jev.load_api_key()) + run.assert_not_called() + with self.assertRaises(ValueError): + jev.load_api_key(keychain_service='dvidia-typesafe') + run.return_value = subprocess.CompletedProcess([], 0, 'secret-for-test\n', '') + self.assertEqual(jev.load_api_key(keychain_service='dvidia-typesafe', keychain_account='dvidia-research'), 'secret-for-test') + self.assertEqual(run.call_args.args[0], ['/usr/bin/security', 'find-generic-password', '-s', 'dvidia-typesafe', '-a', 'dvidia-research', '-w']) + with patch.dict('os.environ', {'TYPESAFE_API_KEY': 'env-secret'}), patch.object(jev.subprocess, 'run') as run: + self.assertEqual(jev.load_api_key(keychain_service='dvidia-typesafe', keychain_account='dvidia-research'), 'env-secret') + run.assert_not_called() + + +if __name__ == '__main__': + unittest.main() From 81c4b0a19b8e03b329102c6b6950848e124846d3 Mon Sep 17 00:00:00 2001 From: DVIDIA Date: Sun, 18 Oct 2026 23:39:25 -0400 Subject: [PATCH 2/5] Publish inspectable image-goal protocol and native validation evidence --- .github/workflows/image-goal.yml | 2 + README.md | 10 + benchmarks/image-goal-20261018/README.md | 72 ++ .../native-validation.json | 745 ++++++++++++++++++ docs/image-goal-benchmark.md | 11 + publishing/huggingface/publish_image_goals.py | 347 ++++++++ .../huggingface/test_publish_image_goals.py | 130 +++ scripts/verify-image-goal-fixtures.py | 304 +++++++ src/dvidia_training/image_goal_report.py | 288 +++++++ tests/test_image_goal_report.py | 181 +++++ 10 files changed, 2090 insertions(+) create mode 100644 benchmarks/image-goal-20261018/README.md create mode 100644 benchmarks/image-goal-20261018/native-validation.json create mode 100644 publishing/huggingface/publish_image_goals.py create mode 100644 publishing/huggingface/test_publish_image_goals.py create mode 100644 scripts/verify-image-goal-fixtures.py create mode 100644 src/dvidia_training/image_goal_report.py create mode 100644 tests/test_image_goal_report.py diff --git a/.github/workflows/image-goal.yml b/.github/workflows/image-goal.yml index 1400bbe..8656397 100644 --- a/.github/workflows/image-goal.yml +++ b/.github/workflows/image-goal.yml @@ -22,3 +22,5 @@ jobs: python -m unittest tests.test_jev_client tests.test_image_goal_benchmark tests.test_image_goal_freeze tests.test_image_goal_scenes tests.test_image_goal_report -v + - name: Preserve published evidence and existing Space files + run: python -m unittest discover -s publishing/huggingface -p 'test_publish_image_goals.py' -v diff --git a/README.md b/README.md index 657b46f..a16ab0e 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,16 @@ **Research alpha.** The recorded demo covers the original one-scene simulation pilot; the frozen benchmark is a separate experiment. Training runs locally. No human-video action bridge or hardware skill is established. +**[Image-goal verification v0.1](docs/image-goal-benchmark.md).** +32 native PiPER fixtures test missing targets, wrong arrangements, unreleased +contact and moving shoes that render identically to settled shoes. The runnable +Jev 1.14.0 protocol compares images alone with images plus simulator measurements. +Native fixtures and offline checks are validated; **Jev accuracy and latency are +not yet measured** while its API credential is pending. +[Inspect the cases](https://dvidia-robot-simulation-lab.static.hf.space/image-goals/v01/index.html) · +[Research note](https://research.dvidia.org/papers/image-goal-jev-v01/) · +[Validation evidence](benchmarks/image-goal-20261018/README.md). + **[Shoe graspability v0.3](https://research.dvidia.org/papers/shoe-graspability-v03/).** A controlled direct-pickup lab separates rigid object shape, 30 g versus 300 g load, and mounted versus absent wrist tool. All 24 fresh simulations are valid; diff --git a/benchmarks/image-goal-20261018/README.md b/benchmarks/image-goal-20261018/README.md new file mode 100644 index 0000000..c7ea66d --- /dev/null +++ b/benchmarks/image-goal-20261018/README.md @@ -0,0 +1,72 @@ +# Image-goal benchmark v0.1: native validation and Jev protocol + +**Jev has not been run.** The 32 native cases and evaluation tools are tested; +all 64 scheduled Jev trials remain unrun while secure API configuration is +pending. There are no measured Jev accuracy, latency or cost results here. + +- [Research note](https://research.dvidia.org/papers/image-goal-jev-v01/) +- [Inspect every case on Hugging Face](https://dvidia-robot-simulation-lab.static.hf.space/image-goals/v01/index.html) +- [Local run instructions](../../docs/image-goal-benchmark.md) +- [Independent native validation](native-validation.json) + +## Executed local checks + +The frozen cohort has four layouts and eight conditions, with 8 physically +complete and 24 incomplete states. Eight cases lack the intended target, counting +the missing-target and orange-decoy conditions. Four moving/settled pairs have +identical PNGs and different physical completion labels. Native generation took +6.297 seconds on this host, including model construction, stepping and rendering. +That is not model inference or robot reaction time. + +The independent verifier checks all 67 original files (5,019,443 bytes), derives +the physical predicates and dwell from recorded numeric measurements/history, +cross-checks raw poses and velocities, and validates telemetry extrema. All +32 cases pass these consistency checks. Repeating verification gives identical +JSON. A separate native test suite rebuilds the scenes and renders the controls. + +The shoe is a 30 g rigid proxy narrowed to half the catalog width. Real simulated +finger contact establishes the unreleased negative control under unchanged +PiPER actuator gains. This is not evidence of full-size shoe pickup, learning, +deformation or physical robot transfer. + +## Interpretation before any model calls + +Images alone have 16 visibly incomplete cases and 16 cases whose physical +completion is unobservable from one frame. The telemetry condition has 8 complete +and 24 incomplete cases. These class balances imply simple reference baselines: + +| Fixed answer | RGB expected-verdict accuracy | RGB + telemetry expected-verdict accuracy | +| --- | ---: | ---: | +| Always incomplete | 16/32 | 24/32 | +| Always unknown | 16/32 | 0/32 | +| Always complete | 0/32 | 8/32 | + +These are arithmetic baselines from the frozen labels, **not Jev results**. +Accuracy must be read alongside false completion, coverage, justified abstention +and request errors. Four layouts are correlated clusters; this cohort does not +establish generalization or production tail latency. + +## Frozen provenance + +- Native generator/evaluator source: `19acf0734d63ecebc07140c8d84302a54baee4c4`. +- Base PiPER study: `191856b`. +- Protocol SHA-256: `d48e363d24cc3c9f3be67ffb87390aa3f9b37272bd28d4b9b22424ccf336fba4`. +- Cases SHA-256: `f8b1b205b16e9833df7bd7063e0572355794dcede49a024f5d73ae6f65bbb1aa`. +- Evaluation-plan SHA-256: `5f02006190d97bdaca43f139f5bf221cece23bba868a5e9ee0f424618ea87d8f`. + +The presentation, independent verifier and publisher are later additions in this +branch. Their source and artifact hashes are retained separately; they do not +change the frozen cases or Jev prompts. Native GPU rendering can vary slightly +between runs; the four within-run alias pairs must remain identical. + +```sh +python scripts/verify-image-goal-fixtures.py \ + --fixtures PATH_TO_DOWNLOADED_FIXTURES --output native-validation.json +``` + +The verifier reads frozen native observations; it does not re-run physics or +contact Jev. Use the documented generator command for a fresh native run. + +Original DVIDIA code and authored fixtures: MIT. PiPER assets preserve their +upstream MIT notice. Hosted Jev remains subject to TypeSafe's separate terms; +no weights, credentials or customer footage are distributed. diff --git a/benchmarks/image-goal-20261018/native-validation.json b/benchmarks/image-goal-20261018/native-validation.json new file mode 100644 index 0000000..6a7ea49 --- /dev/null +++ b/benchmarks/image-goal-20261018/native-validation.json @@ -0,0 +1,745 @@ +{ + "schema": "dvidia.image-goal-native-validation/v1", + "status": "passed", + "scope": "Independent recomputation from frozen native state measurements/history and raw coordinates. No images re-rendered, no fresh physics run, no model service called and no hardware validation.", + "fixture_generation_scope": "Authored native physical-state verification. Four deterministic layouts, not statistical independent real-world scenes or a trained policy benchmark.", + "telemetry_scope": "Privileged native state and contact history; not an implemented camera or hardware estimator.", + "source_digests": { + "protocol_sha256": "d48e363d24cc3c9f3be67ffb87390aa3f9b37272bd28d4b9b22424ccf336fba4", + "cases_sha256": "f8b1b205b16e9833df7bd7063e0572355794dcede49a024f5d73ae6f65bbb1aa", + "integrity_sha256": "e8b6bbe4f2928e85f6e88d16af7d719dc32b1a3b588ce9633805061528cbe682", + "verifier_sha256": "514be68c1dbd9ecb797f46615524f2bffa8654a5df3a62b90f4f8cc897414c08" + }, + "files_verified": 67, + "bytes_verified": 5019443, + "additional_runner_metadata": [ + { + "path": "evaluation-plan.json", + "bytes": 24662, + "sha256": "5f02006190d97bdaca43f139f5bf221cece23bba868a5e9ee0f424618ea87d8f", + "scope": "Runner metadata added after fixture generation; fingerprinted here, not claimed to be covered by the original fixture integrity manifest." + } + ], + "native_generation_wall_seconds": 6.2972065838985145, + "native_generation_timing_scope": "Recorded fixture construction, stepping and rendering; excludes Jev inference and this verification.", + "recorded_mujoco_version": "3.15.0", + "verification_host": { + "system": "Darwin", + "release": "25.5.0", + "machine": "arm64", + "processor": "arm", + "logical_cpu_count": 10, + "python_version": "3.12.9", + "scope": "Host running this verifier; CPU model, RAM and GPU specifications are not inferred." + }, + "criteria": { + "position_tolerance_m": 0.025, + "yaw_tolerance_rad": 0.2617993877991494, + "linear_speed_max_m_s": 0.01, + "angular_speed_max_rad_s": 0.05, + "support_force_min_n": 0.01, + "robot_force_max_n": 0.01, + "continuous_dwell_s": 0.2 + }, + "case_count": 32, + "category_counts": { + "complete": 4, + "missing_target": 4, + "moving_alias": 4, + "occluded": 4, + "unreleased": 4, + "wrong_object": 4, + "wrong_orientation": 4, + "wrong_position": 4 + }, + "physical_complete": 8, + "physical_incomplete": 24, + "zero_solver_warnings": true, + "no_added_object_attachment_constraints": true, + "no_recorded_external_forces": true, + "native_equalities_per_model": 1, + "equality_scope": "Recorded upstream gripper coupling count; verified against frozen state metadata, without recompiling models.", + "all_physical_labels_recomputed": true, + "all_telemetry_extrema_recomputed": true, + "unreleased_rgb_unknown_cases": 4, + "aliases": [ + { + "case_id": "layout-01-moving_alias", + "pixel_identical_to": "layout-01-complete", + "png_sha256": "550516b315cb198407b0ef3be997af97805c11fcfe6b587b1091affd8060e8fb", + "png_bytes": 79345, + "byte_identical": true, + "pose_identical": true, + "velocity_and_completion_differ": true + }, + { + "case_id": "layout-02-moving_alias", + "pixel_identical_to": "layout-02-complete", + "png_sha256": "a3221e451c24b164f0348f2cb6517d2bd2589a444c3d8d71a9942a98dee0bdf3", + "png_bytes": 79868, + "byte_identical": true, + "pose_identical": true, + "velocity_and_completion_differ": true + }, + { + "case_id": "layout-03-moving_alias", + "pixel_identical_to": "layout-03-complete", + "png_sha256": "9e077553701f0b54929386b69abc0d4a531f4dd8d0ef8c164dfbd4e11c2eec22", + "png_bytes": 78462, + "byte_identical": true, + "pose_identical": true, + "velocity_and_completion_differ": true + }, + { + "case_id": "layout-04-moving_alias", + "pixel_identical_to": "layout-04-complete", + "png_sha256": "76cba333e8bdf56f1b9e6011247095507ede97697ef99ada4b47ea7d1c71d3fa", + "png_bytes": 77762, + "byte_identical": true, + "pose_identical": true, + "velocity_and_completion_differ": true + } + ], + "cases": [ + { + "id": "layout-01-complete", + "category": "complete", + "present": true, + "complete": true, + "physical_conditions_dwell_s": 0.2, + "supported": true, + "released": true, + "position_error_m": 1.0856805225155038e-07, + "yaw_error_rad": 2.1965561156353097e-09, + "table_support_force_n": 0.2942999990014351, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.860075820885874e-10, + "angular_speed_rad_s": 2.7627744970784057e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "complete" + } + }, + { + "id": "layout-01-missing_target", + "category": "missing_target", + "present": false, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": false, + "released": true, + "position_error_m": null, + "yaw_error_rad": null, + "table_support_force_n": 0.0, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": null, + "angular_speed_rad_s": null, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-01-wrong_object", + "category": "wrong_object", + "present": false, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": false, + "released": true, + "position_error_m": null, + "yaw_error_rad": null, + "table_support_force_n": 0.0, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": null, + "angular_speed_rad_s": null, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-01-wrong_position", + "category": "wrong_position", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 0.1699999772538407, + "yaw_error_rad": 2.1965561161782934e-09, + "table_support_force_n": 0.29429999900143505, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.860075879684076e-10, + "angular_speed_rad_s": 2.7627747009029746e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-01-wrong_orientation", + "category": "wrong_orientation", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 1.0856805256849158e-07, + "yaw_error_rad": 1.5707963245983405, + "table_support_force_n": 0.2942999990014349, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.860076309132707e-10, + "angular_speed_rad_s": 2.7627746996178486e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-01-unreleased", + "category": "unreleased", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": false, + "position_error_m": 0.00574563428980446, + "yaw_error_rad": 0.08440615838439973, + "table_support_force_n": 0.2995624074490855, + "robot_contact_force_n": 0.7619265340093823, + "linear_speed_m_s": 9.969011014506973e-05, + "angular_speed_rad_s": 0.003462491733548149, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-01-moving_alias", + "category": "moving_alias", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 1.0856805225155038e-07, + "yaw_error_rad": 2.1965561156353097e-09, + "table_support_force_n": 0.3624151236609657, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 0.12, + "angular_speed_rad_s": 2.7627744970784057e-09, + "native_history_samples": 0, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-01-occluded", + "category": "occluded", + "present": true, + "complete": true, + "physical_conditions_dwell_s": 0.2, + "supported": true, + "released": true, + "position_error_m": 1.0856805225155038e-07, + "yaw_error_rad": 2.1965561156353097e-09, + "table_support_force_n": 0.2942999990014351, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.860075820885874e-10, + "angular_speed_rad_s": 2.7627744970784057e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "complete" + } + }, + { + "id": "layout-02-complete", + "category": "complete", + "present": true, + "complete": true, + "physical_conditions_dwell_s": 0.2, + "supported": true, + "released": true, + "position_error_m": 1.0856805243386722e-07, + "yaw_error_rad": 2.1965563118397924e-09, + "table_support_force_n": 0.294299999001435, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.86007602226177e-10, + "angular_speed_rad_s": 2.7627747344185066e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "complete" + } + }, + { + "id": "layout-02-missing_target", + "category": "missing_target", + "present": false, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": false, + "released": true, + "position_error_m": null, + "yaw_error_rad": null, + "table_support_force_n": 0.0, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": null, + "angular_speed_rad_s": null, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-02-wrong_object", + "category": "wrong_object", + "present": false, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": false, + "released": true, + "position_error_m": null, + "yaw_error_rad": null, + "table_support_force_n": 0.0, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": null, + "angular_speed_rad_s": null, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-02-wrong_position", + "category": "wrong_position", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 0.1699999858102173, + "yaw_error_rad": 2.1965563118397924e-09, + "table_support_force_n": 0.29429999900143505, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.86007596264554e-10, + "angular_speed_rad_s": 2.7627749443387563e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-02-wrong_orientation", + "category": "wrong_orientation", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 1.0856805246501234e-07, + "yaw_error_rad": 1.5707963245983367, + "table_support_force_n": 0.2942999990014351, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.86007540976695e-10, + "angular_speed_rad_s": 2.762774938435085e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-02-unreleased", + "category": "unreleased", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": false, + "position_error_m": 0.00112237494618688, + "yaw_error_rad": 0.00569434757841164, + "table_support_force_n": 0.3014209771616913, + "robot_contact_force_n": 0.7693653082066636, + "linear_speed_m_s": 0.00014160096890228054, + "angular_speed_rad_s": 0.0009941703968526277, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-02-moving_alias", + "category": "moving_alias", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 1.0856805243386722e-07, + "yaw_error_rad": 2.1965563118397924e-09, + "table_support_force_n": 0.36239906646539655, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 0.12, + "angular_speed_rad_s": 2.7627747344185066e-09, + "native_history_samples": 0, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-02-occluded", + "category": "occluded", + "present": true, + "complete": true, + "physical_conditions_dwell_s": 0.2, + "supported": true, + "released": true, + "position_error_m": 1.0856805243386722e-07, + "yaw_error_rad": 2.1965563118397924e-09, + "table_support_force_n": 0.294299999001435, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.86007602226177e-10, + "angular_speed_rad_s": 2.7627747344185066e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "complete" + } + }, + { + "id": "layout-03-complete", + "category": "complete", + "present": true, + "complete": true, + "physical_conditions_dwell_s": 0.2, + "supported": true, + "released": true, + "position_error_m": 1.0856805227409043e-07, + "yaw_error_rad": 2.19655620081749e-09, + "table_support_force_n": 0.2942999990014348, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.860076492373484e-10, + "angular_speed_rad_s": 2.762774894432646e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "complete" + } + }, + { + "id": "layout-03-missing_target", + "category": "missing_target", + "present": false, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": false, + "released": true, + "position_error_m": null, + "yaw_error_rad": null, + "table_support_force_n": 0.0, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": null, + "angular_speed_rad_s": null, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-03-wrong_object", + "category": "wrong_object", + "present": false, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": false, + "released": true, + "position_error_m": null, + "yaw_error_rad": null, + "table_support_force_n": 0.0, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": null, + "angular_speed_rad_s": null, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-03-wrong_position", + "category": "wrong_position", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 0.16999996884296226, + "yaw_error_rad": 2.19655620081749e-09, + "table_support_force_n": 0.2942999990014348, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.86007647555542e-10, + "angular_speed_rad_s": 2.762774718512132e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-03-wrong_orientation", + "category": "wrong_orientation", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 1.0856805243889215e-07, + "yaw_error_rad": 1.5707963245983425, + "table_support_force_n": 0.29429999900143444, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.86007598072673e-10, + "angular_speed_rad_s": 2.7627747152164955e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-03-unreleased", + "category": "unreleased", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": false, + "position_error_m": 0.0006551670719148867, + "yaw_error_rad": 0.003703847249796388, + "table_support_force_n": 0.2639875277923961, + "robot_contact_force_n": 0.7863052928276089, + "linear_speed_m_s": 0.00010986191904928065, + "angular_speed_rad_s": 0.003368756823804283, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-03-moving_alias", + "category": "moving_alias", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 1.0856805227409043e-07, + "yaw_error_rad": 2.19655620081749e-09, + "table_support_force_n": 0.362350885628384, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 0.12, + "angular_speed_rad_s": 2.762774894432646e-09, + "native_history_samples": 0, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-03-occluded", + "category": "occluded", + "present": true, + "complete": true, + "physical_conditions_dwell_s": 0.2, + "supported": true, + "released": true, + "position_error_m": 1.0856805227409043e-07, + "yaw_error_rad": 2.19655620081749e-09, + "table_support_force_n": 0.2942999990014348, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.860076492373484e-10, + "angular_speed_rad_s": 2.762774894432646e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "complete" + } + }, + { + "id": "layout-04-complete", + "category": "complete", + "present": true, + "complete": true, + "physical_conditions_dwell_s": 0.2, + "supported": true, + "released": true, + "position_error_m": 1.0856805211389336e-07, + "yaw_error_rad": 2.1965565755177607e-09, + "table_support_force_n": 0.294299999001435, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.860076016164131e-10, + "angular_speed_rad_s": 2.7627745907984584e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "complete" + } + }, + { + "id": "layout-04-missing_target", + "category": "missing_target", + "present": false, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": false, + "released": true, + "position_error_m": null, + "yaw_error_rad": null, + "table_support_force_n": 0.0, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": null, + "angular_speed_rad_s": null, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-04-wrong_object", + "category": "wrong_object", + "present": false, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": false, + "released": true, + "position_error_m": null, + "yaw_error_rad": null, + "table_support_force_n": 0.0, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": null, + "angular_speed_rad_s": null, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-04-wrong_position", + "category": "wrong_position", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 0.16999999337338886, + "yaw_error_rad": 2.1965565755177607e-09, + "table_support_force_n": 0.29429999900143494, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.86007605782391e-10, + "angular_speed_rad_s": 2.762774742684285e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-04-wrong_orientation", + "category": "wrong_orientation", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 1.0856805194707237e-07, + "yaw_error_rad": 1.57079632459834, + "table_support_force_n": 0.294299999001435, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.86007598437561e-10, + "angular_speed_rad_s": 2.7627747672417554e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "incomplete", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-04-unreleased", + "category": "unreleased", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": false, + "position_error_m": 0.0010491825564488413, + "yaw_error_rad": 0.03690074331083231, + "table_support_force_n": 0.3144476609893772, + "robot_contact_force_n": 0.790406083741399, + "linear_speed_m_s": 0.00010064949189093267, + "angular_speed_rad_s": 0.002606933470747755, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-04-moving_alias", + "category": "moving_alias", + "present": true, + "complete": false, + "physical_conditions_dwell_s": 0.0, + "supported": true, + "released": true, + "position_error_m": 1.0856805211389336e-07, + "yaw_error_rad": 2.1965565755177607e-09, + "table_support_force_n": 0.3623192350084462, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 0.12, + "angular_speed_rad_s": 2.7627745907984584e-09, + "native_history_samples": 0, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "incomplete" + } + }, + { + "id": "layout-04-occluded", + "category": "occluded", + "present": true, + "complete": true, + "physical_conditions_dwell_s": 0.2, + "supported": true, + "released": true, + "position_error_m": 1.0856805211389336e-07, + "yaw_error_rad": 2.1965565755177607e-09, + "table_support_force_n": 0.294299999001435, + "robot_contact_force_n": 0.0, + "linear_speed_m_s": 9.860076016164131e-10, + "angular_speed_rad_s": 2.7627745907984584e-09, + "native_history_samples": 100, + "expected_verdict": { + "rgb": "unknown", + "rgb_telemetry": "complete" + } + } + ] +} diff --git a/docs/image-goal-benchmark.md b/docs/image-goal-benchmark.md index 1406a2c..12c6b20 100644 --- a/docs/image-goal-benchmark.md +++ b/docs/image-goal-benchmark.md @@ -71,6 +71,17 @@ Without credentials, the runner makes no network requests and writes explicit unrun trials. Accuracy, latency and cost remain unmeasured. Use a fresh output directory for each evaluation; existing evidence cannot be overwritten. +Build the static case viewer after the evaluation returns: + +```sh +python -m dvidia_training.image_goal_report runs/image-goal-fixtures \ + runs/image-goal-report --results runs/image-goal-evaluation/results.json +python -m http.server 8080 --bind 127.0.0.1 --directory runs/image-goal-report +``` + +Open `http://127.0.0.1:8080/`. The viewer makes no provider requests. An active or +interrupted request must be reconciled before its run can be exported. + The evaluation seals images, labels, prompts, implementation hashes and the full paired trial inventory before calls. It prevalidates every request, executes at most 64 requests serially with no automatic retries, and retains all results diff --git a/publishing/huggingface/publish_image_goals.py b/publishing/huggingface/publish_image_goals.py new file mode 100644 index 0000000..abd89f1 --- /dev/null +++ b/publishing/huggingface/publish_image_goals.py @@ -0,0 +1,347 @@ +"""Append a reviewed image-goal report to the existing public simulation Space. + +The only mutable existing files are the three root pointers. Historical Git/LFS +entries and attributes must remain identical. Every new file stays in regular +Git so HTML and JavaScript remain on the static Space origin. Credentials use +the existing memory-only macOS Keychain adapter, never a command-line token. +""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path +import re +import sys +import tempfile + +from publish import PublicationError, checked_remote, client, verified_downloads +from publish_prerequisites import (encoded, git_blob, inspect_bytes, matches_tree, + plain, public_bytes, read, record, tree) + +REPO_ID = "Dvidia/robot-simulation-lab" +NAMESPACE = "image-goals/v01" +ROOT_FILES = {"index.html", "README.md", "provenance.json"} +SHA = re.compile(r"[a-f0-9]{40}\Z") +MAX_FILE = 4 * 1024 * 1024 +MAX_TOTAL = 64 * 1024 * 1024 +MAX_FILES = 256 +CONDITIONS = {"rgb", "rgb_telemetry"} +PRIVATE_KEYS = {"authorization", "api_key", "apikey", "credential", "credentials", "token", "headers", "raw_response", "response"} + + +def inventory(directory): + directory = plain(directory) + if not directory.is_dir(): + raise PublicationError("Expected a plain report export directory.") + payload, size = {}, 0 + for path in sorted(directory.rglob("*")): + plain(path) + if path.is_dir(): + continue + name = path.relative_to(directory).as_posix() + raw = read(path, MAX_FILE) + if name in {"protocol.sha256", NAMESPACE + "/protocol.sha256"}: + if re.fullmatch(rb"[a-f0-9]{64}\n", raw) is None: + raise PublicationError("Expected exactly one newline-terminated SHA-256 digest.") + inspect_bytes(name + ".txt", raw) + else: + inspect_bytes(name, raw) + if name.endswith(".json"): + stack = [json.loads(raw)] + while stack: + value = stack.pop() + if isinstance(value, dict): + if any(key.lower() in PRIVATE_KEYS for key in value): + raise PublicationError("Remove raw provider responses, headers and credential fields from the public export.") + stack.extend(value.values()) + elif isinstance(value, list): + stack.extend(value) + payload[name] = raw + size += len(raw) + if len(payload) > MAX_FILES or size > MAX_TOTAL: + raise PublicationError("Image-goal report exceeds its publication bound.") + return payload + + +def check_export(export, source_commit): + payload = inventory(export) + required = {"index.html", "report.json", "report-integrity.json", "integrity.json", "publication-integrity.json", "cases.json", + "protocol.json", "evaluation-plan.json", "results.json", "provenance.json", "LICENSE"} + if not required <= payload.keys(): + raise PublicationError("Missing image-goal report, protocol, results, source provenance or license.") + seal = json.loads(payload["publication-integrity.json"]) + if seal.get("files") != {name: record(raw) for name, raw in payload.items() if name != "publication-integrity.json"}: + raise PublicationError("Export does not match its complete integrity seal.") + provenance = json.loads(payload["provenance.json"]) + plan = json.loads(payload["evaluation-plan.json"]) + result = json.loads(payload["results.json"]) + if result.get("status") == "running" or result.get("active_trial") is not None: + raise PublicationError("Resolve the active or interrupted request before publication.") + public = json.loads(payload["report.json"]) + cases = json.loads(payload["cases.json"]).get("cases", []) + if (provenance.get("source_commit") != source_commit or plan.get("source_commit") != source_commit + or plan.get("model") != "jev-1.14.0" or result.get("model") != plan["model"] + or public.get("model") != plan["model"] + or result.get("evaluation_plan_sha256") != record(payload["evaluation-plan.json"])["sha256"]): + raise PublicationError("Pinned source, model and evaluated plan must agree.") + if not {"cases.json", "protocol.json"} <= plan.get("files", {}).keys(): + raise PublicationError("Frozen fixture inventory is incomplete.") + for name, expected_file in plan["files"].items(): + if name not in payload or record(payload[name]) != expected_file: + raise PublicationError("Frozen fixture bytes changed or were omitted.") + if (not 1 <= len(cases) <= 32 or len({case["id"] for case in cases}) != len(cases) + or set(plan.get("conditions", [])) != CONDITIONS): + raise PublicationError("Expected the frozen paired image-goal cohort.") + expected = {(case["id"], condition) for case in cases for condition in CONDITIONS} + for rows in (plan.get("trials", []), result.get("results", []), public.get("results", [])): + identities = [(row["case_id"], row["condition"]) for row in rows] + if len(identities) != len(set(identities)) or set(identities) != expected: + raise PublicationError("Every planned case and condition must remain in the report, including unrun trials.") + public_cases = {case["id"]: case for case in public.get("cases", [])} + if len(public_cases) != len(cases): + raise PublicationError("Viewer case inventory differs from the frozen cohort.") + for case in cases: + shown = public_cases.get(case["id"], {}) + if (shown.get("ground_truth") != {key: case["physical_truth"][key] for key in ("present", "complete")} + or shown.get("expected_verdict") != case.get("expected_verdict") + or shown.get("expected_presence") != case.get("expected_presence")): + raise PublicationError("Viewer labels differ from frozen physical and observability references.") + for key in ("current_image", "goal_image"): + if shown.get(key) not in payload or case.get(key) not in payload or payload[shown[key]] != payload[case[key]]: + raise PublicationError("Viewer images differ from the evaluated fixture images.") + if public.get("status") != result.get("status") or provenance.get("provider_status") != result.get("status"): + raise PublicationError("Provider status must agree across publication artifacts.") + pending = all(row.get("status") == "not_run" for row in result["results"]) + if result.get("status") == "not_run": + if not pending or result.get("metrics") is not None: + raise PublicationError("Unrun Jev evidence must have no measured provider metrics.") + for row in result["results"]: + if (row.get("presence") not in (None, "unknown") or row.get("verdict") not in (None, "unknown") + or any(row.get(key) is not None for key in ("p_present", "p_complete", "online_total_ms", "estimated_input_cost_usd")) + or row.get("attempts", 0) != 0): + raise PublicationError("Unrun provider trials cannot carry predictions or service measurements.") + if b"Jev has not been run" not in payload["index.html"]: + raise PublicationError("Unrun provider status must be visible in the report.") + elif result.get("status") != "completed" or pending: + raise PublicationError("Unexpected provider run status.") + viewer_seal = json.loads(payload["report-integrity.json"]) + for name, expected_file in viewer_seal.get("files", {}).items(): + if name not in payload or record(payload[name]) != expected_file: + raise PublicationError("Viewer export differs from its original file inventory.") + source_root = str(Path(__file__).resolve().parents[2] / "src") + if source_root not in sys.path: + sys.path.insert(0, source_root) + from dvidia_training.image_goal_report import build_report + with tempfile.TemporaryDirectory(prefix="dvidia-image-goal-publication-") as temporary: + expected_viewer = Path(temporary).resolve() / "viewer" + expected_inventory = build_report(export, expected_viewer) + if viewer_seal != expected_inventory: + raise PublicationError("Viewer inventory is not the independently regenerated report.") + for name in (*expected_inventory["files"], "report-integrity.json"): + if payload.get(name) != (expected_viewer / name).read_bytes(): + raise PublicationError("Viewer differs from independently regenerated results and presentation.") + return payload + + +def snapshot(output, parent_commit): + if SHA.fullmatch(parent_commit) is None: + raise PublicationError("Snapshot requires an exact parent commit.") + destination = plain(output) + if destination.exists(): + raise PublicationError("Snapshot output must be fresh.") + from huggingface_hub import HfApi + api = HfApi(token=False) + info = api.repo_info(REPO_ID, repo_type="space") + checked_remote(info, "space") + if info.sha != parent_commit: + raise PublicationError("Space moved before its snapshot.") + previous = tree(api, parent_commit) + if any(name == NAMESPACE or name.startswith(NAMESPACE + "/") for name in previous): + raise PublicationError("Image-goal namespace already exists.") + if not ROOT_FILES | {".gitattributes"} <= previous.keys(): + raise PublicationError("Existing Space root is incomplete.") + roots = {name: public_bytes(name, parent_commit, MAX_FILE) for name in ROOT_FILES | {".gitattributes"}} + if any(not matches_tree(raw, previous[name]) for name, raw in roots.items()): + raise PublicationError("Root bytes differ from the pinned tree.") + destination.mkdir(parents=True) + for name, raw in roots.items(): + (destination / name).write_bytes(raw) + (destination / "snapshot.json").write_bytes(encoded({"schema": "dvidia.image-goal-parent/v1", + "repo_id": REPO_ID, "namespace": NAMESPACE, "parent_commit": parent_commit, "tree": previous})) + return {"parent_commit": parent_commit, "retained_tree_entries": len(previous)} + + +def updated_roots(parent, source_commit, export): + status = json.loads(export["results.json"])["status"] + pending = status == "not_run" + count = len(json.loads(export["cases.json"])["cases"]) + description = (f"Explore {count} frozen PiPER image-goal cases, including absent targets, wrong arrangements and visually identical moving scenes. " + + ("Native fixture validation is published. Jev accuracy and service latency are not yet measured." + if pending else "Inspect all paired Jev judgments, failures, abstentions and measured service latency.") + + " Synthetic diagnostic fixtures; passive evaluation; no robot controller or hardware qualification.") + index = parent["index.html"].decode() + marker = '
' + if index.count(marker) != 1 or 'id="image-goals-title"' in index: + raise PublicationError("Expected one existing graspability section and no image-goal section.") + card = ('

Does it only look finished?

' + f'

{description}

Inspect image goals and physical references →
') + index = index.replace(marker, card + marker) + readme = parent["README.md"] + ("\n\n## Image-goal benchmark v0.1\n\n" + f"[Inspect the image-goal benchmark]({NAMESPACE}/index.html). {description}\n\n" + f"[Pinned source](https://github.com/Dvidia-Inference/dvidia-training/tree/{source_commit}) · " + "[Research](https://research.dvidia.org/papers/image-goal-jev-v01/)\n").encode() + provenance = json.loads(parent["provenance.json"]) + if "image_goals_v01" in provenance: + raise PublicationError("Image-goal root provenance already exists.") + provenance["image_goals_v01"] = {"path": NAMESPACE, "source_commit": source_commit, + "integrity": record(export["publication-integrity.json"]), "provider_status": status, + "mode": "recorded-passive-evaluation", "learned_policy": False, "physical_robot_ready": False} + return {"index.html": index.encode(), "README.md": readme, "provenance.json": encoded(provenance)} + + +def stage(export, parent, output, source_commit): + if SHA.fullmatch(source_commit) is None: + raise PublicationError("An exact source commit is required.") + parent, destination, export = plain(parent), plain(output), plain(export) + if any(destination == source or source in destination.parents or destination in source.parents for source in (parent, export)): + raise PublicationError("Stage output must be separate from its inputs.") + baseline = json.loads(read(parent / "snapshot.json", 2 * 1024 * 1024)) + if (baseline.get("repo_id") != REPO_ID or baseline.get("namespace") != NAMESPACE + or SHA.fullmatch(baseline.get("parent_commit", "")) is None): + raise PublicationError("Unexpected parent snapshot.") + previous = baseline["tree"] + parent_files = {name: read(parent / name, MAX_FILE) for name in ROOT_FILES} + attributes = read(parent / ".gitattributes", 65536) + if any(not matches_tree(raw, previous[name]) for name, raw in {**parent_files, ".gitattributes": attributes}.items()): + raise PublicationError("Pinned parent root bytes changed.") + release = check_export(export, source_commit) + payload = {NAMESPACE + "/" + name: raw for name, raw in release.items()} + payload.update(updated_roots(parent_files, source_commit, release)) + if set(payload) & set(previous) != ROOT_FILES: + raise PublicationError("Only root pointers may replace existing files.") + destination.mkdir(parents=True, exist_ok=False) + for name, raw in payload.items(): + path = destination / "payload" / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(raw) + (destination / "parent-gitattributes.txt").write_bytes(attributes) + plan = {"schema": "dvidia.image-goal-publication/v1", "repo_id": REPO_ID, "namespace": NAMESPACE, + "source_commit": source_commit, "parent_commit": baseline["parent_commit"], "parent_tree": previous, + "parent_attributes": record(attributes), "files": {name: record(raw) for name, raw in sorted(payload.items())}} + (destination / "plan.json").write_bytes(encoded(plan)) + check(destination / "plan.json") + return {"plan": str(destination / "plan.json"), "files": len(payload), "bytes": sum(map(len, payload.values()))} + + +def check(path): + path = plain(path) + plan = json.loads(read(path, 2 * 1024 * 1024)) + if (plan.get("schema") != "dvidia.image-goal-publication/v1" or plan.get("repo_id") != REPO_ID + or plan.get("namespace") != NAMESPACE + or any(SHA.fullmatch(plan.get(key, "")) is None for key in ("source_commit", "parent_commit"))): + raise PublicationError("Unexpected image-goal publication plan.") + payload = inventory(path.parent / "payload") + if plan.get("files") != {name: record(raw) for name, raw in payload.items()}: + raise PublicationError("Payload changed after staging.") + if (set(payload) & set(plan["parent_tree"]) != ROOT_FILES + or any(name not in ROOT_FILES and not name.startswith(NAMESPACE + "/") for name in payload) + or any(name == NAMESPACE or name.startswith(NAMESPACE + "/") for name in plan["parent_tree"])): + raise PublicationError("Only the fresh namespace and three root pointers may change.") + attributes = read(path.parent / "parent-gitattributes.txt", 65536) + if record(attributes) != plan["parent_attributes"] or not matches_tree(attributes, plan["parent_tree"][".gitattributes"]): + raise PublicationError("Pinned Hub metadata changed.") + check_export(path.parent / "payload" / NAMESPACE, plan["source_commit"]) + return plan, payload + + +def publish(path, gate_path): + path = plain(path) + plan, payload = check(path) + gate = json.loads(read(gate_path, 1024 * 1024)) + if (gate.get("source_commit") != plan["source_commit"] + or any(gate.get(key) is not True for key in ("tests_passed", "browser_qa_passed", "benchmark_reviewed")) + or gate.get("integrity_sha256") != plan["files"][NAMESPACE + "/publication-integrity.json"]["sha256"]): + raise PublicationError("Publication requires source-bound test, browser and evidence review.") + api, token = client(path.parent / "publisher-cache") + if any(token.encode() in raw for raw in payload.values()): + raise PublicationError("Credential occurs in upload bytes.") + from huggingface_hub.utils import get_session + session = get_session() + session.timeout = type(session.timeout)(60., connect=20., read=120., write=60., pool=20.) + if "Dvidia" not in {org.get("name") for org in api.whoami().get("orgs", [])}: + raise PublicationError("Publisher is not a member of the intended organization.") + pending_path = plain(path.parent / "pending.json") + plan_hash = record(read(path, 2 * 1024 * 1024))["sha256"] + if pending_path.exists(): + pending = json.loads(read(pending_path)) + if pending.get("plan_sha256") != plan_hash or SHA.fullmatch(pending.get("commit", "")) is None: + raise PublicationError("Retry receipt belongs to another plan.") + revision = pending["commit"] + else: + info = api.repo_info(REPO_ID, repo_type="space") + checked_remote(info, "space") + if info.sha != plan["parent_commit"] or tree(api, info.sha) != plan["parent_tree"]: + raise PublicationError("Space changed after its reviewed snapshot.") + from huggingface_hub import CommitOperationAdd + operations = [] + for name, raw in payload.items(): + operation = CommitOperationAdd(path_in_repo=name, path_or_fileobj=raw) + # The pinned SDK respects this classification. Regular files stay + # on the static origin; never move active entrypoints to LFS/CDN. + operation._upload_mode = "regular" + operation._should_ignore = False + operations.append(operation) + result = api.create_commit(REPO_ID, repo_type="space", parent_commit=plan["parent_commit"], + operations=operations, commit_message="Publish image-goal diagnostics with explicit provider status and complete evidence") + revision = result.oid + pending_path.write_bytes(encoded({"commit": revision, "plan_sha256": plan_hash})) + after = tree(api, revision) + if set(after) != set(plan["parent_tree"]) | set(payload): + raise PublicationError("Unexpected paths added or removed.") + unchanged = set(plan["parent_tree"]) - set(payload) + if any(after[name] != plan["parent_tree"][name] for name in unchanged): + raise PublicationError("Historical file or Hub metadata changed.") + if any(after[name].get("lfs_sha256") is not None or not matches_tree(raw, after[name]) for name, raw in payload.items()): + raise PublicationError("New files must exactly match regular Git bytes.") + verified_downloads({"repo_id": REPO_ID, "repo_type": "space", "files": [ + {"path": name, **expected} for name, expected in plan["files"].items()]}, revision, False) + receipt = {"schema": "dvidia.image-goal-hf-publication/v1", "repo_id": REPO_ID, "namespace": NAMESPACE, + "source_commit": plan["source_commit"], "parent_commit": plan["parent_commit"], "commit": revision, + "plan_sha256": plan_hash, "verified_public_files": len(payload), "unchanged_historical_tree_entries": len(unchanged), + "all_new_files_regular_git": True, "files": plan["files"]} + (path.parent / "after-tree.json").write_bytes(encoded(after)) + (path.parent / "publication-receipt.json").write_bytes(encoded(receipt)) + return receipt + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="action", required=True) + snap = sub.add_parser("snapshot"); snap.add_argument("output"); snap.add_argument("--parent-commit", required=True) + stage_parser = sub.add_parser("stage") + for name in ("export", "parent", "output"): + stage_parser.add_argument(name) + stage_parser.add_argument("--source-commit", required=True) + check_parser = sub.add_parser("check"); check_parser.add_argument("plan") + publish_parser = sub.add_parser("publish"); publish_parser.add_argument("plan"); publish_parser.add_argument("gate") + args = parser.parse_args() + try: + if args.action == "snapshot": + result = snapshot(args.output, args.parent_commit) + elif args.action == "stage": + result = stage(args.export, args.parent, args.output, args.source_commit) + elif args.action == "check": + plan, payload = check(args.plan) + result = {"checked_files": len(payload), "source_commit": plan["source_commit"]} + else: + result = publish(args.plan, args.gate) + print(json.dumps(result, indent=2)) + except Exception as error: + # Exception messages from SDK transports can contain request context. + print(json.dumps({"status": "stopped", "error_type": type(error).__name__, + "http_status": getattr(getattr(error, "response", None), "status_code", None)})) + raise SystemExit(1) + + +if __name__ == "__main__": + main() diff --git a/publishing/huggingface/test_publish_image_goals.py b/publishing/huggingface/test_publish_image_goals.py new file mode 100644 index 0000000..af02ded --- /dev/null +++ b/publishing/huggingface/test_publish_image_goals.py @@ -0,0 +1,130 @@ +"""Publication rejects changed history, hidden failed cases and fabricated scores.""" +import base64 +import json +from pathlib import Path +import sys +import tempfile +import unittest + +import publish_image_goals as publisher +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'src')) +from dvidia_training.image_goal_report import build_report + +PNG = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jfZkAAAAASUVORK5CYII=') + + +class ImageGoalPublicationTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name).resolve() + self.parent, self.export = self.root / 'parent', self.root / 'export' + self.parent.mkdir(); self.export.mkdir() + self.original = {'index.html': b'
prior work
', + 'README.md': b'---\nsdk: static\n---\nPrior releases\n', + 'provenance.json': b'{"previous":{"keep":true}}\n', '.gitattributes': b'*.bin filter=lfs diff=lfs merge=lfs -text\n'} + self.tree = {name: {'blob': publisher.git_blob(raw), 'size': len(raw), 'lfs_sha256': None} for name, raw in self.original.items()} + self.tree['prerequisites/v03/native.bin'] = {'blob': 'c'*40, 'size': 99, 'lfs_sha256': 'd'*64} + for name, raw in self.original.items(): + (self.parent / name).write_bytes(raw) + (self.parent / 'snapshot.json').write_bytes(publisher.encoded({'repo_id': publisher.REPO_ID, + 'namespace': publisher.NAMESPACE, 'parent_commit': 'a'*40, 'tree': self.tree})) + case = {'id': 'missing', 'physical_truth': {'present': False, 'complete': False}, + 'current_image': 'images/current.png', 'goal_image': 'images/goal.png', + 'expected_verdict': {key: 'incomplete' for key in publisher.CONDITIONS}, + 'expected_presence': {key: 'absent' for key in publisher.CONDITIONS}} + rows = [{'case_id': 'missing', 'condition': key, 'status': 'not_run'} for key in sorted(publisher.CONDITIONS)] + self.release = {'images/current.png': PNG, + 'images/goal.png': PNG, 'cases.json': publisher.encoded({'cases': [case]}), + 'protocol.json': b'{"scope":"synthetic fixture"}\n', 'integrity.json': b'{"files":{}}\n', 'protocol.sha256': b'd'*64+b'\n', 'LICENSE': b'MIT test fixture', + 'provenance.json': publisher.encoded({'source_commit': 'b'*40, 'provider_status': 'not_run'})} + plan = {'source_commit': 'b'*40, 'model': 'jev-1.14.0', 'conditions': sorted(publisher.CONDITIONS), 'trials': rows, + 'files': {name: publisher.record(self.release[name]) for name in ('cases.json', 'protocol.json', 'integrity.json', 'protocol.sha256', 'images/current.png', 'images/goal.png')}} + self.release['evaluation-plan.json'] = publisher.encoded(plan) + self.release['results.json'] = publisher.encoded({'model': 'jev-1.14.0', 'status': 'not_run', 'metrics': None, + 'results': rows, 'evaluation_plan_sha256': publisher.record(self.release['evaluation-plan.json'])['sha256']}) + self.save() + viewer = self.root / 'viewer' + build_report(self.export, viewer) + for path in viewer.rglob('*'): + if path.is_file(): + self.release[path.relative_to(viewer).as_posix()] = path.read_bytes() + self.save() + + def save(self): + for name, raw in self.release.items(): + path = self.export / name + path.parent.mkdir(parents=True, exist_ok=True); path.write_bytes(raw) + (self.export / 'publication-integrity.json').write_bytes(publisher.encoded({'files': {name: publisher.record(raw) for name, raw in self.release.items()}})) + + def stage(self): + return publisher.stage(self.export, self.parent, self.root / 'stage', 'b'*40) + + def test_new_namespace_preserves_history_and_visible_pending_status(self): + result = self.stage() + plan, payload = publisher.check(result['plan']) + self.assertEqual(plan['parent_tree'], self.tree) + self.assertEqual(set(payload) & self.tree.keys(), publisher.ROOT_FILES) + self.assertIn(self.original['README.md'], payload['README.md']) + self.assertIn(b'not yet measured', payload['index.html']) + self.assertEqual(json.loads(payload['provenance.json'])['previous'], {'keep': True}) + self.assertEqual(payload[publisher.NAMESPACE + '/integrity.json'], self.release['integrity.json']) + self.assertIn(publisher.NAMESPACE + '/publication-integrity.json', payload) + + def test_frozen_checksum_extension_is_narrowly_bounded(self): + self.release['protocol.sha256'] = b'not a checksum\n'; self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'SHA-256 digest'): + self.stage() + + def test_omitted_unrun_trial_cannot_shrink_denominator(self): + result = json.loads(self.release['results.json']); result['results'].pop() + self.release['results.json'] = publisher.encoded(result); self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'Every planned case'): + self.stage() + + def test_pending_provider_cannot_publish_measured_accuracy(self): + result = json.loads(self.release['results.json']); result['metrics'] = {'accuracy': 1.0} + self.release['results.json'] = publisher.encoded(result); self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'no measured provider metrics'): + self.stage() + + def test_raw_response_and_credentials_are_rejected(self): + result = json.loads(self.release['results.json']); result['response'] = {'private': 'raw'} + self.release['results.json'] = publisher.encoded(result); self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'raw provider responses'): + self.stage() + + def test_resealed_changed_fixture_or_viewer_label_is_rejected(self): + self.release['images/current.png'] = b'new image'; self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'Frozen fixture bytes'): + self.stage() + self.release['images/current.png'] = PNG + report = json.loads(self.release['report.json']); report['cases'][0]['ground_truth']['complete'] = True + self.release['report.json'] = publisher.encoded(report); self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'Viewer labels'): + self.stage() + + def test_staged_payload_tampering_and_parent_tampering_are_rejected(self): + result = self.stage() + (self.root / 'stage/payload/index.html').write_text('changed') + with self.assertRaisesRegex(publisher.PublicationError, 'Payload changed'): + publisher.check(result['plan']) + (self.parent / 'README.md').write_text('changed') + with self.assertRaisesRegex(publisher.PublicationError, 'root bytes changed'): + publisher.stage(self.export, self.parent, self.root / 'other-stage', 'b'*40) + + def test_resealed_fabricated_provider_answer_or_summary_is_rejected(self): + public = json.loads(self.release['report.json']) + public['results'][0].update(status='ok', verdict='complete', p_complete=1, + decision={'present': True, 'complete': True}) + self.release['report.json'] = publisher.encoded(public) + viewer_seal = json.loads(self.release['report-integrity.json']) + viewer_seal['files']['report.json'] = publisher.record(self.release['report.json']) + self.release['report-integrity.json'] = publisher.encoded(viewer_seal) + self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'independently regenerated'): + self.stage() + + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/verify-image-goal-fixtures.py b/scripts/verify-image-goal-fixtures.py new file mode 100644 index 0000000..05130db --- /dev/null +++ b/scripts/verify-image-goal-fixtures.py @@ -0,0 +1,304 @@ +#!/usr/bin/env python3 +"""Independently audit frozen image-goal evidence, without rendering or an API. + +The numerical predicate below deliberately does not call the simulator's +``observe``/``_measure`` functions or trust their stored completion flags. +It checks recorded native evidence, not a fresh simulation or hardware run. +""" +from __future__ import annotations + +import argparse +from collections import Counter +from hashlib import sha256 +import json +import math +import os +from pathlib import Path, PurePosixPath +import platform +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from dvidia_training.image_goal_scenes import CRITERIA, expected_labels # noqa: E402 + +SCHEMA = "dvidia.image-goal-native-validation/v1" +METRICS = ("position_error_m", "yaw_error_rad", "linear_speed_m_s", + "angular_speed_rad_s", "table_support_force_n", "robot_contact_force_n") + + +def require(condition, message): + if not condition: + raise ValueError(message) + + +def finite(value, label): + require(type(value) in (int, float) and math.isfinite(value), f"Invalid finite number: {label}") + return float(value) + + +def same_number(actual, expected, label): + if expected is None: + require(actual is None, f"Expected null: {label}") + else: + require(math.isclose(finite(actual,label), finite(expected,label), rel_tol=1e-10, abs_tol=1e-12), + f"Numeric mismatch: {label}") + + +def regular_file(root, name): + require(isinstance(name,str) and "\\" not in name, "Manifest path must be POSIX text") + relative=PurePosixPath(name) + require(not relative.is_absolute() and relative.as_posix()==name and + all(part not in ("", ".", "..") for part in relative.parts), "Unsafe manifest path") + path=root.joinpath(*relative.parts) + require(all(not p.is_symlink() for p in (path,*path.parents) if p==root or root in p.parents), + f"Symlink in fixture path: {name}") + require(path.is_file(), f"Missing regular fixture file: {name}") + return path + + +def load_json(path): + def reject(value): + raise ValueError(f"Non-finite JSON value in {path.name}: {value}") + return json.loads(path.read_text(encoding="utf-8"),parse_constant=reject) + + +def physical_predicate(row, criteria): + """Recompute from numbers, ignoring instantaneous_conditions_met labels.""" + require(type(row["target_present"]) is bool, "Presence must be a boolean") + if not row["target_present"]: + require(all(row[key] is None for key in METRICS[:4]), "Absent target has pose/speed evidence") + return False + values={key:finite(row[key],key) for key in METRICS} + require(all(value>=0 for value in values.values()), "Negative scalar physical measurement") + return (values["position_error_m"]<=criteria["position_tolerance_m"] + and values["yaw_error_rad"]<=criteria["yaw_tolerance_rad"] + and values["linear_speed_m_s"]<=criteria["linear_speed_max_m_s"] + and values["angular_speed_rad_s"]<=criteria["angular_speed_max_rad_s"] + and values["table_support_force_n"]>criteria["support_force_min_n"] + and values["robot_contact_force_n"]<=criteria["robot_force_max_n"]) + + +def recompute_truth(state, criteria, dt): + current=state["current_measurement"] + history=state["history"] + require(isinstance(history,list) and len(history)<=100, "Unexpected native observation window") + current_ok=physical_predicate(current,criteria) + count=0 + for row in reversed(history): + if not physical_predicate(row,criteria): + break + count+=1 + dwell=count*dt if current_ok else 0. + return {"present":current["target_present"], + "complete":bool(current_ok and dwell+1e-12>=criteria["continuous_dwell_s"]), + "physical_conditions_dwell_s":dwell, + "position_error_m":current["position_error_m"], + "yaw_error_rad":current["yaw_error_rad"], + "supported":finite(current["table_support_force_n"],"support")>criteria["support_force_min_n"], + "released":finite(current["robot_contact_force_n"],"robot contact")<=criteria["robot_force_max_n"], + "linear_speed_m_s":current["linear_speed_m_s"], + "angular_speed_rad_s":current["angular_speed_rad_s"]} + + +def compare_truth(stored, derived, label): + require(set(stored)==set(derived), f"Truth fields differ: {label}") + for key,value in derived.items(): + if type(value) is bool: + require(type(stored[key]) is bool and stored[key] is value, f"Truth mismatch: {label}.{key}") + else: + same_number(stored[key],value,f"{label}.{key}") + + +def verify_snapshot(state, layout): + """Cross-check raw native coordinates, independent of measurement labels.""" + measurement=state["current_measurement"] + for name in ("qpos","qvel","ctrl"): + require(isinstance(state[name],list), f"Invalid {name}") + for value in state[name]: + finite(value,name) + require(len(state["ctrl"])==7, "Expected seven pinned PiPER actuator commands") + if not measurement["target_present"]: + return + qpos,qvel=state["qpos"],state["qvel"] + require(len(qpos)==15 and len(qvel)==14, "Expected PiPER plus one free shoe") + for actual,recorded in zip(qpos[8:11],measurement["target_position_m"]): + same_number(actual,recorded,"free-body target position") + xy_error=math.hypot(qpos[8]-layout["xy_m"][0],qpos[9]-layout["xy_m"][1]) + same_number(xy_error,measurement["position_error_m"],"XY goal error") + w,x,y,z=qpos[11:15] + same_number(sum(v*v for v in (w,x,y,z)),1.,"unit free-body quaternion") + yaw=math.atan2(2*(w*z+x*y),1-2*(y*y+z*z)) + yaw_error=abs(math.atan2(math.sin(yaw-layout["yaw_rad"]),math.cos(yaw-layout["yaw_rad"]))) + same_number(yaw_error,measurement["yaw_error_rad"],"yaw goal error") + same_number(math.sqrt(sum(v*v for v in qvel[8:11])),measurement["linear_speed_m_s"],"native linear speed") + same_number(math.sqrt(sum(v*v for v in qvel[11:14])),measurement["angular_speed_rad_s"],"native angular speed") + + +def verify_telemetry(state, inputs, dt): + current,history=state["current_measurement"],state["history"] + require("complete" not in inputs and "instantaneous_conditions_met" not in inputs, + "A verdict leaked into measured inputs") + for key in ("target_present","target_position_m",*METRICS): + require(inputs[key]==current[key], f"Telemetry snapshot differs: {key}") + require(inputs["native_observation_count"]==len(history), "Native tick count differs") + require(inputs["target_present_observation_count"]==sum(row["target_present"] for row in history), + "Presence observation count differs") + same_number(inputs["native_timestep_s"],dt,"native dt") + same_number(inputs["observation_window_s"],len(history)*dt,"observation duration") + for key in METRICS: + values=[row[key] for row in history if row[key] is not None] + expected={"min":min(values) if values else None,"max":max(values) if values else None} + for statistic,value in expected.items(): + same_number(inputs["native_window_extrema"][key][statistic],value,f"{key}.{statistic}") + + +def verify(fixtures): + root=Path(fixtures).absolute() + require(root.is_dir() and not root.is_symlink(), "Fixtures must be a regular directory") + integrity=load_json(regular_file(root,"integrity.json")) + records=integrity["files"] + names=[record["path"] for record in records] + require(len(set(names))==len(names), "Duplicate integrity path") + actual={p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_file()} + required=set(names)|{"integrity.json"} + require(required<=actual and actual-required<={"evaluation-plan.json"}, + "Integrity manifest coverage differs or unknown additional files exist") + additional_metadata=[] + for name in sorted(actual-required): + data=regular_file(root,name).read_bytes() + additional_metadata.append({"path":name,"bytes":len(data),"sha256":sha256(data).hexdigest(), + "scope":"Runner metadata added after fixture generation; fingerprinted here, not claimed to be covered by the original fixture integrity manifest."}) + byte_count=0 + for record in records: + path=regular_file(root,record["path"]) + data=path.read_bytes() + require(len(data)==record["bytes"], f"Byte count differs: {record['path']}") + require(sha256(data).hexdigest()==record["sha256"], f"Hash differs: {record['path']}") + byte_count+=len(data) + protocol_path=regular_file(root,"protocol.json") + protocol=load_json(protocol_path) + manifest=load_json(regular_file(root,"cases.json")) + protocol_digest=sha256(protocol_path.read_bytes()).hexdigest() + require(protocol_digest==regular_file(root,"protocol.sha256").read_text().strip()==manifest["protocol_sha256"], + "Protocol digests disagree") + require(protocol["criteria"]==manifest["criteria"]==CRITERIA,"Criteria differ from the frozen verifier contract") + criteria=protocol["criteria"] + dt=finite(protocol["physics_timestep_s"],"physics timestep") + same_number(dt,.002,"fixed native timestep") + layouts={row["id"]:row for row in protocol["layouts"]} + cases=manifest["cases"] + require(len(cases)==32 and {case["id"] for case in cases}==set(protocol["case_ids"]), + "Frozen case coverage differs") + category_counts=Counter(case["category"] for case in cases) + require(set(category_counts)==set(protocol["categories"]) and set(category_counts.values())=={4}, + "Unbalanced category coverage") + rows=[];states={} + for case in cases: + identifier=case["id"] + state=load_json(regular_file(root,case["state_file"])) + states[identifier]=state + truth=recompute_truth(state,criteria,dt) + compare_truth(state["physical_truth"],truth,identifier+".state") + compare_truth(case["physical_truth"],truth,identifier+".case") + verify_snapshot(state,layouts[case["layout_id"]]) + verify_telemetry(state,case["measured_inputs"],dt) + require(state["solver_warnings"]==0,"Native solver warning recorded") + require(state["object_attachment_constraints_added"]==0,"Object attachment recorded") + require(state["model_equalities"]==1,"Unexpected equality constraint count") + require(state["external_forces_applied"] is False,"Undeclared external force recorded") + for row in (state["current_measurement"],*state["history"]): + require(row["instantaneous_conditions_met"] is physical_predicate(row,criteria), + f"Stored instantaneous label disagrees with numbers: {identifier}") + labels=expected_labels(case["category"],truth) + for key in labels: + require(case[key]==labels[key], f"Observability labels differ: {identifier}") + visibility=case["visibility"] + if case["category"] in ("occluded","missing_target","wrong_object"): + require(visibility["visible_target_pixels"]==0 and visibility["target_box_xyxy_normalized"] is None, + "An invisible/absent target has a visible box") + else: + require(visibility["visible_target_pixels"]>0 and visibility["target_box_xyxy_normalized"] is not None, + "Visible target is missing its native segmentation box") + if case["category"]=="unreleased": + require(truth["supported"] and not truth["released"] and not truth["complete"], + "Unreleased fixture lacks supported robot contact") + require(truth["position_error_m"]<=criteria["position_tolerance_m"] and + truth["yaw_error_rad"]<=criteria["yaw_tolerance_rad"], + "Unreleased fixture is outside the goal pose") + require(case["expected_verdict"]=={"rgb":"unknown","rgb_telemetry":"incomplete"}, + "Unreleased RGB overclaims physical contact force") + rows.append({"id":identifier,"category":case["category"],"present":truth["present"], + "complete":truth["complete"],"physical_conditions_dwell_s":truth["physical_conditions_dwell_s"], + "supported":truth["supported"],"released":truth["released"], + "position_error_m":truth["position_error_m"],"yaw_error_rad":truth["yaw_error_rad"], + "table_support_force_n":state["current_measurement"]["table_support_force_n"], + "robot_contact_force_n":state["current_measurement"]["robot_contact_force_n"], + "linear_speed_m_s":truth["linear_speed_m_s"],"angular_speed_rad_s":truth["angular_speed_rad_s"], + "native_history_samples":len(state["history"]),"expected_verdict":labels["expected_verdict"]}) + index={case["id"]:case for case in cases} + aliases=[] + require(len(manifest["aliases"])==4,"Expected four exact-pixel aliases") + require({pair["case_id"] for pair in manifest["aliases"]}== + {case["id"] for case in cases if case["category"]=="moving_alias"}, + "Alias manifest does not cover all four moving controls") + for pair in manifest["aliases"]: + moving,stationary=index[pair["case_id"]],index[pair["pixel_identical_to"]] + require(moving["category"]=="moving_alias" and stationary["category"]=="complete" and + moving["layout_id"]==stationary["layout_id"],"Alias identity differs") + a=regular_file(root,moving["current_image"]).read_bytes() + b=regular_file(root,stationary["current_image"]).read_bytes() + require(a==b,"Current-image PNG alias bytes differ") + require(states[moving["id"]]["qpos"]==states[stationary["id"]]["qpos"],"Alias native poses differ") + require(states[moving["id"]]["qvel"]!=states[stationary["id"]]["qvel"],"Alias native velocities do not differ") + require(not moving["physical_truth"]["complete"] and stationary["physical_truth"]["complete"], + "Alias physical truth does not differ") + require(len(states[moving["id"]]["history"])==0,"Post-intervention observation time was invented") + aliases.append({**pair,"png_sha256":sha256(a).hexdigest(),"png_bytes":len(a), + "byte_identical":True,"pose_identical":True,"velocity_and_completion_differ":True}) + completed=sum(row["complete"] for row in rows) + require(completed==8 and len(rows)-completed==24,"Expected 8 complete and 24 incomplete states") + return {"schema":SCHEMA,"status":"passed", + "scope":"Independent recomputation from frozen native state measurements/history and raw coordinates. No images re-rendered, no fresh physics run, no model service called and no hardware validation.", + "fixture_generation_scope":manifest["scope"],"telemetry_scope":manifest["telemetry_scope"], + "source_digests":{"protocol_sha256":protocol_digest, + "cases_sha256":sha256(regular_file(root,"cases.json").read_bytes()).hexdigest(), + "integrity_sha256":sha256(regular_file(root,"integrity.json").read_bytes()).hexdigest(), + "verifier_sha256":sha256(Path(__file__).read_bytes()).hexdigest()}, + "files_verified":len(records),"bytes_verified":byte_count, + "additional_runner_metadata":additional_metadata, + "native_generation_wall_seconds":manifest["generation_wall_seconds"], + "native_generation_timing_scope":"Recorded fixture construction, stepping and rendering; excludes Jev inference and this verification.", + "recorded_mujoco_version":manifest["mujoco_version"], + "verification_host":{"system":platform.system(),"release":platform.release(), + "machine":platform.machine(),"processor":platform.processor() or None, + "logical_cpu_count":os.cpu_count(),"python_version":platform.python_version(), + "scope":"Host running this verifier; CPU model, RAM and GPU specifications are not inferred."}, + "criteria":criteria,"case_count":len(rows),"category_counts":dict(sorted(category_counts.items())), + "physical_complete":completed,"physical_incomplete":len(rows)-completed, + "zero_solver_warnings":True,"no_added_object_attachment_constraints":True, + "no_recorded_external_forces":True,"native_equalities_per_model":1, + "equality_scope":"Recorded upstream gripper coupling count; verified against frozen state metadata, without recompiling models.", + "all_physical_labels_recomputed":True,"all_telemetry_extrema_recomputed":True, + "unreleased_rgb_unknown_cases":4,"aliases":aliases,"cases":rows} + + +def main(argv=None): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument("--fixtures",required=True,type=Path) + parser.add_argument("--output",required=True,type=Path) + args=parser.parse_args(argv) + require(args.output.absolute()!=args.fixtures.absolute() and args.fixtures.absolute() not in args.output.absolute().parents, + "Write validation outside the frozen fixture directory") + try: + result=verify(args.fixtures) + code=0 + except (ValueError,KeyError,TypeError,OSError,json.JSONDecodeError) as error: + result={"schema":SCHEMA,"status":"failed","error":str(error)} + code=1 + args.output.parent.mkdir(parents=True,exist_ok=True) + args.output.write_text(json.dumps(result,indent=2,allow_nan=False)+"\n",encoding="utf-8") + print(json.dumps({key:result[key] for key in ("status","case_count","physical_complete","physical_incomplete","files_verified","bytes_verified","error") if key in result})) + return code + + +if __name__=="__main__": + raise SystemExit(main()) diff --git a/src/dvidia_training/image_goal_report.py b/src/dvidia_training/image_goal_report.py new file mode 100644 index 0000000..3199c80 --- /dev/null +++ b/src/dvidia_training/image_goal_report.py @@ -0,0 +1,288 @@ +"""Build a portable, credential-free viewer for an image-goal experiment. + +Only the explicit public fields below are exported. Provider payloads, headers, +raw responses and arbitrary free-form errors never enter the report. The viewer +does not call Jev, issue robot commands or turn an unrun experiment into a score. +""" +from __future__ import annotations + +import argparse +from hashlib import sha256 +import json +import math +from pathlib import Path +import re + +_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,119}\Z") +_SUCCESS = {"ok", "success", "completed"} +_PENDING = {"pending", "not_run", "not-run", "unrun", "missing_api_key", "missing_credentials"} +_MAX_JSON = 8 * 1024 * 1024 +_MAX_IMAGE = 4 * 1024 * 1024 + + +def _json(path: Path) -> dict: + if path.is_symlink() or not path.is_file() or path.stat().st_size > _MAX_JSON: + raise ValueError("Expected a bounded, plain experiment JSON file.") + value = json.loads(path.read_bytes(), parse_constant=lambda _: (_ for _ in ()).throw(ValueError("Nonfinite JSON."))) + if not isinstance(value, dict): + raise ValueError("Experiment JSON must be an object.") + return value + + +def _name(value, label): + if not isinstance(value, str) or not _NAME.fullmatch(value): + raise ValueError(f"Invalid public {label}.") + return value + + +def _number(value, probability=False): + if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value): + return None + if value < 0 or (probability and value > 1): + return None + return value + + +def _boolean(value): + return value if type(value) is bool else None + + +def _image(root, name, assets): + if not isinstance(name, str) or "\\" in name: + raise ValueError("Image reference must be a relative artifact path.") + relative = Path(name) + if relative.is_absolute() or any(part in {"", ".", ".."} for part in relative.parts): + raise ValueError("Image reference must remain inside the experiment.") + path = root / relative + if any(item.is_symlink() for item in (path, *path.parents)) or not path.is_file(): + raise ValueError("Image reference must be a plain local file.") + if path.stat().st_size > _MAX_IMAGE: + raise ValueError("Report image exceeds its byte bound.") + raw = path.read_bytes() + suffix = ".png" if raw.startswith(b"\x89PNG\r\n\x1a\n") else ".jpg" if raw.startswith(b"\xff\xd8\xff") else None + if suffix is None: + raise ValueError("Report images must be PNG or JPEG.") + public = "images/" + sha256(raw).hexdigest() + suffix + assets[public] = raw + return public + + +def _decisions(row): + """Read explicit decisions only; probabilities do not imply a threshold.""" + decision = row.get("decision", {}) + if not isinstance(decision, dict): + decision = {} + fallbacks = {"present": {"present": True, "absent": False}.get(row.get("presence")), + "complete": {"complete": True, "incomplete": False}.get(row.get("verdict"))} + return {key: _boolean(decision.get(key, row.get("predicted_" + key, fallbacks[key]))) for key in ("present", "complete")} + + +def _quantiles(values): + if not values: + return {"n": 0, "p50": None, "p95": None, "p99": None, "max": None} + values = sorted(values) + def percentile(p): + offset = (len(values) - 1) * p + low, high = math.floor(offset), math.ceil(offset) + return values[low] + (values[high] - values[low]) * (offset - low) + return {"n": len(values), "p50": percentile(.5), "p95": percentile(.95), "p99": percentile(.99), "max": values[-1]} + + +def _summary(cases, rows, conditions): + truth = {case["id"]: case["ground_truth"] for case in cases} + case_by_id = {case["id"]: case for case in cases} + summaries = [] + for condition in conditions: + selected = [row for row in rows if row["condition"] == condition] + attempted = [row for row in selected if row["status"] not in _PENDING] + answered = [row for row in selected if row["status"] in _SUCCESS] + complete_answers = [row for row in answered if row["decision"]["complete"] is not None and truth[row["case_id"]]["complete"] is not None] + incomplete = [case for case in cases if case["ground_truth"]["complete"] is False] + absent = [case for case in cases if case["ground_truth"]["present"] is False] + incomplete_answers = [row for row in complete_answers if truth[row["case_id"]]["complete"] is False] + absent_answers = [row for row in answered if truth[row["case_id"]]["present"] is False and row["decision"]["present"] is not None] + false_complete = sum(row["decision"]["complete"] is True for row in incomplete_answers) + false_present = sum(row["decision"]["present"] is True for row in absent_answers) + known = [case for case in cases if case["expected_verdict"].get(condition) in {"complete", "incomplete"}] + ambiguous = [case for case in cases if case["expected_verdict"].get(condition) == "unknown"] + expected_cases = [case for case in cases if case["expected_verdict"].get(condition) is not None] + expected_correct = sum(row["verdict"] == case_by_id[row["case_id"]]["expected_verdict"].get(condition) + for row in answered if case_by_id[row["case_id"]]["expected_verdict"].get(condition) is not None) + known_correct = sum(row["verdict"] == case_by_id[row["case_id"]]["expected_verdict"].get(condition) + for row in answered if case_by_id[row["case_id"]]["expected_verdict"].get(condition) in {"complete", "incomplete"}) + justified_abstentions = sum(row["verdict"] == "unknown" for row in answered + if case_by_id[row["case_id"]]["expected_verdict"].get(condition) == "unknown") + durations = lambda group: [row["timing_ms"]["total_ms"] for row in group if row["timing_ms"].get("total_ms") is not None] + summaries.append({"condition": condition, "planned": len(cases), "attempted": len(attempted), + "responses": len(answered), "errors": len(attempted) - len(answered), + "not_run": len(cases) - len(attempted), + "completion_accuracy": {"correct": sum(row["decision"]["complete"] == truth[row["case_id"]]["complete"] for row in complete_answers), "answered": len(complete_answers), "planned": sum(case["ground_truth"]["complete"] is not None for case in cases)}, + "completion_abstentions": sum(row["decision"]["complete"] is None for row in answered), + "observability_aware_accuracy": {"correct": expected_correct, "planned": len(expected_cases)}, + "known_verdict_accuracy": {"correct": known_correct, "planned": len(known)}, + "ambiguous_abstention": {"correct": justified_abstentions, "planned": len(ambiguous)}, + "false_completion": {"count": false_complete, "answered_incomplete": len(incomplete_answers), "planned_incomplete": len(incomplete), + "valid_incomplete": sum(truth[row["case_id"]]["complete"] is False for row in answered)}, + "false_presence": {"count": false_present, "answered_absent": len(absent_answers), "planned_absent": len(absent), + "valid_absent": sum(truth[row["case_id"]]["present"] is False for row in answered)}, + "known_usage_estimated_input_cost_usd": sum(row["estimated_input_cost_usd"] for row in attempted if row["estimated_input_cost_usd"] is not None) + if any(row["estimated_input_cost_usd"] is not None for row in attempted) else None, + "unknown_cost_attempts": sum(row["estimated_input_cost_usd"] is None for row in attempted), + "all_attempt_latency_ms": _quantiles(durations(attempted)), + "successful_response_latency_ms": _quantiles(durations(answered))}) + return summaries + + +def build_report(run_dir, output_dir, *, results_path=None): + """Export an offline report into a fresh directory and return its manifest. + + The public data contract is deliberately small: cases contain id, category, + current_image, goal_image and physical_truth.{present,complete}; result rows + contain case_id, condition, status, explicit decision booleans, probabilities, + verdict and finite timing_ms values. Extra input fields are discarded. + """ + root, output = Path(run_dir).absolute(), Path(output_dir).absolute() + if any(path.is_symlink() for path in (root, *root.parents, output, *output.parents)): + raise ValueError("Report paths must not contain symlinks.") + if output.exists(): + raise ValueError("Report output must be a fresh directory.") + catalog = _json(root / "cases.json") + result_path = Path(results_path).absolute() if results_path is not None else root / "results.json" + if any(path.is_symlink() for path in (result_path, *result_path.parents)): + raise ValueError("Results path must not contain symlinks.") + if results_path is not None and not result_path.is_file(): + raise ValueError("Explicit results path does not exist.") + result = _json(result_path) if result_path.exists() else {"status": "not_run", "results": []} + if result.get("status") == "running" or result.get("active_trial") is not None: + raise ValueError("An active or interrupted request must be resolved before exporting a report.") + source_cases = catalog.get("cases") + if not isinstance(source_cases, list) or not 1 <= len(source_cases) <= 128: + raise ValueError("A report requires one to 128 explicit cases.") + cases, assets = [], {} + for case in source_cases: + ground_truth = case.get("physical_truth", case.get("ground_truth", {})) + if not isinstance(ground_truth, dict): + raise ValueError("Ground truth must be an object.") + cases.append({"id": _name(case.get("id", case.get("case_id")), "case id"), + "category": _name(case.get("category", "unspecified"), "case category"), + "current_image": _image(root, case.get("current_image"), assets), + "goal_image": _image(root, case.get("goal_image"), assets), + "ground_truth": {key: _boolean(ground_truth.get(key)) for key in ("present", "complete")}, + "expected_verdict": {key: value for key, value in case.get("expected_verdict", {}).items() + if key in {"rgb", "rgb_telemetry"} and value in {"complete", "incomplete", "unknown"}}, + "expected_presence": {key: value for key, value in case.get("expected_presence", {}).items() + if key in {"rgb", "rgb_telemetry"} and value in {"present", "absent", "unknown"}}}) + ids = {case["id"] for case in cases} + if len(ids) != len(cases): + raise ValueError("Case ids must be unique.") + raw_rows = result.get("results", result.get("rows", [])) + if not isinstance(raw_rows, list) or len(raw_rows) > 1024: + raise ValueError("Expected a bounded list of result rows.") + declared = result.get("planned_conditions", catalog.get("conditions", ["rgb", "rgb_telemetry"])) + if not isinstance(declared, list) or not 1 <= len(declared) <= 8: + raise ValueError("Declare one to eight input conditions.") + conditions = [_name(value, "condition") for value in declared] + if len(set(conditions)) != len(conditions): + raise ValueError("Conditions must be unique.") + rows, seen = [], set() + for row in raw_rows: + case_id, condition = _name(row.get("case_id"), "case id"), _name(row.get("condition"), "condition") + if case_id not in ids or condition not in conditions or (case_id, condition) in seen: + raise ValueError("Every result must name one unique planned case and condition.") + seen.add((case_id, condition)) + timing = row.get("timing_ms", {}) + if isinstance(timing, (int, float)): + timing = {"total_ms": timing} + if not isinstance(timing, dict): + timing = {} + # Restrict names as well as values: metadata is not a covert error log. + public_timing = {key: _number(timing.get(key)) for key in ("total_ms", "prepare_ms", "request_ms", "capture_ms", "encode_ms", "upload_ms", "decode_ms") if key in timing} + verdict = row.get("verdict") + if verdict not in {"complete", "not_complete", "incomplete", "abstain", "unknown", "missing", "present", "error", None}: + verdict = "unknown" + status = _name(row.get("status", "unknown"), "status") + pending = status in _PENDING + rows.append({"case_id": case_id, "condition": condition, + "status": status, "decision": {"present": None, "complete": None} if pending else _decisions(row), + "p_present": None if pending else _number(row.get("p_present"), True), + "p_complete": None if pending else _number(row.get("p_complete"), True), + "verdict": None if pending else verdict, "timing_ms": {} if pending else public_timing, + "estimated_input_cost_usd": None if pending else _number(row.get("estimated_input_cost_usd"))}) + data = {"schema": "dvidia.image-goal-public-report/v1", "status": _name(result.get("status", "not_run"), "run status"), + "model": _name(result.get("model", "jev-1.14.0"), "model"), + "conditions": conditions, "cases": cases, "results": rows, + "summary": _summary(cases, rows, conditions), + "scope": "Synthetic simulation fixtures; passive image-goal judgments; no robot control or trained weights."} + encoded = json.dumps(data, indent=2, allow_nan=False).encode() + b"\n" + embedded = json.dumps(data, allow_nan=False).replace("<", "\\u003c").replace(">", "\\u003e").replace("&", "\\u0026") + html = _HTML.replace("__PUBLIC_DATA__", embedded).encode() + payload = {"index.html": html, "report.json": encoded, **assets} + manifest = {"schema": "dvidia.image-goal-report-files/v1", "files": {name: {"bytes": len(raw), "sha256": sha256(raw).hexdigest()} for name, raw in payload.items()}} + output.mkdir(parents=True) + for name, raw in payload.items(): + path = output / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(raw) + (output / "report-integrity.json").write_text(json.dumps(manifest, indent=2) + "\n") + return manifest + + +_HTML = r''' +Image goals · DVIDIA Research +
DVIDIA
+

Robotics research · Image goals / 0.1

Does it only look finished?

A goal image describes the destination. This experiment asks Jev to judge the current scene, then compares its answer with the simulator’s physical state. Missing shoes and misleading arrangements stay in the results.

Loading the recorded experiment.

+

Current simulated scene
Current observation
Requested goal arrangement
Goal reference

Physical reference

Recorded simulator labels. A single image may not reveal hidden contact or sustained support.

Jev judgment

A remote research judgment. This page sends no images, runs no model and commands no robot.

+

Count the misses. Keep the clock honest.

Judgment accuracy uses the frozen, observability-aware verdict for every planned case: complete, incomplete, or unknown. A correct abstention matters when an image cannot establish hidden motion, contact or dwell. Errors and cases not run remain in the planned denominator; no score is claimed before requests are attempted. Physical false-completion and missing-target counts retain every planned negative case, with answered coverage shown separately.

Decision checkCorrectDenominator
Latency scopeTimed requestsp50p95p99Maximum

Timing is recorded wall-clock latency, not the robot’s reaction time or simulator timestep. Quantiles use linear interpolation and retain timed errors in the all-attempt row. A small cohort cannot establish production tail latency. Missing measurements remain unreported.

This benchmark uses synthetic simulation fixtures and a hosted decision model. It does not publish trained robot weights or demonstrate hardware transfer. The DVIDIA code license does not grant rights to the DVIDIA marks or to an external model.

Public results · JSONFile checksumsJev image documentation ↗
+''' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("run_dir", type=Path) + parser.add_argument("output_dir", type=Path) + parser.add_argument("--results", type=Path) + args = parser.parse_args() + result = build_report(args.run_dir, args.output_dir, results_path=args.results) + print(json.dumps({"files": len(result["files"]), "output": str(args.output_dir)})) + + +if __name__ == "__main__": + main() diff --git a/tests/test_image_goal_report.py b/tests/test_image_goal_report.py new file mode 100644 index 0000000..410383a --- /dev/null +++ b/tests/test_image_goal_report.py @@ -0,0 +1,181 @@ +import base64 +import json +from pathlib import Path +import tempfile +import unittest + +from dvidia_training.image_goal_report import build_report + + +PNG = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jfZkAAAAASUVORK5CYII=') + + +class ImageGoalReportTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.root = Path(self.temp.name).resolve() + self.run = self.root / 'run' + self.run.mkdir() + (self.run / 'current.png').write_bytes(PNG) + (self.run / 'goal.png').write_bytes(PNG) + self.cases = [ + {'id': 'missing', 'category': 'missing-object', 'current_image': 'current.png', 'goal_image': 'goal.png', 'ground_truth': {'present': False, 'complete': False}}, + {'id': 'held', 'category': 'false-completion', 'current_image': 'current.png', 'goal_image': 'goal.png', 'ground_truth': {'present': True, 'complete': False}}, + {'id': 'finished', 'category': 'complete', 'current_image': 'current.png', 'goal_image': 'goal.png', 'ground_truth': {'present': True, 'complete': True}}, + {'id': 'unfinished', 'category': 'incomplete', 'current_image': 'current.png', 'goal_image': 'goal.png', 'ground_truth': {'present': True, 'complete': False}}, + ] + self.write_cases() + + def tearDown(self): + self.temp.cleanup() + + def write_cases(self): + (self.run / 'cases.json').write_text(json.dumps({'cases': self.cases, 'conditions': ['rgb', 'rgb_telemetry']})) + + def result(self, rows, **extra): + (self.run / 'results.json').write_text(json.dumps({'status': 'complete', 'model': 'jev-1.14.0', 'planned_conditions': ['rgb', 'rgb_telemetry'], 'results': rows, **extra})) + + def build(self): + manifest = build_report(self.run, self.root / 'report') + return json.loads((self.root / 'report/report.json').read_text()), manifest + + def test_missing_provider_is_unmeasured_not_zero_accuracy(self): + data, manifest = self.build() + self.assertEqual(data['status'], 'not_run') + self.assertEqual(len(data['cases']), 4) + for summary in data['summary']: + self.assertEqual(summary['attempted'], 0) + self.assertEqual(summary['not_run'], 4) + self.assertEqual(summary['completion_accuracy']['answered'], 0) + self.assertIsNone(summary['successful_response_latency_ms']['p95']) + self.assertEqual(sum(name.startswith('images/') for name in manifest['files']), 1) + self.assertIn('Jev has not been run.', (self.root / 'report/index.html').read_text()) + + def test_denominators_keep_errors_abstentions_and_not_run_cases(self): + self.result([ + {'case_id': 'missing', 'condition': 'rgb', 'status': 'ok', 'decision': {'present': True, 'complete': True}, 'p_complete': .9, 'timing_ms': {'total_ms': 100}}, + {'case_id': 'held', 'condition': 'rgb', 'status': 'error', 'timing_ms': {'total_ms': 5000}}, + {'case_id': 'finished', 'condition': 'rgb', 'status': 'ok', 'decision': {'present': True, 'complete': None}, 'timing_ms': {'total_ms': 300}}, + ]) + data, _ = self.build() + summary = data['summary'][0] + self.assertEqual(summary['planned'], 4) + self.assertEqual(summary['attempted'], 3) + self.assertEqual(summary['responses'], 2) + self.assertEqual(summary['errors'], 1) + self.assertEqual(summary['not_run'], 1) + self.assertEqual(summary['completion_abstentions'], 1) + self.assertEqual(summary['completion_accuracy'], {'correct': 0, 'answered': 1, 'planned': 4}) + self.assertEqual(summary['false_completion'], {'count': 1, 'answered_incomplete': 1, 'planned_incomplete': 3, 'valid_incomplete': 1}) + self.assertEqual(summary['false_presence'], {'count': 1, 'answered_absent': 1, 'planned_absent': 1, 'valid_absent': 1}) + self.assertEqual(summary['all_attempt_latency_ms']['max'], 5000) + self.assertEqual(summary['successful_response_latency_ms']['max'], 300) + self.assertEqual(summary['successful_response_latency_ms']['p50'], 200) + + def test_probabilities_never_invent_a_decision_and_private_fields_are_omitted(self): + secret = 'private-credential-must-not-be-exported' + self.cases[0]['raw_response'] = secret + self.write_cases() + self.result([{'case_id': 'missing', 'condition': 'rgb', 'status': 'ok', 'p_present': .99, + 'p_complete': .99, 'raw_response': secret, 'headers': {'Authorization': secret}, + 'reason': '', 'timing_ms': {'total_ms': 42, 'debug': secret}}], token=secret) + data, _ = self.build() + self.assertIsNone(data['results'][0]['decision']['complete']) + self.assertEqual(data['summary'][0]['completion_accuracy']['answered'], 0) + for path in (self.root / 'report').rglob('*'): + if path.is_file(): + self.assertNotIn(secret.encode(), path.read_bytes()) + self.assertNotIn(b'alert(1)', path.read_bytes()) + + def test_explicit_provider_vocabulary_and_missing_credentials(self): + self.result([ + {'case_id': 'missing', 'condition': 'rgb', 'status': 'ok', 'presence': 'absent', 'verdict': 'incomplete', 'timing_ms': {'total_ms': 10}}, + {'case_id': 'held', 'condition': 'rgb', 'status': 'missing_credentials'}, + ], status='missing_credentials') + data, _ = self.build() + self.assertEqual(data['results'][0]['decision'], {'present': False, 'complete': False}) + self.assertEqual(data['summary'][0]['attempted'], 1) + self.assertEqual(data['summary'][0]['errors'], 0) + self.assertEqual(data['summary'][0]['not_run'], 3) + self.assertIsNone(data['results'][1]['verdict']) + self.assertEqual(data['results'][1]['decision'], {'present': None, 'complete': None}) + + def test_observability_labels_distinguish_correct_abstention_from_completion(self): + for case in self.cases: + case['physical_truth'] = case.pop('ground_truth') + expected = 'complete' if case['physical_truth']['complete'] else 'incomplete' + case['expected_verdict'] = {'rgb': expected, 'rgb_telemetry': expected} + self.cases[2]['expected_verdict']['rgb'] = 'unknown' + self.write_cases() + self.result([ + {'case_id': 'finished', 'condition': 'rgb', 'status': 'ok', 'presence': 'present', 'verdict': 'unknown'}, + {'case_id': 'missing', 'condition': 'rgb', 'status': 'ok', 'presence': 'absent', 'verdict': 'incomplete'}, + ]) + data, _ = self.build() + summary = data['summary'][0] + self.assertEqual(summary['observability_aware_accuracy'], {'correct': 2, 'planned': 4}) + self.assertEqual(summary['known_verdict_accuracy'], {'correct': 1, 'planned': 3}) + self.assertEqual(summary['ambiguous_abstention'], {'correct': 1, 'planned': 1}) + self.assertEqual(summary['completion_accuracy']['answered'], 1) + + def test_rejects_duplicate_or_unplanned_results_instead_of_double_counting(self): + row = {'case_id': 'missing', 'condition': 'rgb', 'status': 'ok'} + self.result([row, row]) + with self.assertRaisesRegex(ValueError, 'unique planned'): + self.build() + self.assertFalse((self.root / 'report').exists()) + self.result([{**row, 'case_id': 'unplanned'}]) + with self.assertRaisesRegex(ValueError, 'unique planned'): + self.build() + + def test_rejects_escaping_and_symlink_images_before_writing(self): + (self.root / 'private.png').write_bytes(PNG) + self.cases[0]['current_image'] = '../private.png' + self.write_cases() + with self.assertRaisesRegex(ValueError, 'inside'): + self.build() + self.assertFalse((self.root / 'report').exists()) + (self.run / 'linked.png').symlink_to(self.root / 'private.png') + self.cases[0]['current_image'] = 'linked.png' + self.write_cases() + with self.assertRaisesRegex(ValueError, 'plain local'): + self.build() + self.assertFalse((self.root / 'report').exists()) + + def test_refuses_existing_output_and_html_in_label(self): + self.cases[0]['id'] = '' + self.write_cases() + with self.assertRaisesRegex(ValueError, 'public case id'): + self.build() + self.cases[0]['id'] = 'missing' + self.write_cases() + (self.root / 'report').mkdir() + (self.root / 'report/keep.txt').write_text('keep') + with self.assertRaisesRegex(ValueError, 'fresh directory'): + self.build() + self.assertEqual((self.root / 'report/keep.txt').read_text(), 'keep') + + def test_inflight_request_cannot_be_hidden_as_missing(self): + self.result([], status='running', active_trial={'case_id': 'missing', 'condition': 'rgb'}) + with self.assertRaisesRegex(ValueError, 'active or interrupted'): + self.build() + self.result([], status='not_run', active_trial={'case_id': 'missing', 'condition': 'rgb'}) + with self.assertRaisesRegex(ValueError, 'active or interrupted'): + self.build() + + def test_external_results_keep_fixture_directory_immutable_and_count_unknown_cost(self): + external = self.root / 'external-results.json' + external.write_text(json.dumps({'status': 'completed', 'results': [ + {'case_id': 'missing', 'condition': 'rgb', 'status': 'ok', 'presence': 'absent', + 'verdict': 'incomplete', 'estimated_input_cost_usd': .001}, + {'case_id': 'held', 'condition': 'rgb', 'status': 'error'}, + ]})) + build_report(self.run, self.root / 'report', results_path=external) + data = json.loads((self.root / 'report/report.json').read_text()) + self.assertFalse((self.run / 'results.json').exists()) + self.assertEqual(data['summary'][0]['known_usage_estimated_input_cost_usd'], .001) + self.assertEqual(data['summary'][0]['unknown_cost_attempts'], 1) + + +if __name__ == '__main__': + unittest.main() From 3d3c8f660f6f9544d6a58c654bab143dd1eddc00 Mon Sep 17 00:00:00 2001 From: DVIDIA Date: Sun, 18 Oct 2026 23:43:00 -0400 Subject: [PATCH 3/5] Include standalone native evidence verifier in benchmark publication --- docs/image-goal-benchmark.md | 2 ++ publishing/huggingface/publish_image_goals.py | 2 ++ publishing/huggingface/test_publish_image_goals.py | 10 ++++++++++ 3 files changed, 14 insertions(+) diff --git a/docs/image-goal-benchmark.md b/docs/image-goal-benchmark.md index 12c6b20..8a407f0 100644 --- a/docs/image-goal-benchmark.md +++ b/docs/image-goal-benchmark.md @@ -50,6 +50,8 @@ requirement for the simulator physics, but rendering support is required. Measured time on one host is not a minimum hardware specification. ```sh +git clone --branch feat/image-goal-benchmark-v01 https://github.com/Dvidia-Inference/dvidia-training.git +cd dvidia-training python -m pip install '.[arm]' dvidia-robot-sim fetch piper --output robot-assets dvidia-image-goal generate --assets robot-assets --output runs/image-goal-fixtures diff --git a/publishing/huggingface/publish_image_goals.py b/publishing/huggingface/publish_image_goals.py index abd89f1..428af7c 100644 --- a/publishing/huggingface/publish_image_goals.py +++ b/publishing/huggingface/publish_image_goals.py @@ -44,6 +44,8 @@ def inventory(directory): if re.fullmatch(rb"[a-f0-9]{64}\n", raw) is None: raise PublicationError("Expected exactly one newline-terminated SHA-256 digest.") inspect_bytes(name + ".txt", raw) + elif name in {"verify-image-goal-fixtures.py", NAMESPACE + "/verify-image-goal-fixtures.py"}: + inspect_bytes(name + ".txt", raw) else: inspect_bytes(name, raw) if name.endswith(".json"): diff --git a/publishing/huggingface/test_publish_image_goals.py b/publishing/huggingface/test_publish_image_goals.py index af02ded..63c4ca8 100644 --- a/publishing/huggingface/test_publish_image_goals.py +++ b/publishing/huggingface/test_publish_image_goals.py @@ -76,6 +76,16 @@ def test_frozen_checksum_extension_is_narrowly_bounded(self): with self.assertRaisesRegex(publisher.PublicationError, 'SHA-256 digest'): self.stage() + def test_only_named_offline_verifier_may_be_python_source(self): + self.release['verify-image-goal-fixtures.py'] = b'print("offline verification")\n' + self.save() + result = self.stage() + publisher.check(result['plan']) + self.release['unrelated-script.py'] = b'print("outside publication scope")\n' + self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'artifact type'): + publisher.check_export(self.export, 'b'*40) + def test_omitted_unrun_trial_cannot_shrink_denominator(self): result = json.loads(self.release['results.json']); result['results'].pop() self.release['results.json'] = publisher.encoded(result); self.save() From a3e9afb59f3a9cd7da617cf61416e24fad0a0a5c Mon Sep 17 00:00:00 2001 From: DVIDIA Date: Mon, 19 Oct 2026 00:14:01 -0400 Subject: [PATCH 4/5] Publish measured Jev image-goal evaluation with completion recall --- benchmarks/image-goal-20261019/README.md | 84 ++ benchmarks/image-goal-20261019/metrics.json | 993 ++++++++++++++++++ publishing/huggingface/publish_image_goals.py | 39 +- .../huggingface/test_publish_image_goals.py | 82 ++ src/dvidia_training/image_goal_report.py | 13 +- tests/test_image_goal_report.py | 22 + 6 files changed, 1220 insertions(+), 13 deletions(-) create mode 100644 benchmarks/image-goal-20261019/README.md create mode 100644 benchmarks/image-goal-20261019/metrics.json diff --git a/benchmarks/image-goal-20261019/README.md b/benchmarks/image-goal-20261019/README.md new file mode 100644 index 0000000..e4213ec --- /dev/null +++ b/benchmarks/image-goal-20261019/README.md @@ -0,0 +1,84 @@ +# Jev image-goal evaluation: first measured run + +DVIDIA research · 19 October 2026 · measured follow-up to protocol v0.1 + +All **64 planned Jev 1.14.0 requests returned valid responses**, with no retries, +API errors or omitted trials. The 32 native fixtures, prompts, thresholds and +scorer are unchanged from the [published protocol](../image-goal-20261018/README.md). +The original unrun release remains available; this follow-up adds measurements. + +- [Research article](https://research.dvidia.org/papers/image-goal-jev-v02/) +- [Inspect every measured case on Hugging Face](https://dvidia-robot-simulation-lab.static.hf.space/image-goals/v02/index.html) +- [Metrics and provenance](metrics.json) +- [Reproduce the benchmark](../../docs/image-goal-benchmark.md) + +## Results + +| Measure | Images only | Images + native telemetry | +| --- | ---: | ---: | +| Valid responses | 32/32 | 32/32 | +| Observability-aware verdict accuracy | 24/32 (75%) | 25/32 (78.125%) | +| Physical false completion | 0/24 | 0/24 | +| Actual complete states called complete | 0/8 | 1/8 | +| Correct abstention when inputs are ambiguous | 8/16 | No ambiguous reference cases | +| Explicit decisions | 24/32 | 29/32 | +| False presence on absent targets | 0/8 | 0/8 | +| Expected presence decision accuracy | 28/32 | 27/32 | +| Completion-probability Brier score | 0.185772 | 0.117438 | +| Online latency p50 | 1.585 s | 1.914 s | +| Online latency p95 | 3.548 s | 3.221 s | + +Zero false completion does not establish a useful completion detector by itself. +The telemetry condition recognized only one of eight physically completed states; +an always-incomplete baseline scores 24/32 on that condition, versus Jev's 25/32. +No RGB-only complete verdict was justified by the frozen reference: a single +frame cannot establish hidden velocity, contact or sustained support. + +The RGB and telemetry accuracy columns have different expected labels. Their +difference is not a controlled estimate of a general accuracy improvement. +The paired cases provide a more specific result: all four moving controls changed +from a justified RGB abstention to a correct incomplete verdict with telemetry. +All four RGB settled controls also abstained correctly, but three still abstained +with telemetry. The physically complete occluded target was called incomplete +in every layout under both input conditions. Visible unreleased grippers were +called incomplete in both conditions; the RGB reference required unknown because +an image cannot prove the declared contact-force threshold. + +All eight absent-target trials per condition received incomplete verdicts. +Presence probabilities did not always cross the frozen absence threshold: +zero false presence is not equivalent to perfect absence detection. + +## Speed, cost and interpretation + +Across all 64 requests, online latency was **1.776 s p50, 3.287 s p95, +3.950 s p99 and 4.091 s maximum**. Timing includes request construction, upload, +network, hosted response and parsing from preloaded images. It excludes rendering, +camera capture and robot motion. A single serial run cannot establish production +tail latency or a real-time deadline. + +The input-token estimate is **$0.007644756** across the run, using the price frozen +on 18 October and returned usage for all 64 requests. It is not a billing receipt. + +These results support investigating Jev as a supervisory image-goal evaluator. +They do not justify putting this hosted call in a fast grip, contact or actuator +loop. A local verifier can check measured physical constraints; the visual layer +still needs testing for object identity, occlusion, uncertainty and evidence use. +This is a small diagnostic with four correlated layouts, privileged simulator +telemetry and a narrowed 30 g rigid shoe proxy. It demonstrates neither a learned +controller nor real-robot transfer. + +## Reproducibility and credential handling + +The frozen generator, client and scorer are commit +`19acf0734d63ecebc07140c8d84302a54baee4c4`. The evaluation plan and original result +hashes are in `metrics.json`. Publication tooling is versioned separately. +No prompts, cases or thresholds were tuned after observing these responses. + +The credential was supplied through a hidden terminal prompt to process memory. +It was not placed in shell arguments or environment files. An exact-value scan +of 1,677 tracked repository and run files found zero matches. Public exports omit +raw provider responses and pass the publication scanner before upload. + +Original DVIDIA code and authored fixtures: MIT. PiPER retains its upstream +notice. Hosted Jev remains subject to TypeSafe's separate terms; no model weights, +customer footage, training or distillation are part of this release. diff --git a/benchmarks/image-goal-20261019/metrics.json b/benchmarks/image-goal-20261019/metrics.json new file mode 100644 index 0000000..05c42ab --- /dev/null +++ b/benchmarks/image-goal-20261019/metrics.json @@ -0,0 +1,993 @@ +{ + "credential_check": { + "exact_credential_matches": 0, + "files_scanned": 1677 + }, + "date": "2026-10-19", + "environment": { + "platform": "macOS-26.5.2-arm64-arm-64bit", + "python": "3.12.9" + }, + "evaluation_plan_sha256": "5f02006190d97bdaca43f139f5bf221cece23bba868a5e9ee0f424618ea87d8f", + "frozen_core_source": "19acf0734d63ecebc07140c8d84302a54baee4c4", + "limitations": [ + "Authored diagnostic fixtures; four scene layouts are correlated clusters, not 32 independent real-world tasks.", + "Preloaded image bytes + request construction + image upload + hosted response + parsing; excludes fixture file loading, prerecorded rendering, live camera, actuation and physical task duration.", + "Telemetry is privileged simulator state, not validated real sensor readings.", + "Jev is a hosted experimental image evaluator; no robot learning or local Jev inference is demonstrated." + ], + "metrics": { + "by_category": { + "complete": { + "ambiguous_abstention": { + "denominator": 4, + "numerator": 4, + "rate": 1.0 + }, + "completed_requests": 8, + "completion_coverage": { + "denominator": 8, + "numerator": 1, + "rate": 0.125 + }, + "known_usage_estimated_input_cost_usd": 0.000974694, + "known_verdict_accuracy": { + "denominator": 4, + "numerator": 1, + "rate": 0.25 + }, + "missing_object_false_presence": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "observability_aware_accuracy": { + "denominator": 8, + "numerator": 5, + "rate": 0.625 + }, + "online_ms_all_attempts": { + "count": 8, + "max": 3371.058833086863, + "p50": 1879.4144374551252, + "p95": 3066.8494352488774, + "p99": 3310.216953519266 + }, + "online_ms_successful_requests": { + "count": 8, + "max": 3371.058833086863, + "p50": 1879.4144374551252, + "p95": 3066.8494352488774, + "p99": 3310.216953519266 + }, + "physical_completion_brier": { + "count": 8, + "mean": 0.2116 + }, + "physical_false_completion": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "physical_false_completion_valid_requests": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "physical_true_completion": { + "denominator": 8, + "numerator": 1, + "rate": 0.125 + }, + "presence_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "request_ms_successful_requests": { + "count": 8, + "max": 3370.0631661340594, + "p50": 1878.3488750923425, + "p95": 3065.9710914711463, + "p99": 3309.244751201477 + }, + "scheduled": 8, + "selective_physical_accuracy": { + "denominator": 1, + "numerator": 1, + "rate": 1.0 + }, + "statuses": { + "ok": 8 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "complete": 1, + "unknown": 7 + } + }, + "missing_target": { + "ambiguous_abstention": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "completed_requests": 8, + "completion_coverage": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "known_usage_estimated_input_cost_usd": 0.0009149280000000001, + "known_verdict_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "missing_object_false_presence": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "observability_aware_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "online_ms_all_attempts": { + "count": 8, + "max": 3866.8453751597553, + "p50": 1950.4183124518022, + "p95": 3663.860964472405, + "p99": 3826.2484930222854 + }, + "online_ms_successful_requests": { + "count": 8, + "max": 3866.8453751597553, + "p50": 1950.4183124518022, + "p95": 3663.860964472405, + "p99": 3826.2484930222854 + }, + "physical_completion_brier": { + "count": 8, + "mean": 0.000975 + }, + "physical_false_completion": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "presence_accuracy": { + "denominator": 8, + "numerator": 3, + "rate": 0.375 + }, + "request_ms_successful_requests": { + "count": 8, + "max": 3864.315749844536, + "p50": 1949.5684374123812, + "p95": 3661.7615042487155, + "p99": 3823.804900725372 + }, + "scheduled": 8, + "selective_physical_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "statuses": { + "ok": 8 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "incomplete": 8 + } + }, + "moving_alias": { + "ambiguous_abstention": { + "denominator": 4, + "numerator": 4, + "rate": 1.0 + }, + "completed_requests": 8, + "completion_coverage": { + "denominator": 8, + "numerator": 4, + "rate": 0.5 + }, + "known_usage_estimated_input_cost_usd": 0.0009373140000000001, + "known_verdict_accuracy": { + "denominator": 4, + "numerator": 4, + "rate": 1.0 + }, + "missing_object_false_presence": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "observability_aware_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "online_ms_all_attempts": { + "count": 8, + "max": 2718.4088330250233, + "p50": 1152.382583473809, + "p95": 2596.373370871879, + "p99": 2694.0017405943945 + }, + "online_ms_successful_requests": { + "count": 8, + "max": 2718.4088330250233, + "p50": 1152.382583473809, + "p95": 2596.373370871879, + "p99": 2694.0017405943945 + }, + "physical_completion_brier": { + "count": 8, + "mean": 0.1247 + }, + "physical_false_completion": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "presence_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "request_ms_successful_requests": { + "count": 8, + "max": 2717.451874865219, + "p50": 1151.4745620079339, + "p95": 2595.5285727162845, + "p99": 2693.067214435432 + }, + "scheduled": 8, + "selective_physical_accuracy": { + "denominator": 4, + "numerator": 4, + "rate": 1.0 + }, + "statuses": { + "ok": 8 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "incomplete": 4, + "unknown": 4 + } + }, + "occluded": { + "ambiguous_abstention": { + "denominator": 4, + "numerator": 0, + "rate": 0.0 + }, + "completed_requests": 8, + "completion_coverage": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "known_usage_estimated_input_cost_usd": 0.0009746100000000001, + "known_verdict_accuracy": { + "denominator": 4, + "numerator": 0, + "rate": 0.0 + }, + "missing_object_false_presence": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "observability_aware_accuracy": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "online_ms_all_attempts": { + "count": 8, + "max": 4091.2106670439243, + "p50": 2238.5471874149516, + "p95": 3809.8620876087803, + "p99": 4034.9409511568956 + }, + "online_ms_successful_requests": { + "count": 8, + "max": 4091.2106670439243, + "p50": 2238.5471874149516, + "p95": 3809.8620876087803, + "p99": 4034.9409511568956 + }, + "physical_completion_brier": { + "count": 8, + "mean": 0.8569125 + }, + "physical_false_completion": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "physical_false_completion_valid_requests": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "physical_true_completion": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "presence_accuracy": { + "denominator": 8, + "numerator": 4, + "rate": 0.5 + }, + "request_ms_successful_requests": { + "count": 8, + "max": 4090.230582980439, + "p50": 2237.0609584031627, + "p95": 3808.84302270133, + "p99": 4033.9530709246174 + }, + "scheduled": 8, + "selective_physical_accuracy": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "statuses": { + "ok": 8 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "incomplete": 8 + } + }, + "unreleased": { + "ambiguous_abstention": { + "denominator": 4, + "numerator": 0, + "rate": 0.0 + }, + "completed_requests": 8, + "completion_coverage": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "known_usage_estimated_input_cost_usd": 0.000980532, + "known_verdict_accuracy": { + "denominator": 4, + "numerator": 4, + "rate": 1.0 + }, + "missing_object_false_presence": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "observability_aware_accuracy": { + "denominator": 8, + "numerator": 4, + "rate": 0.5 + }, + "online_ms_all_attempts": { + "count": 8, + "max": 3029.5940418727696, + "p50": 1690.7629171619192, + "p95": 2836.7423937306735, + "p99": 2991.0237122443505 + }, + "online_ms_successful_requests": { + "count": 8, + "max": 3029.5940418727696, + "p50": 1690.7629171619192, + "p95": 2836.7423937306735, + "p99": 2991.0237122443505 + }, + "physical_completion_brier": { + "count": 8, + "mean": 0.0019125000000000001 + }, + "physical_false_completion": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "presence_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "request_ms_successful_requests": { + "count": 8, + "max": 3027.2396670188755, + "p50": 1689.9886869359761, + "p95": 2834.6627688501026, + "p99": 2988.724287385121 + }, + "scheduled": 8, + "selective_physical_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "statuses": { + "ok": 8 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "incomplete": 8 + } + }, + "wrong_object": { + "ambiguous_abstention": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "completed_requests": 8, + "completion_coverage": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "known_usage_estimated_input_cost_usd": 0.0009154320000000001, + "known_verdict_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "missing_object_false_presence": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "observability_aware_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "online_ms_all_attempts": { + "count": 8, + "max": 2740.4377080965787, + "p50": 1985.6172084109858, + "p95": 2719.875879061874, + "p99": 2736.325342289638 + }, + "online_ms_successful_requests": { + "count": 8, + "max": 2740.4377080965787, + "p50": 1985.6172084109858, + "p95": 2719.875879061874, + "p99": 2736.325342289638 + }, + "physical_completion_brier": { + "count": 8, + "mean": 0.0001 + }, + "physical_false_completion": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "presence_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "request_ms_successful_requests": { + "count": 8, + "max": 2739.7601660341024, + "p50": 1984.5179584808648, + "p95": 2718.8356057275087, + "p99": 2735.5752539727837 + }, + "scheduled": 8, + "selective_physical_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "statuses": { + "ok": 8 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "incomplete": 8 + } + }, + "wrong_orientation": { + "ambiguous_abstention": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "completed_requests": 8, + "completion_coverage": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "known_usage_estimated_input_cost_usd": 0.000973434, + "known_verdict_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "missing_object_false_presence": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "observability_aware_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "online_ms_all_attempts": { + "count": 8, + "max": 2278.634167043492, + "p50": 1260.348187526688, + "p95": 2080.885917076375, + "p99": 2239.084517050069 + }, + "online_ms_successful_requests": { + "count": 8, + "max": 2278.634167043492, + "p50": 1260.348187526688, + "p95": 2080.885917076375, + "p99": 2239.084517050069 + }, + "physical_completion_brier": { + "count": 8, + "mean": 0.006187500000000001 + }, + "physical_false_completion": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "presence_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "request_ms_successful_requests": { + "count": 8, + "max": 2278.104541124776, + "p50": 1259.3578124651685, + "p95": 2080.4035851964722, + "p99": 2238.5643499391153 + }, + "scheduled": 8, + "selective_physical_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "statuses": { + "ok": 8 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "incomplete": 8 + } + }, + "wrong_position": { + "ambiguous_abstention": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "completed_requests": 8, + "completion_coverage": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "known_usage_estimated_input_cost_usd": 0.0009738120000000001, + "known_verdict_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "missing_object_false_presence": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "observability_aware_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "online_ms_all_attempts": { + "count": 8, + "max": 3182.4908750131726, + "p50": 2037.1751248603687, + "p95": 2915.7822856912385, + "p99": 3129.149157148786 + }, + "online_ms_successful_requests": { + "count": 8, + "max": 3182.4908750131726, + "p50": 2037.1751248603687, + "p95": 2915.7822856912385, + "p99": 3129.149157148786 + }, + "physical_completion_brier": { + "count": 8, + "mean": 0.010450000000000001 + }, + "physical_false_completion": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "presence_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "request_ms_successful_requests": { + "count": 8, + "max": 3181.8517500068992, + "p50": 2036.0838960623369, + "p95": 2915.057687507942, + "p99": 3128.492937507108 + }, + "scheduled": 8, + "selective_physical_accuracy": { + "denominator": 8, + "numerator": 8, + "rate": 1.0 + }, + "statuses": { + "ok": 8 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "incomplete": 8 + } + } + }, + "by_condition": { + "rgb": { + "ambiguous_abstention": { + "denominator": 16, + "numerator": 8, + "rate": 0.5 + }, + "completed_requests": 32, + "completion_coverage": { + "denominator": 32, + "numerator": 24, + "rate": 0.75 + }, + "known_usage_estimated_input_cost_usd": 0.0034789440000000003, + "known_verdict_accuracy": { + "denominator": 16, + "numerator": 16, + "rate": 1.0 + }, + "missing_object_false_presence": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "observability_aware_accuracy": { + "denominator": 32, + "numerator": 24, + "rate": 0.75 + }, + "online_ms_all_attempts": { + "count": 32, + "max": 4091.2106670439243, + "p50": 1585.2316045202315, + "p95": 3548.127089440822, + "p99": 4021.6574265598324 + }, + "online_ms_successful_requests": { + "count": 32, + "max": 4091.2106670439243, + "p50": 1585.2316045202315, + "p95": 3548.127089440822, + "p99": 4021.6574265598324 + }, + "physical_completion_brier": { + "count": 32, + "mean": 0.185771875 + }, + "physical_false_completion": { + "denominator": 24, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 24, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "presence_accuracy": { + "denominator": 32, + "numerator": 28, + "rate": 0.875 + }, + "request_ms_successful_requests": { + "count": 32, + "max": 4090.230582980439, + "p50": 1584.1166249010712, + "p95": 3546.3884562021117, + "p99": 4020.1969847083096 + }, + "scheduled": 32, + "selective_physical_accuracy": { + "denominator": 24, + "numerator": 20, + "rate": 0.8333333333333334 + }, + "statuses": { + "ok": 32 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "incomplete": 24, + "unknown": 8 + } + }, + "rgb_telemetry": { + "ambiguous_abstention": { + "denominator": 0, + "numerator": 0, + "rate": null + }, + "completed_requests": 32, + "completion_coverage": { + "denominator": 32, + "numerator": 29, + "rate": 0.90625 + }, + "known_usage_estimated_input_cost_usd": 0.0041658120000000005, + "known_verdict_accuracy": { + "denominator": 32, + "numerator": 25, + "rate": 0.78125 + }, + "missing_object_false_presence": { + "denominator": 8, + "numerator": 0, + "rate": 0.0 + }, + "observability_aware_accuracy": { + "denominator": 32, + "numerator": 25, + "rate": 0.78125 + }, + "online_ms_all_attempts": { + "count": 32, + "max": 3371.058833086863, + "p50": 1913.8612295500934, + "p95": 3220.66000639461, + "p99": 3338.8970565912314 + }, + "online_ms_successful_requests": { + "count": 32, + "max": 3371.058833086863, + "p50": 1913.8612295500934, + "p95": 3220.66000639461, + "p99": 3338.8970565912314 + }, + "physical_completion_brier": { + "count": 32, + "mean": 0.1174375 + }, + "physical_false_completion": { + "denominator": 24, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 24, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 8, + "numerator": 1, + "rate": 0.125 + }, + "presence_accuracy": { + "denominator": 32, + "numerator": 27, + "rate": 0.84375 + }, + "request_ms_successful_requests": { + "count": 32, + "max": 3370.0631661340594, + "p50": 1912.7987290266901, + "p95": 3219.8883001110516, + "p99": 3337.920583873056 + }, + "scheduled": 32, + "selective_physical_accuracy": { + "denominator": 29, + "numerator": 25, + "rate": 0.8620689655172413 + }, + "statuses": { + "ok": 32 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "complete": 1, + "incomplete": 28, + "unknown": 3 + } + } + }, + "overall": { + "ambiguous_abstention": { + "denominator": 16, + "numerator": 8, + "rate": 0.5 + }, + "completed_requests": 64, + "completion_coverage": { + "denominator": 64, + "numerator": 53, + "rate": 0.828125 + }, + "known_usage_estimated_input_cost_usd": 0.007644756000000001, + "known_verdict_accuracy": { + "denominator": 48, + "numerator": 41, + "rate": 0.8541666666666666 + }, + "missing_object_false_presence": { + "denominator": 16, + "numerator": 0, + "rate": 0.0 + }, + "observability_aware_accuracy": { + "denominator": 64, + "numerator": 49, + "rate": 0.765625 + }, + "online_ms_all_attempts": { + "count": 64, + "max": 4091.2106670439243, + "p50": 1776.4145623659715, + "p95": 3287.287432909943, + "p99": 3949.8605331568974 + }, + "online_ms_successful_requests": { + "count": 64, + "max": 4091.2106670439243, + "p50": 1776.4145623659715, + "p95": 3287.287432909943, + "p99": 3949.8605331568974 + }, + "physical_completion_brier": { + "count": 64, + "mean": 0.1516046875 + }, + "physical_false_completion": { + "denominator": 48, + "numerator": 0, + "rate": 0.0 + }, + "physical_false_completion_valid_requests": { + "denominator": 48, + "numerator": 0, + "rate": 0.0 + }, + "physical_true_completion": { + "denominator": 16, + "numerator": 1, + "rate": 0.0625 + }, + "presence_accuracy": { + "denominator": 64, + "numerator": 55, + "rate": 0.859375 + }, + "request_ms_successful_requests": { + "count": 64, + "max": 4090.230582980439, + "p50": 1775.0753960572183, + "p95": 3286.1646063625813, + "p99": 3947.90423810482 + }, + "scheduled": 64, + "selective_physical_accuracy": { + "denominator": 53, + "numerator": 45, + "rate": 0.8490566037735849 + }, + "statuses": { + "ok": 64 + }, + "unknown_cost_attempts": 0, + "verdicts": { + "complete": 1, + "incomplete": 52, + "unknown": 11 + } + } + }, + "model": "jev-1.14.0", + "original_results_sha256": "18c71ad02cd50763b04a90028b46740b98bd91ffa9d6071f5a179d4c22537ae4", + "provider_requests": 64, + "schema": "dvidia.image-goal-measured-summary/v1" +} diff --git a/publishing/huggingface/publish_image_goals.py b/publishing/huggingface/publish_image_goals.py index 428af7c..4cab903 100644 --- a/publishing/huggingface/publish_image_goals.py +++ b/publishing/huggingface/publish_image_goals.py @@ -20,6 +20,14 @@ REPO_ID = "Dvidia/robot-simulation-lab" NAMESPACE = "image-goals/v01" +RELEASES = { + "image-goals/v01": {"id": "image-goals-title", "before": "graspability-title", + "heading": "Does it only look finished?", "readme_heading": "Image-goal benchmark v0.1", + "provenance": "image_goals_v01", "paper": "image-goal-jev-v01"}, + "image-goals/v02": {"id": "image-goals-results-title", "before": "image-goals-title", + "heading": "How does Jev judge completion?", "readme_heading": "Image-goal measured follow-up (publication v0.2)", + "provenance": "image_goals_v02", "paper": "image-goal-jev-v02"}, +} ROOT_FILES = {"index.html", "README.md", "provenance.json"} SHA = re.compile(r"[a-f0-9]{40}\Z") MAX_FILE = 4 * 1024 * 1024 @@ -124,7 +132,8 @@ def check_export(export, source_commit): raise PublicationError("Unrun provider trials cannot carry predictions or service measurements.") if b"Jev has not been run" not in payload["index.html"]: raise PublicationError("Unrun provider status must be visible in the report.") - elif result.get("status") != "completed" or pending: + elif (result.get("status") != "completed" + or any(row.get("status") not in {"ok", "error"} for row in result["results"])): raise PublicationError("Unexpected provider run status.") viewer_seal = json.loads(payload["report-integrity.json"]) for name, expected_file in viewer_seal.get("files", {}).items(): @@ -133,6 +142,11 @@ def check_export(export, source_commit): source_root = str(Path(__file__).resolve().parents[2] / "src") if source_root not in sys.path: sys.path.insert(0, source_root) + if result.get("status") == "completed": + from dvidia_training.image_goal_benchmark import score + expected_metrics = score(result["results"], cases) + if result.get("metrics") != expected_metrics: + raise PublicationError("Provider metrics differ from independently recomputed frozen scores.") from dvidia_training.image_goal_report import build_report with tempfile.TemporaryDirectory(prefix="dvidia-image-goal-publication-") as temporary: expected_viewer = Path(temporary).resolve() / "viewer" @@ -174,28 +188,31 @@ def snapshot(output, parent_commit): def updated_roots(parent, source_commit, export): + release = RELEASES[NAMESPACE] status = json.loads(export["results.json"])["status"] pending = status == "not_run" + if NAMESPACE == "image-goals/v02" and pending: + raise PublicationError("The measured follow-up requires a completed provider run.") count = len(json.loads(export["cases.json"])["cases"]) description = (f"Explore {count} frozen PiPER image-goal cases, including absent targets, wrong arrangements and visually identical moving scenes. " + ("Native fixture validation is published. Jev accuracy and service latency are not yet measured." if pending else "Inspect all paired Jev judgments, failures, abstentions and measured service latency.") + " Synthetic diagnostic fixtures; passive evaluation; no robot controller or hardware qualification.") index = parent["index.html"].decode() - marker = '
' - if index.count(marker) != 1 or 'id="image-goals-title"' in index: - raise PublicationError("Expected one existing graspability section and no image-goal section.") - card = ('

Does it only look finished?

' + marker = f'
' + if index.count(marker) != 1 or f'id="{release["id"]}"' in index: + raise PublicationError("Expected one prior release section and no duplicate release section.") + card = (f'

{release["heading"]}

' f'

{description}

Inspect image goals and physical references →
') index = index.replace(marker, card + marker) - readme = parent["README.md"] + ("\n\n## Image-goal benchmark v0.1\n\n" + readme = parent["README.md"] + (f'\n\n## {release["readme_heading"]}\n\n' f"[Inspect the image-goal benchmark]({NAMESPACE}/index.html). {description}\n\n" f"[Pinned source](https://github.com/Dvidia-Inference/dvidia-training/tree/{source_commit}) · " - "[Research](https://research.dvidia.org/papers/image-goal-jev-v01/)\n").encode() + f'[Research](https://research.dvidia.org/papers/{release["paper"]}/)\n').encode() provenance = json.loads(parent["provenance.json"]) - if "image_goals_v01" in provenance: + if release["provenance"] in provenance: raise PublicationError("Image-goal root provenance already exists.") - provenance["image_goals_v01"] = {"path": NAMESPACE, "source_commit": source_commit, + provenance[release["provenance"]] = {"path": NAMESPACE, "source_commit": source_commit, "integrity": record(export["publication-integrity.json"]), "provider_status": status, "mode": "recorded-passive-evaluation", "learned_policy": False, "physical_robot_ready": False} return {"index.html": index.encode(), "README.md": readme, "provenance.json": encoded(provenance)} @@ -317,7 +334,10 @@ def publish(path, gate_path): def main(): + global NAMESPACE parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--namespace", choices=tuple(RELEASES), default=NAMESPACE, + help="Append-only release namespace; the original protocol remains at v01.") sub = parser.add_subparsers(dest="action", required=True) snap = sub.add_parser("snapshot"); snap.add_argument("output"); snap.add_argument("--parent-commit", required=True) stage_parser = sub.add_parser("stage") @@ -327,6 +347,7 @@ def main(): check_parser = sub.add_parser("check"); check_parser.add_argument("plan") publish_parser = sub.add_parser("publish"); publish_parser.add_argument("plan"); publish_parser.add_argument("gate") args = parser.parse_args() + NAMESPACE = args.namespace try: if args.action == "snapshot": result = snapshot(args.output, args.parent_commit) diff --git a/publishing/huggingface/test_publish_image_goals.py b/publishing/huggingface/test_publish_image_goals.py index 63c4ca8..abb5f95 100644 --- a/publishing/huggingface/test_publish_image_goals.py +++ b/publishing/huggingface/test_publish_image_goals.py @@ -5,10 +5,12 @@ import sys import tempfile import unittest +from unittest.mock import patch import publish_image_goals as publisher sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'src')) from dvidia_training.image_goal_report import build_report +from dvidia_training.image_goal_benchmark import score PNG = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jfZkAAAAASUVORK5CYII=') @@ -60,6 +62,86 @@ def save(self): def stage(self): return publisher.stage(self.export, self.parent, self.root / 'stage', 'b'*40) + def measured_followup(self): + result = json.loads(self.release['results.json']) + result['status'] = 'completed' + for row in result['results']: + row.update(status='ok', attempts=1, presence='absent', verdict='incomplete', + p_present=0.02, p_complete=0.01, online_total_ms=200, + latency_ms={'prepare': 1, 'http': 199, 'total': 200}) + result['metrics'] = score(result['results'], json.loads(self.release['cases.json'])['cases']) + self.release['results.json'] = publisher.encoded(result) + provenance = json.loads(self.release['provenance.json']) + provenance['provider_status'] = 'completed' + self.release['provenance.json'] = publisher.encoded(provenance) + self.save() + viewer = self.root / 'measured-viewer' + build_report(self.export, viewer) + for path in viewer.rglob('*'): + if path.is_file(): + self.release[path.relative_to(viewer).as_posix()] = path.read_bytes() + self.save() + + def prior_protocol_snapshot(self): + self.original['index.html'] = self.original['index.html'].replace( + b'
', + b'

Original unrun protocol

') + self.original['provenance.json'] = b'{"previous":{"keep":true},"image_goals_v01":{"keep":"original protocol"}}\n' + for name, raw in self.original.items(): + (self.parent / name).write_bytes(raw) + self.tree[name] = {'blob': publisher.git_blob(raw), 'size': len(raw), 'lfs_sha256': None} + self.tree['image-goals/v01/index.html'] = {'blob': 'e'*40, 'size': 100, 'lfs_sha256': None} + (self.parent / 'snapshot.json').write_bytes(publisher.encoded({'repo_id': publisher.REPO_ID, + 'namespace': publisher.NAMESPACE, 'parent_commit': 'a'*40, 'tree': self.tree})) + + def test_measured_followup_appends_v02_and_preserves_v01_and_old_roots(self): + self.measured_followup() + with patch.object(publisher, 'NAMESPACE', 'image-goals/v02'): + self.prior_protocol_snapshot() + result = self.stage() + plan, payload = publisher.check(result['plan']) + self.assertEqual(plan['namespace'], 'image-goals/v02') + self.assertEqual(plan['parent_tree'], self.tree) + self.assertFalse(any(name.startswith('image-goals/v01/') for name in payload)) + self.assertEqual(set(payload) & self.tree.keys(), publisher.ROOT_FILES) + self.assertIn(b'Original unrun protocol', payload['index.html']) + self.assertIn(b'How does Jev judge completion?', payload['index.html']) + self.assertIn(self.original['README.md'], payload['README.md']) + provenance = json.loads(payload['provenance.json']) + self.assertEqual(provenance['image_goals_v01'], {'keep': 'original protocol'}) + self.assertEqual(provenance['image_goals_v02']['provider_status'], 'completed') + + def test_measured_followup_rejects_unrun_provider_and_duplicate_card(self): + with patch.object(publisher, 'NAMESPACE', 'image-goals/v02'): + self.prior_protocol_snapshot() + with self.assertRaisesRegex(publisher.PublicationError, 'completed provider run'): + self.stage() + self.measured_followup() + parent = dict(self.original) + parent['index.html'] += b'

Already published

' + with self.assertRaisesRegex(publisher.PublicationError, 'duplicate release'): + publisher.updated_roots(parent, 'b'*40, self.release) + + def test_completed_provider_scores_cannot_be_fabricated_in_source_results(self): + self.measured_followup() + result = json.loads(self.release['results.json']) + result['metrics']['overall']['physical_false_completion']['numerator'] = 999 + self.release['results.json'] = publisher.encoded(result) + self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'independently recomputed frozen scores'): + self.stage() + + def test_completed_provider_cannot_hide_an_unrun_trial(self): + self.measured_followup() + result = json.loads(self.release['results.json']) + result['results'][0] = {'case_id': 'missing', 'condition': result['results'][0]['condition'], + 'status': 'not_run', 'attempts': 0} + result['metrics'] = None + self.release['results.json'] = publisher.encoded(result) + self.save() + with self.assertRaisesRegex(publisher.PublicationError, 'provider run status'): + self.stage() + def test_new_namespace_preserves_history_and_visible_pending_status(self): result = self.stage() plan, payload = publisher.check(result['plan']) diff --git a/src/dvidia_training/image_goal_report.py b/src/dvidia_training/image_goal_report.py index 3199c80..e5146a7 100644 --- a/src/dvidia_training/image_goal_report.py +++ b/src/dvidia_training/image_goal_report.py @@ -98,10 +98,12 @@ def _summary(cases, rows, conditions): answered = [row for row in selected if row["status"] in _SUCCESS] complete_answers = [row for row in answered if row["decision"]["complete"] is not None and truth[row["case_id"]]["complete"] is not None] incomplete = [case for case in cases if case["ground_truth"]["complete"] is False] + complete = [case for case in cases if case["ground_truth"]["complete"] is True] absent = [case for case in cases if case["ground_truth"]["present"] is False] incomplete_answers = [row for row in complete_answers if truth[row["case_id"]]["complete"] is False] absent_answers = [row for row in answered if truth[row["case_id"]]["present"] is False and row["decision"]["present"] is not None] false_complete = sum(row["decision"]["complete"] is True for row in incomplete_answers) + positive_answers = [row for row in complete_answers if truth[row["case_id"]]["complete"] is True] false_present = sum(row["decision"]["present"] is True for row in absent_answers) known = [case for case in cases if case["expected_verdict"].get(condition) in {"complete", "incomplete"}] ambiguous = [case for case in cases if case["expected_verdict"].get(condition) == "unknown"] @@ -123,6 +125,9 @@ def _summary(cases, rows, conditions): "ambiguous_abstention": {"correct": justified_abstentions, "planned": len(ambiguous)}, "false_completion": {"count": false_complete, "answered_incomplete": len(incomplete_answers), "planned_incomplete": len(incomplete), "valid_incomplete": sum(truth[row["case_id"]]["complete"] is False for row in answered)}, + "true_completion": {"count": sum(row["decision"]["complete"] is True for row in positive_answers), + "planned_complete": len(complete), "answered_complete": len(positive_answers), + "valid_complete": sum(truth[row["case_id"]]["complete"] is True for row in answered)}, "false_presence": {"count": false_present, "answered_absent": len(absent_answers), "planned_absent": len(absent), "valid_absent": sum(truth[row["case_id"]]["present"] is False for row in answered)}, "known_usage_estimated_input_cost_usd": sum(row["estimated_input_cost_usd"] for row in attempted if row["estimated_input_cost_usd"] is not None) @@ -233,7 +238,7 @@ def build_report(run_dir, output_dir, *, results_path=None): :root{font-family:system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;color:#26302d;background:#faf9f6;font-synthesis:none}*{box-sizing:border-box}body{margin:0;background-image:radial-gradient(#28332c09 .55px,transparent .65px);background-size:5px 5px}a{color:inherit;text-underline-offset:4px}button,select{font:inherit}button,select,a{touch-action:manipulation}button,select{min-height:44px}button:focus-visible,select:focus-visible,a:focus-visible{outline:3px solid #3c6cff;outline-offset:3px}.skip{position:absolute;left:20px;top:-60px;padding:12px;background:white;z-index:10}.skip:focus{top:8px}.masthead{min-height:84px;padding:20px 5vw;border-bottom:1px solid #dedfd6;display:flex;align-items:center;justify-content:space-between;gap:20px}.brand{display:flex;gap:11px;align-items:center;text-decoration:none;font-weight:750;font-size:20px;letter-spacing:2px}.brand svg{width:34px;height:34px}.top-links{font-size:13px;display:flex;gap:22px}main{max-width:1400px;margin:auto;padding:46px 5vw 36px}.eyebrow{font-size:11px;letter-spacing:2px;text-transform:uppercase;color:#69726b;font-weight:650}h1,h2,h3{font-family:Georgia,"Times New Roman",serif;font-weight:400}h1{font-size:clamp(38px,5vw,64px);letter-spacing:-2px;margin:15px 0 18px;line-height:1.05}.intro{max-width:760px;color:#59665f;line-height:1.8;font-size:16px}.run-state{padding:16px 20px;border:1px solid #d7ded2;background:#f1f3ec;margin:28px 0;font-size:14px;line-height:1.6}.toolbar{display:flex;justify-content:space-between;align-items:end;gap:22px;margin:28px 0 20px}.toolbar label{display:block;font-size:12px;color:#5b6b60;margin-bottom:7px}select{border:1px solid #cfd6cb;border-radius:5px;background:#fffefb;color:inherit;padding:10px 34px 10px 12px;max-width:100%}.meta{font-size:12px;color:#637166;line-height:1.6}.metrics{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));border-block:1px solid #dcded5;margin:18px 0 30px}.metric{padding:19px 20px 21px 0}.metric+.metric{padding-left:20px;border-left:1px solid #dcded5}.metric dt{font-size:12px;color:#627064;line-height:1.4}.metric dd{font:29px Georgia,serif;margin:10px 0 7px}.metric small{font-size:11px;color:#637166;line-height:1.5;display:block}.workspace{display:grid;grid-template-columns:230px minmax(0,1fr);gap:28px}.case-nav{border-top:1px solid #dcded5}.case-nav h2{font:12px system-ui;letter-spacing:1px;text-transform:uppercase;margin:16px 0}.filter{width:100%;font-size:12px;margin-bottom:12px}.case-list{display:grid;gap:4px;max-height:690px;overflow:auto;padding:2px}.case-list button{text-align:left;border:0;background:transparent;padding:11px 10px;color:#516057;border-radius:4px;cursor:pointer;font-size:12px;line-height:1.5}.case-list button[aria-pressed=true]{background:#e9eee4;color:#263a2a;box-shadow:inset 3px 0 #5f775d}.case-list small{display:block;color:#6d786d;margin-top:3px}.case-list .error{color:#924733}.case-heading{display:flex;justify-content:space-between;align-items:baseline;gap:15px}.case-heading h2{margin:0 0 16px;font-size:25px}.image-pair{display:grid;grid-template-columns:1fr 1fr;gap:12px}figure{margin:0;min-width:0}figure img{width:100%;height:auto;aspect-ratio:4/3;object-fit:contain;display:block;background:#e9e9df;border:1px solid #ddded4}figcaption{font-size:12px;color:#5c6a5e;margin:10px 0 16px}.comparison{display:grid;grid-template-columns:1fr 1fr;gap:16px;border-top:1px solid #dbdfd3;padding-top:19px;margin-top:6px}.comparison h3{font-size:20px;margin:0 0 14px}.facts{display:grid;grid-template-columns:minmax(0,1fr) auto;gap:11px;font-size:12px;margin:0}.facts dt{color:#617064}.facts dd{margin:0;text-align:right;font-variant-numeric:tabular-nums}.note{font-size:12px;color:#657264;line-height:1.75}.notice{background:#f2f2e9;padding:13px 15px;border-left:3px solid #a8b398;margin:18px 0;font-size:13px;line-height:1.6}.fine{margin-top:34px;padding-top:24px;border-top:1px solid #dcded5;max-width:950px}.fine h2{font-size:25px}.fine p{font-size:13px;line-height:1.8;color:#5b695e}.table-wrap{overflow:auto}table{border-collapse:collapse;width:100%;font-size:12px;white-space:nowrap}th,td{text-align:left;padding:10px 14px 10px 0;border-bottom:1px solid #e0e2d9}th{font-weight:550;color:#637165}.downloads{display:flex;flex-wrap:wrap;gap:14px;font-size:13px;margin-top:22px}footer{border-top:1px solid #dcdfd5;margin-top:32px;padding-top:23px;font-size:12px;color:#657264;display:flex;justify-content:space-between;gap:22px;line-height:1.8}.empty{padding:25px;color:#657264;font-size:13px}@media(max-width:900px){.workspace{grid-template-columns:190px minmax(0,1fr);gap:18px}.image-pair{grid-template-columns:1fr}.metrics{grid-template-columns:repeat(2,minmax(0,1fr))}.metric:nth-child(3){border-left:0;padding-left:0}.metric:nth-child(n+3){border-top:1px solid #dcded5}}@media(max-width:620px){main{padding:30px 18px}.masthead{padding:16px 18px;min-height:74px}.top-links a:first-child{display:none}.toolbar{display:block}.toolbar .meta{margin-top:14px}.workspace{grid-template-columns:1fr}.case-list{grid-template-columns:repeat(2,minmax(0,1fr));max-height:195px}.case-nav{margin-bottom:14px}.comparison{grid-template-columns:1fr}.comparison>section+section{border-top:1px solid #dcded5;padding-top:17px}.case-heading{display:block}.metric dd{font-size:26px}footer{display:block}footer p{margin:0 0 12px}.intro{font-size:15px}h1{letter-spacing:-1px}}
DVIDIA

Robotics research · Image goals / 0.1

Does it only look finished?

A goal image describes the destination. This experiment asks Jev to judge the current scene, then compares its answer with the simulator’s physical state. Missing shoes and misleading arrangements stay in the results.

Loading the recorded experiment.

-

Current simulated scene
Current observation
Requested goal arrangement
Goal reference

Physical reference

Recorded simulator labels. A single image may not reveal hidden contact or sustained support.

Jev judgment

A remote research judgment. This page sends no images, runs no model and commands no robot.

+

Current simulated scene
Current observation
Requested goal arrangement
Goal reference

Physical reference

Recorded simulator labels. A single image may not reveal hidden contact or sustained support.

Jev judgment

A remote research judgment. This page sends no images, runs no model and commands no robot.

Count the misses. Keep the clock honest.

Judgment accuracy uses the frozen, observability-aware verdict for every planned case: complete, incomplete, or unknown. A correct abstention matters when an image cannot establish hidden motion, contact or dwell. Errors and cases not run remain in the planned denominator; no score is claimed before requests are attempted. Physical false-completion and missing-target counts retain every planned negative case, with answered coverage shown separately.

Decision checkCorrectDenominator
Latency scopeTimed requestsp50p95p99Maximum

Timing is recorded wall-clock latency, not the robot’s reaction time or simulator timestep. Quantiles use linear interpolation and retain timed errors in the all-attempt row. A small cohort cannot establish production tail latency. Missing measurements remain unreported.

This benchmark uses synthetic simulation fixtures and a hosted decision model. It does not publish trained robot weights or demonstrate hardware transfer. The DVIDIA code license does not grant rights to the DVIDIA marks or to an external model.

Public results · JSONFile checksumsJev image documentation ↗