diff --git a/README.md b/README.md index 659ea94..67270f5 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,14 @@ All four covers are **AI-generated editorial illustrations**, not footage, measu ## Latest studies +[From a conversation to compatible robot skills](reports/Chat%20to%20compatible%20robot%20skills.md) +connects a photo and desired outcome to scene review, compatible skill retrieval, +verified installation and inspectable simulation through a proposed chat workflow. +Real local object detection and one native LIBERO reset are verified; OpenVLA +weights remain incomplete, and no learned or steered robot actions have run. +[Inspect the runtime receipt](topics/physical-grounding/runs/2026-10-27-robot-lab/README.md) +and [Hugging Face concept card](https://huggingface.co/spaces/Dvidia/robot-simulation-lab/blob/main/docs/robot-chat-v01/concept.md). + [Live PiPER shoe workstation v0.1](reports/Live%20PiPER%20shoe%20workstation%20v01.md) lets readers drag a tool target, adjust the gripper and nudge a free rigid shoe diff --git a/docs/downloads/reports/Chat to compatible robot skills.md b/docs/downloads/reports/Chat to compatible robot skills.md new file mode 100644 index 0000000..a8a8556 --- /dev/null +++ b/docs/downloads/reports/Chat to compatible robot skills.md @@ -0,0 +1,233 @@ +# Chat to compatible robot skills + +DVIDIA research · 27 October 2026 · Product architecture and local runtime note + +Original writing: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Referenced models, code and assets retain their own terms. + +The intended interface is a conversation: attach a photo, describe a desired +outcome, and let DVIDIA find an executable skill for the available robot. The +conversation should carry the workflow from observation through installation, +simulation and review. A user should not need to assemble those tools manually. + +The current prototype connects photo analysis and conservative skill retrieval +to a separate local worker for ten LIBERO-10 task descriptions sharing one +revision-pinned OpenVLA checkpoint. **Real local object detection and one native +simulator reset are verified. OpenVLA model inference, learned robot actions and +activation-steered robot actions have not been executed.** The checkpoint +transfer stopped after an 8 MiB partial download; this is not an installation or +a successful task. The [runtime receipt](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/runtime-receipt.json) +and [evidence notes](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/README.md) +preserve that boundary. A companion [Hugging Face concept card](https://huggingface.co/spaces/Dvidia/robot-simulation-lab/blob/main/docs/robot-chat-v01/concept.md) +tracks the same documentation scope; it does not host OpenVLA inference. + +## Implementation follow-up — 28 October 2026 + +The local prototype now implements the photo/outcome composer and conversation +cards. A unique compatible unblocked task starts verified installation and real +scene preparation; the user sees the simulator camera before choosing Run in +simulation. Connection settings, object corrections and action/neuron inspection +are expandable. This is a typed tool workflow with conservative lexical +retrieval; the conversation does not add a general language planner. + +Nineteen focused chat/client/planner tests and 42 worker tests pass. New +regressions reject reversed spatial destinations and wrong target colors. +Installation cancellation races and explicit stop-tracking recovery are handled. +Browser control initially timed out, then recovered in a fresh tab. Development +and built desktop previews verified real photo detection, an unsupported +dishwashing reply, actual task-5 camera capture, unique compatible retrieval and +live installation progress. Browser error logs were empty. Phone-sized layout +acceptance remains pending. The repaired downloader has resumed the real +checkpoint transfer, but no completed installation, learned action or steering +result is established. The 26 October receipt below remains unchanged. + +![Local conversational prototype showing actual simulator input and an incomplete policy download](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/chat-interface.png) + +The screenshot shows installation progress, not robot execution. + +## One conversation carries the task + +A useful request is “Put the book in the back compartment of the caddy,” with a +photo attached. DVIDIA needs to identify the intended object and destination, +look for a compatible behavior, check its prerequisites and expose any missing +evidence. The product should present these as progress in one conversation: + +| Conversation event | What it represents | +| --- | --- | +| Photo and desired outcome | User input, including explicit corrections | +| “I found a book; please identify its destination” | A grounded question when the evidence is insufficient | +| Skill card with prerequisites and compatibility | A candidate, with reasons it can or cannot run | +| Download progress and verified installation | Actual artifact state, including interruption and retry | +| Simulator preview and run progress | A real task reset followed by actual policy actions when available | +| Outcome and inspectable trace | Success, failure or cancellation with actions and observations | +| “Try this intervention” | An experimental rerun with the same initial simulator state | + +This is the conversational simplification implemented in the local prototype. +Detailed connection settings, manifests, action vectors and channel controls can +live in expandable cards. The conversation remains an interface to typed tools; +assistant prose cannot mark an artifact installed or a task completed. + +“Tidy this room” expresses a valid desired outcome but leaves important +decisions unresolved: where toys belong, which clothes are dirty, whether a +container is reachable and whether the robot has the required behaviors. A photo +does not answer all of these questions. The first version should ask for missing +destinations or state evidence and report unsupported tasks. General household +task decomposition and automatic selection of multi-skill sequences are future +work. No dishwashing, laundry or toy-sorting competence is established by this +prototype. + +## Retrieve by intended change, then check compatibility + +The architecture connects five distinct decisions: + +```text +Photo + desired outcome + -> observations with source and uncertainty + -> candidate skills with prerequisites and effects + -> compatible artifact + verified installation + -> actual simulator state + policy execution + -> measured outcome + inspectable trace +``` + +An object name is a useful search clue, but the intended change determines +whether a skill is relevant. A mug-placement task does not satisfy a request to +wash a mug. A drawer task that closes the drawer adds an unwanted effect if the +user asked for it to remain open. The current browser planner uses bounded +lexical matching and aliases; it checks object identities, destinations, +relations, state changes and negation, then rejects incompatible profiles. It is +not a learned affordance estimator or a general language planner. + +This separation is motivated by [SayCan](https://say-can.github.io/), which +combines language-based skill relevance with state-dependent affordances. Our +current rule-based filter does not reproduce SayCan's learned value functions +or its reported performance. The design implication is to keep goal relevance, +observed prerequisites and runtime compatibility independently inspectable. + +Every executable candidate currently carries a skill ID, instruction, required +objects, effects, prerequisites, profile ID, checkpoint repository, immutable +revision and benchmark task ID. The present profile is +`libero-panda-openvla-v1`: a Franka Panda simulation using seven-dimensional OSC +pose actions. Its checkpoint is +[`openvla/openvla-7b-finetuned-libero-10`](https://huggingface.co/openvla/openvla-7b-finetuned-libero-10/tree/80970322773f81baa2e22fe495d0487b93a05cfa). +The ten task entries share the same checkpoint; downloading each task does not +create ten separate learned policies. + +The [example v1 manifest](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/skill-manifest.example.json) +is documentation only and contains no installed artifact. + +A broader skill registry will also need versioned observation and action +schemas, units, control rate, gripper convention, tested operating envelope, +required sensors, adapter versions and task-specific evaluation receipts. +Hardware compatibility would require its own tested adapter and qualification. +Those extended registry fields are a proposed contract, not an implemented +universal skill package. + +## Execution belongs to the local worker + +The browser communicates with an isolated loopback worker using the typed +`dvidia.robot-lab.v1` protocol and an in-memory capability. The worker performs +real detector calls, artifact verification, native resets, model loading and +bounded runs. It contains no physical robot driver. Checkpoint size, revision +and digests are verified before atomic activation; partial files never count as +an installed skill. The runtime loads explicit classes from pinned OpenVLA +source rather than executing a checkpoint's remote class mapping. + +An arbitrary household photo is planning input. It is not an executable LIBERO +state. Before a run, the user must select a supported benchmark task and capture +its real simulator camera and initial state. Task metadata is marked as +simulator evidence, distinct from image detections and user corrections. A run +is bound to that task, scene, seed and installed checkpoint. Completion must +come from the simulator's task-success signal. + +[OpenVLA](https://github.com/openvla/openvla) supplies a language-conditioned +robot action policy. The selected [checkpoint card](https://huggingface.co/openvla/openvla-7b-finetuned-libero-10) +describes a LIBERO-10 fine-tune; it is a concrete starting point for this +benchmark-specific experiment, not a generally installable housekeeping model. +The runtime uses `predict_action` with the `libero_10` normalization statistics. +Code, checkpoint and underlying model terms need separate review; the upstream +MIT labels do not replace the applicable Llama 2 terms or third-party asset +licenses. + +## What has actually run + +The retained receipt was recorded on 26 October 2026 and is published here as a +historical engineering result. It is not a fresh benchmark of this note's date. + +| Component | Recorded evidence | Limit | +| --- | --- | --- | +| Local runtime | Python 3.12.9, Torch 2.6.0 and MPS detected on an Apple Silicon Mac | Full OpenVLA memory fit and latency unverified | +| Real photo inference | Verified TorchVision SSDlite checkpoint; cup 0.9957, person 0.8254 and dining table 0.4730 on one editorial image | Confidence scores are detector outputs, not measured accuracy; no dirt, color, ownership or task-state judgment | +| Native task reset | LIBERO-10 task 5, seed 7, book/caddy camera image and flattened-state digest | Zero policy actions; ten no-op settling steps only | +| Reset repeatability | Worker-thread and independent main-thread reset digests match | Matching initial state does not establish deterministic inference or rollouts | +| Artifact installation | Interrupted transfer retained 8 MiB of a roughly 15.1 GB checkpoint | No activated checkpoint | +| Policy and steering | Transport and hook fixtures tested | No learned rollout, task result or behavioral steering measurement | + +The reset digest is +`b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e`. +The macOS worker uses MuJoCo's official offscreen CGL context to avoid initializing +GLFW/AppKit from an HTTP worker thread. This changes context selection, not the +physics or observed state. Only the task-5 asset closure was provisioned for the +recorded runtime check; other task assets remain locally unverified. + +![Actual LIBERO task-5 reset camera, before any policy action](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/libero-task5-reset.png) + +The photograph used for detector testing was an editorial illustration from the +app, not a physical robot camera capture. The simulator image above is an actual +rendered reset. Neither is evidence that a robot performed the requested action. + +## Neuron exploration is a separate measured intervention + +[Lyra's neuron explorer](https://github.com/lyramakesmusic/neuron-explorer) +inspired the interaction: inspect model units, change a unit's activation and +retain the resulting behavior. Its text-model experiments do not by themselves +establish a robot control method. The directly relevant robotics precedent is +[Mechanistic Interpretability for Steering Vision Language Action Models](https://vla-mech-interp.github.io/), +which studies semantic activation directions and interventions in OpenVLA and +π₀. Its [research code](https://github.com/Physical-AI-Safety-Institute/mechanistic-steering-vlas) +is a methodological reference; the DVIDIA worker uses an original hook and does +not vendor that toolkit. + +The implemented hook observes a zero-based intermediate Llama MLP channel at the +input to `down_proj`. It records the last token's value on each autoregressive +forward pass and can add a bounded raw dose to that channel. The displayed +per-action value averages the pre-intervention values across generation passes. +This is **a raw channel intervention**, not an identified “careful,” “fast” or +“move left” neuron. It is also not an implementation-equivalent reproduction of +the paper's semantic cluster interventions. Hook cleanup and cancellation have +unit coverage; their effects on a loaded robot policy remain unmeasured. + +Before behavioral claims, compare an unsteered baseline and intervention on the +same checkpoint, task, seed, preprocessing and exact reset digest. Record all +action dimensions, end-effector motion, activation summaries, cameras, +completion and failures. Current portable Pillow preprocessing is explicitly a +numerical variant of upstream's TensorFlow path, so published OpenVLA benchmark +scores are not a performance reference for this Mac configuration. + +## Staged evaluation and release gates + +1. **Conversation and retrieval:** freeze a small set of supported and + unsupported requests, including ambiguous destinations, negation and desired + state changes. Independently label compatible candidates. Report retrieval + precision and recall, unsupported-task handling and clarification behavior. + An interaction that looks correct is not evidence of execution. +2. **Installation and one-action diagnostic:** finish the revision-pinned + transfer, verify every artifact, load the full model and execute one real + action on the recorded task-5 reset. Record peak memory, load/inference time, + actual action vector and failure status. Stop if memory or rendering is + insufficient; a substitute controller does not satisfy this gate. +3. **Baseline simulation:** run bounded unsteered episodes on a frozen set of + seeds. Retain every attempt and the actual LIBERO success signal. Report + strict completion, step counts, latency and invalid attempts separately. +4. **Causal channel study:** choose units and doses using development scenes, + then freeze them. Compare paired baseline, sham-hook, positive/negative-dose + and randomly selected channel controls on held-out seeds. Measure task + completion alongside motion changes; do not select only successful videos. +5. **Skill sequencing and household transfer:** introduce separately qualified + skills, evidence refresh after each action, explicit recovery and destination + rules. Hardware execution needs independent adapter, sensing and task + qualification. This stage is not implemented or run. + +Stages 2–5 remain unrun. The first deliverable is an honest conversation that +connects to real tools and makes missing capability understandable. The research +question is whether that interface can select a skill whose measured behavior +achieves the user's intended change, and whether controlled interventions alter +that behavior reproducibly without degrading completion. diff --git a/docs/downloads/topics/physical-grounding/README.md b/docs/downloads/topics/physical-grounding/README.md index 7235a81..ae82743 100644 --- a/docs/downloads/topics/physical-grounding/README.md +++ b/docs/downloads/topics/physical-grounding/README.md @@ -1,5 +1,16 @@ # Physical grounding +## Chat to compatible robot skills — 27 October 2026 + +[From a conversation to compatible robot skills](../../reports/Chat%20to%20compatible%20robot%20skills.md) +proposes one conversation from photo and desired outcome to grounded skill +selection, installation, simulation and trace inspection. The local prototype +records real object detection and one repeated native task-5 reset. An interrupted +8 MiB partial checkpoint is not an installation; OpenVLA actions, steering +measurements, general household planning and hardware execution remain unrun. +[The runtime archive](runs/2026-10-27-robot-lab/README.md) preserves the exact +receipt, actual reset image and a documentation-only skill manifest. + ## Live PiPER shoe workstation — 13 October 2026 diff --git a/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/README.md b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/README.md new file mode 100644 index 0000000..9904fcb --- /dev/null +++ b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/README.md @@ -0,0 +1,68 @@ +# Photo, outcome and compatible robot skills + +Publication date: 27 October 2026. Runtime receipt: 26 October 2026. + +This archive supports [Chat to compatible robot skills](../../../../reports/Chat%20to%20compatible%20robot%20skills.md). +It contains real local runtime checks and an actual simulator reset, with no +learned policy rollout or steering result. + +- [Runtime receipt](runtime-receipt.json): sanitized dependency versions, + upstream source revisions, detector outputs, task/seed, reset digest and + original worker source digests. +- [Example v1 skill manifest](skill-manifest.example.json): current catalog fields + for task 5, marked unavailable until an actual runtime checks compatibility. + It bundles no policy and is not an installation. +- [Native task-5 reset camera](libero-task5-reset.png): 256 × 256 image rendered + after ten no-op settling steps, before any learned action. + +The receipt is copied byte-for-byte from the app's original engineering evidence. +Its source digests describe that snapshot, not later installer or interface +revisions. The separate test fixtures recorded in it are explicitly synthetic; +the detector inference and simulator reset are actual runtime observations. +No private capability, photo payload or account credential is included. + +## Reproduction boundary + +The public upstream sources are pinned as follows: + +| Artifact | Revision | +| --- | --- | +| OpenVLA source | `c8f03f48af692657d3060c19588038c7220e9af9` | +| LIBERO source | `8f1084e3132a39270c3a13ebe37270a43ece2a01` | +| `openvla/openvla-7b-finetuned-libero-10` | `80970322773f81baa2e22fe495d0487b93a05cfa` | + +The model revision identifies the intended checkpoint; its weights were not +installed for this receipt. The interrupted transfer retained an 8 MiB partial +file and no activation marker. Only the task-5 study/book/caddy asset closure +was fetched. A complete standalone worker release is separate from this +documentation archive; these two evidence files are not an installable skill. + +The reset uses task order index 0, LIBERO-10 task index 5 and seed 7. It performs +ten no-op settling steps and hashes the flattened simulator state. The worker +thread and independent main-thread run produced the same digest: + +```text +b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e +``` + +Apple MPS availability does not prove the full model fits or executes correctly. +The camera is simulation evidence, not physical hardware evidence. All upstream +model, simulator and asset terms remain applicable. + +## Hugging Face publication + +[Publication receipt](huggingface-publication.json) records the documentation-only +commit, exact file hashes and anonymous readback. The update preserves all 3,771 +pre-existing files except the root card, which receives one additional concept +section. It adds no model weights, live inference endpoint or hardware control. + +[Chat implementation follow-up](huggingface-followup.json) records the second +documentation-only update. The [interface screenshot](chat-interface.png) shows +the local built prototype with an actual simulator camera and a still-incomplete +checkpoint transfer. Development and built desktop browser workflows were +verified after the initial browser-control interruption; no policy action ran. + +[Final documentation and screenshot revision](huggingface-final.json) adds the +verified desktop interface image and updates the implementation status. All +three files pass anonymous readback at the recorded immutable Hub commit; all +3,774 other existing Space files remain unchanged. diff --git a/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/chat-interface.png b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/chat-interface.png new file mode 100644 index 0000000..c5bbf63 Binary files /dev/null and b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/chat-interface.png differ diff --git a/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-final.json b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-final.json new file mode 100644 index 0000000..b3b0533 --- /dev/null +++ b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-final.json @@ -0,0 +1,25 @@ +{ + "repo": "Dvidia/robot-simulation-lab", + "parent": "f4ae2b2cdce4a4a40359fc44d4dd1c99fb954dd9", + "files": [ + { + "path": "docs/robot-chat-v01/concept.md", + "bytes": 14785, + "sha256": "92c653b5c8aa674cf4cbd9f10e0f6d59d97a8011649ce9be4309fd5a6ed8e4a9" + }, + { + "path": "README.md", + "bytes": 41682, + "sha256": "88672953b3bb44095c14a96d86aa698de40c2aa3df357568f115ff8c4a38d14c" + }, + { + "path": "docs/robot-chat-v01/chat-interface.png", + "bytes": 92631, + "sha256": "4246dbb3f82cebee62ef4678e5712c3de7869e2c1a8dcf5aa54bf07de68e08a6" + } + ], + "commit": "a2e4b8a90fa2160e0ba0cfa51a668de5fef6233b", + "anonymous_readback_verified": true, + "unchanged_existing_files": 3774, + "scope": "concept, root documentation and interface screenshot only; no live inference, model weights or hardware" +} diff --git a/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-followup.json b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-followup.json new file mode 100644 index 0000000..18757ef --- /dev/null +++ b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-followup.json @@ -0,0 +1,20 @@ +{ + "repo": "Dvidia/robot-simulation-lab", + "parent": "9e6b5e36e96d15d5d456f4362f548972e46c3373", + "files": [ + { + "path": "docs/robot-chat-v01/concept.md", + "bytes": 14369, + "sha256": "95b10be488051d3ae4fdaa33a577cd6517ea1ff4b912615b43feb37cb61e3e96" + }, + { + "path": "README.md", + "bytes": 41628, + "sha256": "4d6d130bbfadeb073cbbc26dc004e6f0dadb77fa4391acd0c6fd44cd9373e1b4" + } + ], + "commit": "f4ae2b2cdce4a4a40359fc44d4dd1c99fb954dd9", + "anonymous_readback_verified": true, + "unchanged_existing_files": 3774, + "scope": "two documentation files only; no live inference, model weights or hardware" +} diff --git a/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-publication.json b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-publication.json new file mode 100644 index 0000000..e8158bf --- /dev/null +++ b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-publication.json @@ -0,0 +1,40 @@ +{ + "repo": "Dvidia/robot-simulation-lab", + "type": "space", + "parent": "9eaeeae67def2697ca664b54a754e915f0741c78", + "files": [ + { + "path": "README.md", + "bytes": 41454, + "sha256": "2467367a33640553b528937339a3700fd9fed62f24b03e76c77799639a40f3b9" + }, + { + "path": "docs/robot-chat-v01/concept.md", + "bytes": 13191, + "sha256": "c36d5556b7b176a3b93e0dcfc7f2ba1920806a24dc513811863e2b96de2ae5b7" + }, + { + "path": "docs/robot-chat-v01/runtime-evidence.json", + "bytes": 4868, + "sha256": "edb0ac36726120f0f65b94770183af14a0cc9234f69f212a7d383fb231260252" + }, + { + "path": "docs/robot-chat-v01/libero-reset.png", + "bytes": 62199, + "sha256": "c7bbaad0bfc572f01917f477ec03cf6149c0fb24993a6829ae74c4b0fa1fd95f" + }, + { + "path": "docs/robot-chat-v01/skill-manifest.example.json", + "bytes": 891, + "sha256": "70a4800ba1bafb3198e51de0dfe5379f029025507fc14236adf09c4dcfb5b060" + } + ], + "preserve_existing_files": 3772, + "scope": "documentation only; no live inference or hardware", + "commit": "9e6b5e36e96d15d5d456f4362f548972e46c3373", + "url": "https://huggingface.co/spaces/Dvidia/robot-simulation-lab/commit/9e6b5e36e96d15d5d456f4362f548972e46c3373", + "anonymous_readback_verified": true, + "unchanged_existing_files": 3771, + "live_inference": false, + "model_weights_uploaded": false +} diff --git a/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/libero-task5-reset.png b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/libero-task5-reset.png new file mode 100644 index 0000000..4188d37 Binary files /dev/null and b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/libero-task5-reset.png differ diff --git a/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/runtime-receipt.json b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/runtime-receipt.json new file mode 100644 index 0000000..f464b45 --- /dev/null +++ b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/runtime-receipt.json @@ -0,0 +1,152 @@ +{ + "recordedAt": "2026-10-26T07:00:08.623063+00:00", + "scope": "Local Mac runtime verification; no OpenVLA weights or learned rollout", + "profileId": "libero-panda-openvla-v1", + "model": { + "checkpoint": "openvla/openvla-7b-finetuned-libero-10", + "revision": "80970322773f81baa2e22fe495d0487b93a05cfa", + "weightsInstalled": false, + "inferenceExecuted": false, + "steeredRobotActionsExecuted": false + }, + "sources": { + "openvlaRevision": "c8f03f48af692657d3060c19588038c7220e9af9", + "liberoRevision": "8f1084e3132a39270c3a13ebe37270a43ece2a01", + "liberoAssetScope": "Only task 5 study/book/caddy asset closure was fetched; other task assets unverified" + }, + "runtime": { + "python": "3.12.9", + "mpsAvailable": true, + "torch": "2.6.0", + "torchvision": "0.21.0", + "numpy": "1.26.4", + "Pillow": "13.0.0", + "transformers": "4.40.1", + "tokenizers": "0.19.1", + "timm": "0.9.10", + "accelerate": "0.30.1", + "huggingface-hub": "0.23.0", + "safetensors": "0.4.5", + "mujoco": "3.1.1", + "robosuite": "1.4.1", + "bddl": "1.0.1", + "gym": "0.25.2", + "cloudpickle": "2.1.0", + "easydict": "1.9", + "imageio": "2.38.1", + "PyYAML": "6.0.3", + "sentencepiece": "0.2.2", + "numba": "0.68.0", + "llvmlite": "0.50.0", + "scipy": "1.17.1", + "mpsContext": "Host graphics access; sandboxed process reports unavailable", + "future": "1.0.0", + "matplotlib": "3.10.8", + "contourpy": "1.3.2" + }, + "sourceClasses": { + "verified": true, + "classes": [ + "OpenVLAConfig", + "OpenVLAForActionPrediction", + "PrismaticProcessor" + ], + "modelInstantiated": false + }, + "photoDetection": { + "verified": true, + "image": "public/skill-cup.jpg", + "imageSize": [ + 1024, + 768 + ], + "detector": "TorchVision SSDlite320 MobileNetV3 COCO", + "checkpointSha256": "a79551df90c79834bcd3bb3845ef9d966b5449a3a9b2833ae8404778ca5d65d2", + "elapsedSeconds": 22.981, + "objects": [ + { + "id": "detected-0", + "label": "cup", + "confidence": 0.9956732392311096, + "box": [ + 0.39903393387794495, + 0.47789963086446124, + 0.6025710105895996, + 0.7762130896250407 + ], + "source": "detector" + }, + { + "id": "detected-1", + "label": "person", + "confidence": 0.8253621459007263, + "box": [ + 0.4701840579509735, + 0.1547549068927765, + 1, + 0.7927147547403971 + ], + "source": "detector" + }, + { + "id": "detected-2", + "label": "dining table", + "confidence": 0.4729897677898407, + "box": [ + 0, + 0.5605224768320719, + 1, + 0.9700164794921875 + ], + "source": "detector" + } + ], + "limitations": "Coarse detector hypotheses, no color/dirt/ownership/task-state inference" + }, + "simulatorReset": { + "verified": true, + "taskId": 5, + "seed": 7, + "instruction": "pick up the book and place it in the back compartment of the caddy", + "resetHash": "b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e", + "image": "docs/evidence/robot-lab-libero-reset-20261026.png", + "imageSize": [ + 256, + 256 + ], + "actualState": [ + -0.29572729160672406, + 0.013003376967992303, + 1.1680005404829354, + 0.9995841690598055, + 0.0017969395174486705, + -0.02877830619454855, + 0.0002628040344518339, + 0.03872207014699441, + -0.038723508260440485 + ], + "elapsedSeconds": 15.78, + "threaded": true, + "policyActionsExecuted": 0, + "settlingNoopSteps": 10, + "renderer": "Official mujoco.cgl.GLContext via original constructor adapter for robosuite1.4.1", + "independentMainThreadResetHash": "b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e", + "repeatResetHashMatched": true, + "cameraVisuallyInspected": true, + "initialMacFailureResolved": "GLFW initialized AppKit off the main thread; official offscreen CGL adapter removes that path" + }, + "workerTests": { + "command": "python -m unittest discover -s robot-lab/tests -q", + "python": "3.12.9", + "passed": 30, + "failed": 0, + "fixtureScope": "Unit/protocol/transport/hook tests include explicit synthetic fixtures; separate simulator and detector results are actual runtime measurements" + }, + "workerSourceSha256": { + "runtime.py": "5ff4db2097335304e755a82c299a3521830b3f3e9bc545f95dbea5f1161df07f", + "server.py": "acc0f9e5f2ad873df47feae801094696463df66ce9d388a9fe057516a4d74e76", + "artifacts.py": "0567fd4c4ccf369e8fef2574e643daa7df32b6a05e0a06c2ee7941cade05ebe6", + "catalog.py": "dbc9916f9ec403507e1f305eb31d7c7b4f3be4e67807d85fa7aedb347e45f595", + "requirements.txt": "d30b0762abaedf377dcac22270a87a895f1e587c6d241bcb9705011b05d25eac" + } +} diff --git a/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/skill-manifest.example.json b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/skill-manifest.example.json new file mode 100644 index 0000000..4daa472 --- /dev/null +++ b/docs/downloads/topics/physical-grounding/runs/2026-10-27-robot-lab/skill-manifest.example.json @@ -0,0 +1,26 @@ +{ + "id": "openvla-libero10-5", + "title": "Put book into caddy", + "description": "OpenVLA LIBERO-10 task 5; one shared learned checkpoint.", + "instruction": "pick up the book and place it in the back compartment of the caddy", + "objects": [ + "book", + "caddy" + ], + "effects": [ + "book in back compartment of caddy" + ], + "prerequisites": [ + "LIBERO-10 matching task reset", + "Franka Panda simulation with 7D OSC pose actions", + "Reachable task objects in benchmark arrangement" + ], + "profileId": "libero-panda-openvla-v1", + "checkpoint": "openvla/openvla-7b-finetuned-libero-10", + "revision": "80970322773f81baa2e22fe495d0487b93a05cfa", + "license": "MIT (OpenVLA code and checkpoint); LIBERO MIT; Llama 2 terms apply", + "taskId": 5, + "installed": false, + "compatible": false, + "blocker": "Documentation example; runtime compatibility has not been checked" +} diff --git a/docs/feed.xml b/docs/feed.xml index e694ae4..fc3a329 100644 --- a/docs/feed.xml +++ b/docs/feed.xml @@ -1,2 +1,2 @@ -DVIDIA Researchhttps://research.dvidia.org/Research footprints on demonstrations, calibrated simulation and robot skills.enLearning grip feedback with a frozen PiPER arm referencehttps://research.dvidia.org/papers/learned-piper-grip-v02/https://research.dvidia.org/papers/learned-piper-grip-v02/Tue, 27 Oct 2026 12:00:00 GMTJaw feedback 6/24 · blind 8/24 · feedforward 5/24 · teacher 24/24 · no demonstrated feedback advantage. After five seven-command residual candidates each failed all 16 development cases, a separate 25-node learned jaw adapter preserves the frozen six-axis PiPER arm references. Trained on 64 synthetic teacher episodes and 64,000 causal labels, it completes 6/24 frozen confirmation cases, versus 8/24 for its matched blind control, 5/24 feedforward, 24/24 authored teacher and 0/24 open jaws. Feedback succeeds in 0/12 interrupted cases; 14/24 terminal-only passes conceal strict task failures. All 120 confirmation attempts are valid, and an independent numeric audit reconstructs 360 retained attempts. CPU inference p95 is 0.501 ms; this establishes no live-feedback advantage, recovered grip, human-video learning or hardware qualification.Learning PiPER block placement from synthetic demonstrationshttps://research.dvidia.org/papers/learned-piper-placement-v01/https://research.dvidia.org/papers/learned-piper-placement-v01/Tue, 27 Oct 2026 12:00:00 GMTLearned 5/12 · teacher 12/12 · three controls 0/12 each · simulation only. A compact feedforward joint-and-jaw primitive trained on 32 successful synthetic demonstrations completes 5/12 frozen PiPER placement attempts from unseen starting positions. The authored teacher completes 12/12; three controls complete 0/12 each. All twelve learned terminal poses pass settling checks, while seven fail the full native contact-history gates. Complete tapes, failed development candidates, an independent audit and an installable local runner preserve the reliability gap. This is privileged-state reference distillation for a fixed rigid block and two trained goals, not human-video learning, learned feedback or physical robot qualification.Independent camera and actuator corroboration v0.4https://research.dvidia.org/papers/sensor-corroboration-v04/https://research.dvidia.org/papers/sensor-corroboration-v04/Mon, 19 Oct 2026 12:00:00 GMT272 cases · 69/112 false completions prevented · 43 remain · 73/80 true completions abstained. A fresh eight-layout PiPER simulation benchmark adds separately processed RGB and drive checks to a local physical-goal verifier. The checks prevent 69 of 112 false completion readings while leaving 43 replay, identity, orientation, support and sampling blind spots; the conservative gate abstains on 73 of 80 baseline true completions, including all 64 deliberate sensor-fault controls. The measured CPU path is 22.1 ms median and 31.5 ms p95, excluding 200 ms acquisition. All raw channels and results are preserved; this is synthetic evidence, not hardware validation.A local verifier for physical completionhttps://research.dvidia.org/papers/local-goal-verifier-v03/https://research.dvidia.org/papers/local-goal-verifier-v03/Mon, 19 Oct 2026 12:00:00 GMTNative: 16/16 true complete, 0/48 false complete · forged readings: 8/8 false complete. An offline audit traces the seven missed Jev completions without claiming an internal model cause. A local deterministic verifier then checks timestamped pose, motion, support, release and uncertainty on 200 fresh cases. It recognizes all 16 native positives and rejects all 48 native negatives; 128 evidence faults yield 120 abstentions and 8 incomplete answers. All 8 consistent forged streams produce physical false completions, exposing the remaining sensor-trust problem. CPU-only p95 is 0.318 ms, excluding the required 200 ms history and sensing. Complete native inputs, every decision, independent review and explicit failure cohorts remain public; no new Jev calls or hardware qualification.Jev image-goal verification: the measured follow-uphttps://research.dvidia.org/papers/image-goal-jev-v02/https://research.dvidia.org/papers/image-goal-jev-v02/Mon, 19 Oct 2026 12:00:00 GMT64/64 valid responses · telemetry: 0/24 false complete, 1/8 true complete. The first measured run preserves all 64 requests from the image-goal v0.1 protocol. Jev returned valid structured responses for every trial and made no false-completion declaration, but recognized only 1/8 physically complete cases with privileged native telemetry and 0/8 with images alone. All four occluded complete scenes were called incomplete in both conditions. RGB abstention matched 8/16 ambiguous references. Median online latency was 1.776 seconds, p95 3.287 seconds. The release preserves every result, the independent review and the original unrun protocol. This small synthetic diagnostic does not qualify a robot completion gate, actuator loop or learned skill.Image-goal verification: native benchmark and Jev protocolhttps://research.dvidia.org/papers/image-goal-jev-v01/https://research.dvidia.org/papers/image-goal-jev-v01/Sun, 18 Oct 2026 12:00:00 GMT32 native fixtures · 4 identical-image pairs · 64 Jev trials not run. A frozen PiPER image-goal cohort crosses eight physical conditions with four layouts. Four moving/settled pairs have byte-identical images and different completion labels, exposing when one image cannot establish task completion. Native measurements verify 32 authored fixtures; a paired 64-request Jev protocol distinguishes RGB judgments from privileged simulator telemetry. The provider evaluation has not run because credentials were unavailable: accuracy, false-completion rates and service latency remain unmeasured. Every native state, image, frozen request hash and explicit unrun result is retained. No learned policy or hardware qualification is claimed.A visual goal still needs physical feedbackhttps://research.dvidia.org/papers/goal-images-physical-feedback/https://research.dvidia.org/papers/goal-images-physical-feedback/Sun, 18 Oct 2026 12:00:00 GMT14/14 exact passive replays · 249,502 native ticks · visual models unbenchmarked. A goal image specifies a destination; physical feedback verifies whether a robot gets there. We describe a bounded decision role for TypeSafe Jev, distinguish it from action-conditioned world-model planning, and propose an independently scored reaching experiment. A separate passive audit reproduces fourteen previously inspected rigid-box arm episodes exactly and records 249,502 native ticks. Millisecond contact gaps hidden between stored observations become visible, while task outcomes remain unchanged. Complete inputs, traces, source snapshots and a verified downloader preserve every case. This is measurement work, not a new policy benchmark, PiPER shoe result or hardware qualification.Give the gripper a fair testhttps://research.dvidia.org/papers/shoe-graspability-v03/https://research.dvidia.org/papers/shoe-graspability-v03/Wed, 14 Oct 2026 12:00:00 GMT24/24 valid · 30 g: 8/12 lifts · 300 g: 0/12 · no net gain from removing rake. A frozen 24-attempt PiPER diagnostic separates three rigid shapes, 30 g and 300 g loads, and mounted versus absent rake conditions. All attempts are valid; eight of twelve lightweight controls retain a direct lift while every 300 g attempt fails, including the small block. Removing the rake gains one narrow-shoe result and loses one block result, with no net completion gain. A verified settled reference, dense contact traces, unchanged drive fingerprint and independent scoring expose limits in the modeled grasp and force response. Recorded replays and runnable CPU code retain every failure. No autonomous tool removal, retrieval, learned skill or physical hardware qualification is claimed.From reaching a shoe to keeping holdhttps://research.dvidia.org/papers/continuous-shoe-grip-v02/https://research.dvidia.org/papers/continuous-shoe-grip-v02/Tue, 13 Oct 2026 12:00:00 GMT20/20 valid · 2/10 completed under both policies · 9 contact-loss alarms verified. A frozen twenty-attempt native MuJoCo comparison adds continuous acquisition checks and bounded lowering to the mounted-rake PiPER controller. Both policies complete the same 2/10 fresh cases: no gains or losses. Nine bilateral-contact-loss alarms follow a trace-verified native origin by 20–52 ms, median 30 ms; three acquisition failures are separate. Twelve supported lowering responses produce no additional completion. A non-finger-support gap remains unmonitored. Complete recorded trajectories, independent audit and development failures are public. Authored policies, synthetic contact and privileged observations establish no learned skill or physical robot qualification.Skill prerequisites: access, reach and recoveryhttps://research.dvidia.org/papers/skill-prerequisites-v01/https://research.dvidia.org/papers/skill-prerequisites-v01/Tue, 13 Oct 2026 12:00:00 GMT16/16 valid · 2 lid contacts avoided · 3/4 retrieval checkpoints · 0/4 retained tool lifts. A frozen sixteen-attempt native MuJoCo study tests hinged shoe-box access and a mounted rake for an out-of-workspace shoe. Guarded access avoids two baseline lid collisions while task completion stays at 2/4; a transient withdrawal adds 0.70 simulated seconds without being necessary in its matched baseline. The mounted rake reaches an intermediate retrieval checkpoint in 3/4 cases, but all subsequent retained lifts fail and the short-tool case is deferred. All native traces and failures remain public. The browser replays recorded states; fresh physics runs locally. Privileged observations, synthetic contact and authored policies establish no learned skill or hardware qualification.Live PiPER shoe workstation v0.1https://research.dvidia.org/papers/live-piper-shoe-workstation-v01/https://research.dvidia.org/papers/live-piper-shoe-workstation-v01/Tue, 13 Oct 2026 12:00:00 GMTLive browser physics · 14/16 targets · strict contact parity failed · shoe lifting unqualified. A six-axis PiPER and its coupled gripper respond to user targets while MuJoCo runs locally in a browser worker. The final native run retains 20 valid attempts: 14/16 target trials pass, both high-target repeats fail, and jaw, impulse, disabled-table and 600-second hold checks pass. A ten-minute headless Chromium run reports 51.389 render calls per second and a 0.9999795 simulation/wall-time ratio on the tested Apple M5 host; simulated movement response is 61.9 ms p95 across 61 commands. Render calls do not measure GPU presentation. Native/WASM numerical and loaded-contact-count checks pass, while strict raw contact counts differ in 13/1,500 states. All fifteen development shoe-lift attempts fail. Complete development, failure and timing records remain separate. This authored rigid-shoe environment establishes no learned skill, calibrated material model or physical robot qualification.Reusable shoe asset library v0.1https://research.dvidia.org/papers/shoe-asset-library-v01/https://research.dvidia.org/papers/shoe-asset-library-v01/Mon, 12 Oct 2026 12:00:00 GMTSix shoe proxies · three arms · support 27/27, disabled table 0/27 · manipulation unqualified. Six original MIT rigid shoe proxies compose with the pinned UR5e/Robotiq, PiPER and SO-101 models through versioned scene descriptors and offline native 3D previews. A frozen 54-attempt test records 27/27 supported pairs and 0/27 support successes when table collisions are disabled; all 54 gravity checks pass without numerical warnings. No loaded non-table shoe contacts appear in the 20 ms samples, which cannot rule out shorter transients. All sampled trajectories and 27,000 native-step durations remain available. On one Apple M5 host, 54 simulated seconds take 12.3616 wall seconds including fresh model construction; the slowest attempt takes 6.7047 seconds. Geometry and physical properties are authored assumptions. Loading, gravity and passive support do not establish shoe grasping, calibrated deformation, a learned policy or hardware transfer.PiPER placement reliability v0.5https://research.dvidia.org/papers/piper-reliability-v05/https://research.dvidia.org/papers/piper-reliability-v05/Mon, 12 Oct 2026 12:00:00 GMT23/36 revised grasp · 15/36 v0.4 · 8 gained, none lost · hardware unqualified. Three frozen fresh PiPER cohorts retain 252 numerically valid attempts and 144 failed negative controls. A fixed diagonal grasp with bounded pose planning completes 23/36 placements versus 15/36 for frozen v0.4 and 18/36 with second-lift recovery alone: eight gained completions and none lost. The fifteen shared successes are 0.772 simulated seconds slower on average. Three lift-only second probes qualify and finish their tasks; the full grasp candidate needs none. Twelve low-friction acquisitions and one later retention failure remain. Every trajectory, native task outcome and raw computation duration is retained, including 1,035 windows exceeding 20 ms and a 3.082-second outlier on a loaded host. Native mechanics and scoring remain unchanged. Separate development keeps 200 earlier attempts, including discarded regressions. Authored privileged simulation establishes no learned skill, real-time guarantee or physical qualification.PiPER contact recovery v0.4https://research.dvidia.org/papers/piper-contact-recovery-v04/https://research.dvidia.org/papers/piper-contact-recovery-v04/Sun, 11 Oct 2026 12:00:00 GMTAll modes 4/12 · two resumed tasks faster · controls 0/48 · hardware unqualified. A fresh frozen 84-attempt PiPER comparison completes 4/12 placements for v0.3 history, v0.4 carry-only and v0.4 full; all 48 matched open-jaw and hold controls fail. Two completed post-hold placements finish 2.440 and 2.640 simulated seconds sooner with unchanged 220/240 ms settling holds. Four low-friction failures reach refusal 2.88–2.90 seconds sooner without becoming successes. Fresh nominal-friction 60 g layouts expose two held-but-too-low resume failures in every mode; neither reaches speed restoration. Six earlier acquisition failures are traced to insufficient support under the pinned position servo and side pinch, not the actuator force cap. All native mechanics and outcome criteria remain unchanged. Complete trajectories and 84,000 raw computation windows are retained; none exceeds 20 ms on this host. Source-snapshotted development remains separate. Privileged authored simulation establishes no learned skill, true-slip accuracy or physical qualification.PiPER contact history v0.3https://research.dvidia.org/papers/piper-contact-history-v03/https://research.dvidia.org/papers/piper-contact-history-v03/Sun, 11 Oct 2026 12:00:00 GMTHistory 6/12 · baseline and probe-only 4/12 · controls 0/48 · hardware unqualified. A frozen paired PiPER experiment retains all 84 numerically valid attempts across twelve synthetic rigid-block scenes. History completes 6/12 placements versus 4/12 for both original motion and the identical acquisition probe without carry monitoring; all 48 separately matched open-jaw and hold controls fail. The two added completions occur at nominal friction with 60 g blocks. Four tasks complete after five closed holds/resumes, but only four of five resume windows meet the independent low absolute TCP-speed criterion. All six supported acquisition retries fail. Native gains, forces and outcome rules remain unchanged; requested movement reduction is distinct from actual speed, and two slowdown windows include early holds. Seventy-five of 84,000 computation windows exceed 20 ms. Complete sealed trajectories, original model notices, source-snapshotted development and pinned offline inspection preserve both success and failure. Privileged authored simulation establishes no learned skill, true-slip detector accuracy, shoe competence or physical qualification.PiPER grip validation and recoveryhttps://research.dvidia.org/papers/piper-grip-validation-and-recovery/https://research.dvidia.org/papers/piper-grip-validation-and-recovery/Sun, 11 Oct 2026 12:00:00 GMTModerate-friction regression · baseline 4/6, guarded 2/6 · hardware unqualified. A slow test lift and bounded supported retry add recorded acquisition decisions to the pinned PiPER contact pilot, but completion regresses: at moderate friction baseline completes 2/2 and guarded completes 0/2. Both complete 2/2 normal cases; neither completes the two low-friction cases. All 30 attempts remain numerically valid, including 18 failed controls and full-duration aborts. Four qualification decisions are followed by a brief independent following window, yet two tasks still fail; no successful recovery or overall improvement is demonstrated. Actual motion continues after hold requests, and 53/30,000 computation windows exceed 20 ms. Complete compressed receipts, original model notices and pinned offline inspection preserve the evidence. Authored privileged-state simulation establishes no detector accuracy, learned shoe skill, calibrated contact or physical robot qualification.Native grasp task benchmarkhttps://research.dvidia.org/papers/native-grasp-task-benchmark/https://research.dvidia.org/papers/native-grasp-task-benchmark/Sun, 11 Oct 2026 12:00:00 GMT6/6 simulated placements · controls 0/12 · real-time and hardware unqualified. A pinned six-axis arm and coupled gripper complete 6/6 frozen synthetic rigid-block placements, while matched recorded-arm/open-jaw and hold controls complete 0/12. Native contact must produce grasp, sustained lift, retained carry, release and settled placement with tool retreat. All 18 attempts and timing traces remain in the receipt; one placement takes 29.51 wall seconds for 20 simulated seconds and misses 251 of 1,000 computation windows, so real-time operation is not established. An offline 3D mesh replay exposes the recorded motion. The authored privileged-state baseline provides a benchmark for a future compact DVIDIA action policy, not a learned footage-to-action model, validated shoe simulation or physical robot qualification.Hardware matched robot simulationhttps://research.dvidia.org/papers/hardware-matched-robot-simulation/https://research.dvidia.org/papers/hardware-matched-robot-simulation/Sun, 11 Oct 2026 12:00:00 GMTPinned CPU mechanics · PiPER contact pilot · physical fidelity unmeasured. A separate UR5e/Robotiq reference preserves pinned geometry, inertias, moving-link collision proxies and coupled gripper structure: fourteen modeled coordinates and seven commands. Eighteen deterministic unloaded motion/closure runs are numerically valid; at a 1 ms step, CPU physics throughput measures 40.80–47.29 simulated seconds per measured stepping wall second on one host, excluding loading, sensing, rendering, policy and real actuation. Adjacent timestep comparisons and a matched static overlap expose numerical sensitivity and collision participation. A pure-Python paired trace scorer measures joint, tool and jaw mismatch without automatic qualification. Left/right five-finger references are separately audited; LeRobot collection with an original DVIDIA policy is proposed. Serial calibration, physical response, shoe task learning and bimanual execution remain unmeasured. An offline 3D mesh replay now exposes the recorded arm/gripper poses without rerunning physics. PiPER and SO-101 now have pinned CPU profiles with 6/6 valid unloaded numerical attempts each and explicit native command units. PiPER also has a frozen authored rigid-block contact pilot: nominal placement 2/2, low-friction placement 0/1, open-jaw 0/3 and hold 0/3, with all nine attempts numerically valid. One of 9,000 computation windows exceeds 20 ms; hardware real-time response is unqualified. SO-101 object tasks, learned policies and physical qualification remain open. YAM, OpenArm, ARX X5 and PAROL6 remain upcoming.DVIDIA Observation Lab pilothttps://research.dvidia.org/papers/dvidia-observation-lab/https://research.dvidia.org/papers/dvidia-observation-lab/Sun, 11 Oct 2026 12:00:00 GMTLocal review release · motion and actuator source pilots · hardware unqualified. DVIDIA Observation Lab supplies local object/palm proposals, reviewed timeline revisions and separate causal image-motion forecasts; real annotation and motion accuracy remain unmeasured. A new CPU pilot learns next-20-ms joint/jaw deltas from current native state and issued commands. Twelve seeded groups retain 36 recordings with 8/2/2 train/development/test grouping; 4,553 test transitions are not independent robot trials. The 6,438-byte action model computes at 0.207 ms p95 on one host and improves RMSE, while jaw MAE/worst error regress against persistence and an initial opening is substantially overpredicted. These are privileged simulator measurements and model-only timings. A proposed shoe capture plan has no footage or executed shoe benchmark. The v0.1 release and hosted synthetic review demo remain separate; no shoe policy, physical adapter, validated shoe physics or hardware qualification is established.Robotics research pipelinehttps://research.dvidia.org/papers/robotics-research-pipeline/https://research.dvidia.org/papers/robotics-research-pipeline/Fri, 09 Oct 2026 12:00:00 GMTArchitecture proposal · current simulation alpha · hardware unqualified. An updated architecture connects immutable Skillspace recordings, reviewed task-label proposals and synchronized measured telemetry to grouped datasets, separate visual and motor learners, and a proposed local state supervisor. The implemented CPU baseline retains authored phases, gripper decisions and recovery around its learned joint-delta head. New annotation and supervisor pilots, fresh frozen confirmation, physics calibration and compatible skill-installation gates define the next milestones. Automatic labeling, learned supervision, human-video action recovery and physical arm qualification remain unestablished; this publication adds no new training or hardware result.Skillspace footage training pipelinehttps://research.dvidia.org/papers/skillspace-footage-training/https://research.dvidia.org/papers/skillspace-footage-training/Thu, 08 Oct 2026 12:00:00 GMTPublic CPU package · one native scene qualified · simulation-only. The public Dvidia-Inference/dvidia-training repository provides a standalone CPU pipeline that audits Skillspace media, freezes grouped splits and exports inspectable artifacts. Ten synthetic six-second clips train a visual predictor with a 0.5106-second median pipeline time across three repeats on one macOS host. Aligned native simulator telemetry supplies 4,029 training labels for a learned joint-delta head, evaluated on one held-out recording. Its capsule passes native placement qualification in that recording's exact scene; identical joint targets with open jaws fail. Installation and hardware guides accompany the package. Human-video action recovery, shoe organization, demonstration sufficiency, independent scene generalization and physical robot competence remain unestablished.Skill capsule acquisition and optimizationhttps://research.dvidia.org/papers/skill-capsule-acquisition/https://research.dvidia.org/papers/skill-capsule-acquisition/Wed, 07 Oct 2026 12:00:00 GMTSimulation-only · teacher/student 7/8 · no overall speedup. A compact learned CPU movement head replaces joint-target computation inside an authored placement supervisor. Teacher and student both pass 7/8 in a new frozen evaluation, while episode wall time increases from 6.273 to 6.360 seconds. A 39,845-byte JSON capsule imports into the local lab, checks its exact runtime and qualifies the current scene through native placement and a recorded-arm/open-jaw control. Scene changes require a fresh test. One known scene and repeat pass native network denial; perception, complete task learning, other embodiments and hardware remain unqualified.Skillspace arm benchmarks and evolutionhttps://research.dvidia.org/papers/skillspace-arm-benchmarks/https://research.dvidia.org/papers/skillspace-arm-benchmarks/Wed, 07 Oct 2026 12:00:00 GMTMixed held-out result · 30/32 baseline, 29/32 updated. Two controller revisions improve development completion from 14/20 to 17/20, but the frozen paired test declines from 30/32 to 29/32: both pass all 24 normal-strength cases while actuator-stress completion falls from 6/8 to 5/8. Instrumented CPU throughput also declines. We publish the regression, unchanged contact and success rules, exact controller snapshots, richer physical inputs, runtime-bound installation and a proposed staged Skillspace contract. Contact forces and penetration remain uncalibrated; learned perception and physical qualification are open.Installing a Skillspace on a simulated armhttps://research.dvidia.org/papers/skillspace-arm-installation/https://research.dvidia.org/papers/skillspace-arm-installation/Wed, 07 Oct 2026 12:00:00 GMTPublic skill link to authored adapter · 6/6 scene placements. The first link-to-arm product path installs an existing DVIDIA placement envelope, binds a local authored controller to one simulated arm, and reuses it in independently grounded scenes. Native jaw contacts carry a free box; six layouts complete, while no-motion and identical joint commands with open jaws each complete zero of six. Installed execution passes process-level offline checks. Task metadata contains no learned policy; perception, cup geometry, full arm collisions and physical qualification remain open.Offline cable training environmenthttps://research.dvidia.org/papers/offline-cable-training-environment/https://research.dvidia.org/papers/offline-cable-training-environment/Wed, 07 Oct 2026 12:00:00 GMTCPU simulation-only · Controller search and frozen evaluation executed. The first runnable product loop combines an anchored cable environment, local controller-parameter search, frozen evaluation, offline replay and a manifest-backed skill pack. The selected controller and a fixed feedback baseline both completed 10/10 held-out simulated episodes; zero-force and release controls completed 0/10. A native macOS network-denial run completed 3/3 additional episodes. Ideal endpoint attachment and privileged state bound the result; physical rope calibration, gripping, sensors and GPU throughput remain untested.Fast accurate robot training simulationhttps://research.dvidia.org/papers/fast-accurate-robot-training-simulation/https://research.dvidia.org/papers/fast-accurate-robot-training-simulation/Wed, 07 Oct 2026 12:00:00 GMT54 CPU fixture trials · GPU and physical transfer untested. A design for an offline, self-hosted training environment that turns demonstrations into calibrated practice and qualified robot skills. We examine contact, deformation, sensor feedback and the performance–accuracy tradeoff, then report 54 deterministic native MuJoCo CPU fixture trials. Fine endpoint agreement did not establish accurate contact transients, and coarse timesteps missed collisions. Minimum GPU memory, rope accuracy and physical transfer remain unmeasured.Grok robotics research on Xhttps://research.dvidia.org/papers/grok-robotics-research-on-x/https://research.dvidia.org/papers/grok-robotics-research-on-x/Wed, 07 Oct 2026 12:00:00 GMTPrimary sources checked · Experiments proposed. Three Grok 4.7 research passes surface observation-to-action, synthetic-practice and reusable-repair ideas. Independent primary-source checks compare those findings with our thesis: observation supplies goals and state, language retrieves a skill, spatial expectations guide attention, and executable feedback with evaluation establishes competence. X posts remain discovery leads. No universal demonstration count or arbitrary-robot transfer follows from the reviewed results.Spatially grounded downloadable robot skillshttps://research.dvidia.org/papers/spatially-grounded-downloadable-robot-skills/https://research.dvidia.org/papers/spatially-grounded-downloadable-robot-skills/Wed, 07 Oct 2026 12:00:00 GMTSource review · DVIDIA experiments not run. A synthesis of learning from human demonstrations, language-conditioned spatial attention, simulation expansion and robot-specific skill delivery. Spatial priors can accelerate inspection but do not determine contact state or control. The proposed downloadable skill packages a controller, initiation and termination conditions, compatible sensing and hardware, recovery behavior and measured evidence. Separate experiments would test attention, data coverage, feedback and transfer.Clef and grounded video observationshttps://research.dvidia.org/papers/clef-and-grounded-video-observations/https://research.dvidia.org/papers/clef-and-grounded-video-observations/Tue, 06 Oct 2026 12:00:00 GMTFive licensed excerpts · Accuracy evaluation outstanding. A preregistered image-API feasibility run processed five licensed three-second POV excerpts using four timestamped frames, a Gemma caption and a Clef-flash check. All five completed at a price-derived inference estimate of $0.00140634. Post-run inspection found an object error accepted by the checker and omitted state changes. API and schema compatibility were demonstrated; annotation accuracy and whole-task success were not.From Skillspace to robot skillhttps://research.dvidia.org/papers/from-skillspace-to-robot-skill/https://research.dvidia.org/papers/from-skillspace-to-robot-skill/Tue, 06 Oct 2026 12:00:00 GMTGraduation and robot training not run. An Argus review and a proposal to connect original recordings, reviewed episodes, immutable datasets, training runs, held-out evaluation and robot-tested releases. The review separates video annotation from action supervision and defines compatibility beyond a robot’s degree-of-freedom count. A proposed 40-episode annotation pilot and evidence-based graduation criteria remain unrun.Turn physical footage into testable evidencehttps://research.dvidia.org/papers/physical-skill-research-agenda/https://research.dvidia.org/papers/physical-skill-research-agenda/Tue, 06 Oct 2026 12:00:00 GMTFour baseline protocols proposed. A research agenda covering visible capture evidence, relations over time, evaluation provenance and physical grounding. Each question has a bounded protocol and a failure condition. The aim is to measure whether structure and physical signals improve a defined task before scaling collection. Strategic opinions, author-reported results and our untested proposals are kept distinct. +DVIDIA Researchhttps://research.dvidia.org/Research footprints on demonstrations, calibrated simulation and robot skills.enFrom a conversation to compatible robot skillshttps://research.dvidia.org/papers/chat-to-robot-skills/https://research.dvidia.org/papers/chat-to-robot-skills/Tue, 27 Oct 2026 12:00:00 GMTPhoto inference and one native reset verified · OpenVLA actions and steering not run. A conversation can carry a photo and desired outcome through scene review, skill retrieval, compatible installation, simulation and inspectable action traces. The local prototype uses ten LIBERO-10 task manifests sharing a pinned OpenVLA checkpoint. Real SSDlite inference and a repeatable native task-5 reset are recorded; the checkpoint transfer stopped after an 8 MiB partial file, so learned policy actions and activation-steered behavior remain unrun. The note separates observed evidence, candidate relevance, executable compatibility and measured completion, and defines staged evaluation for the chat workflow and raw MLP channel interventions. General household planning and hardware execution remain unqualified.Learning grip feedback with a frozen PiPER arm referencehttps://research.dvidia.org/papers/learned-piper-grip-v02/https://research.dvidia.org/papers/learned-piper-grip-v02/Tue, 27 Oct 2026 12:00:00 GMTJaw feedback 6/24 · blind 8/24 · feedforward 5/24 · teacher 24/24 · no demonstrated feedback advantage. After five seven-command residual candidates each failed all 16 development cases, a separate 25-node learned jaw adapter preserves the frozen six-axis PiPER arm references. Trained on 64 synthetic teacher episodes and 64,000 causal labels, it completes 6/24 frozen confirmation cases, versus 8/24 for its matched blind control, 5/24 feedforward, 24/24 authored teacher and 0/24 open jaws. Feedback succeeds in 0/12 interrupted cases; 14/24 terminal-only passes conceal strict task failures. All 120 confirmation attempts are valid, and an independent numeric audit reconstructs 360 retained attempts. CPU inference p95 is 0.501 ms; this establishes no live-feedback advantage, recovered grip, human-video learning or hardware qualification.Learning PiPER block placement from synthetic demonstrationshttps://research.dvidia.org/papers/learned-piper-placement-v01/https://research.dvidia.org/papers/learned-piper-placement-v01/Tue, 27 Oct 2026 12:00:00 GMTLearned 5/12 · teacher 12/12 · three controls 0/12 each · simulation only. A compact feedforward joint-and-jaw primitive trained on 32 successful synthetic demonstrations completes 5/12 frozen PiPER placement attempts from unseen starting positions. The authored teacher completes 12/12; three controls complete 0/12 each. All twelve learned terminal poses pass settling checks, while seven fail the full native contact-history gates. Complete tapes, failed development candidates, an independent audit and an installable local runner preserve the reliability gap. This is privileged-state reference distillation for a fixed rigid block and two trained goals, not human-video learning, learned feedback or physical robot qualification.Independent camera and actuator corroboration v0.4https://research.dvidia.org/papers/sensor-corroboration-v04/https://research.dvidia.org/papers/sensor-corroboration-v04/Mon, 19 Oct 2026 12:00:00 GMT272 cases · 69/112 false completions prevented · 43 remain · 73/80 true completions abstained. A fresh eight-layout PiPER simulation benchmark adds separately processed RGB and drive checks to a local physical-goal verifier. The checks prevent 69 of 112 false completion readings while leaving 43 replay, identity, orientation, support and sampling blind spots; the conservative gate abstains on 73 of 80 baseline true completions, including all 64 deliberate sensor-fault controls. The measured CPU path is 22.1 ms median and 31.5 ms p95, excluding 200 ms acquisition. All raw channels and results are preserved; this is synthetic evidence, not hardware validation.A local verifier for physical completionhttps://research.dvidia.org/papers/local-goal-verifier-v03/https://research.dvidia.org/papers/local-goal-verifier-v03/Mon, 19 Oct 2026 12:00:00 GMTNative: 16/16 true complete, 0/48 false complete · forged readings: 8/8 false complete. An offline audit traces the seven missed Jev completions without claiming an internal model cause. A local deterministic verifier then checks timestamped pose, motion, support, release and uncertainty on 200 fresh cases. It recognizes all 16 native positives and rejects all 48 native negatives; 128 evidence faults yield 120 abstentions and 8 incomplete answers. All 8 consistent forged streams produce physical false completions, exposing the remaining sensor-trust problem. CPU-only p95 is 0.318 ms, excluding the required 200 ms history and sensing. Complete native inputs, every decision, independent review and explicit failure cohorts remain public; no new Jev calls or hardware qualification.Jev image-goal verification: the measured follow-uphttps://research.dvidia.org/papers/image-goal-jev-v02/https://research.dvidia.org/papers/image-goal-jev-v02/Mon, 19 Oct 2026 12:00:00 GMT64/64 valid responses · telemetry: 0/24 false complete, 1/8 true complete. The first measured run preserves all 64 requests from the image-goal v0.1 protocol. Jev returned valid structured responses for every trial and made no false-completion declaration, but recognized only 1/8 physically complete cases with privileged native telemetry and 0/8 with images alone. All four occluded complete scenes were called incomplete in both conditions. RGB abstention matched 8/16 ambiguous references. Median online latency was 1.776 seconds, p95 3.287 seconds. The release preserves every result, the independent review and the original unrun protocol. This small synthetic diagnostic does not qualify a robot completion gate, actuator loop or learned skill.Image-goal verification: native benchmark and Jev protocolhttps://research.dvidia.org/papers/image-goal-jev-v01/https://research.dvidia.org/papers/image-goal-jev-v01/Sun, 18 Oct 2026 12:00:00 GMT32 native fixtures · 4 identical-image pairs · 64 Jev trials not run. A frozen PiPER image-goal cohort crosses eight physical conditions with four layouts. Four moving/settled pairs have byte-identical images and different completion labels, exposing when one image cannot establish task completion. Native measurements verify 32 authored fixtures; a paired 64-request Jev protocol distinguishes RGB judgments from privileged simulator telemetry. The provider evaluation has not run because credentials were unavailable: accuracy, false-completion rates and service latency remain unmeasured. Every native state, image, frozen request hash and explicit unrun result is retained. No learned policy or hardware qualification is claimed.A visual goal still needs physical feedbackhttps://research.dvidia.org/papers/goal-images-physical-feedback/https://research.dvidia.org/papers/goal-images-physical-feedback/Sun, 18 Oct 2026 12:00:00 GMT14/14 exact passive replays · 249,502 native ticks · visual models unbenchmarked. A goal image specifies a destination; physical feedback verifies whether a robot gets there. We describe a bounded decision role for TypeSafe Jev, distinguish it from action-conditioned world-model planning, and propose an independently scored reaching experiment. A separate passive audit reproduces fourteen previously inspected rigid-box arm episodes exactly and records 249,502 native ticks. Millisecond contact gaps hidden between stored observations become visible, while task outcomes remain unchanged. Complete inputs, traces, source snapshots and a verified downloader preserve every case. This is measurement work, not a new policy benchmark, PiPER shoe result or hardware qualification.Give the gripper a fair testhttps://research.dvidia.org/papers/shoe-graspability-v03/https://research.dvidia.org/papers/shoe-graspability-v03/Wed, 14 Oct 2026 12:00:00 GMT24/24 valid · 30 g: 8/12 lifts · 300 g: 0/12 · no net gain from removing rake. A frozen 24-attempt PiPER diagnostic separates three rigid shapes, 30 g and 300 g loads, and mounted versus absent rake conditions. All attempts are valid; eight of twelve lightweight controls retain a direct lift while every 300 g attempt fails, including the small block. Removing the rake gains one narrow-shoe result and loses one block result, with no net completion gain. A verified settled reference, dense contact traces, unchanged drive fingerprint and independent scoring expose limits in the modeled grasp and force response. Recorded replays and runnable CPU code retain every failure. No autonomous tool removal, retrieval, learned skill or physical hardware qualification is claimed.From reaching a shoe to keeping holdhttps://research.dvidia.org/papers/continuous-shoe-grip-v02/https://research.dvidia.org/papers/continuous-shoe-grip-v02/Tue, 13 Oct 2026 12:00:00 GMT20/20 valid · 2/10 completed under both policies · 9 contact-loss alarms verified. A frozen twenty-attempt native MuJoCo comparison adds continuous acquisition checks and bounded lowering to the mounted-rake PiPER controller. Both policies complete the same 2/10 fresh cases: no gains or losses. Nine bilateral-contact-loss alarms follow a trace-verified native origin by 20–52 ms, median 30 ms; three acquisition failures are separate. Twelve supported lowering responses produce no additional completion. A non-finger-support gap remains unmonitored. Complete recorded trajectories, independent audit and development failures are public. Authored policies, synthetic contact and privileged observations establish no learned skill or physical robot qualification.Skill prerequisites: access, reach and recoveryhttps://research.dvidia.org/papers/skill-prerequisites-v01/https://research.dvidia.org/papers/skill-prerequisites-v01/Tue, 13 Oct 2026 12:00:00 GMT16/16 valid · 2 lid contacts avoided · 3/4 retrieval checkpoints · 0/4 retained tool lifts. A frozen sixteen-attempt native MuJoCo study tests hinged shoe-box access and a mounted rake for an out-of-workspace shoe. Guarded access avoids two baseline lid collisions while task completion stays at 2/4; a transient withdrawal adds 0.70 simulated seconds without being necessary in its matched baseline. The mounted rake reaches an intermediate retrieval checkpoint in 3/4 cases, but all subsequent retained lifts fail and the short-tool case is deferred. All native traces and failures remain public. The browser replays recorded states; fresh physics runs locally. Privileged observations, synthetic contact and authored policies establish no learned skill or hardware qualification.Live PiPER shoe workstation v0.1https://research.dvidia.org/papers/live-piper-shoe-workstation-v01/https://research.dvidia.org/papers/live-piper-shoe-workstation-v01/Tue, 13 Oct 2026 12:00:00 GMTLive browser physics · 14/16 targets · strict contact parity failed · shoe lifting unqualified. A six-axis PiPER and its coupled gripper respond to user targets while MuJoCo runs locally in a browser worker. The final native run retains 20 valid attempts: 14/16 target trials pass, both high-target repeats fail, and jaw, impulse, disabled-table and 600-second hold checks pass. A ten-minute headless Chromium run reports 51.389 render calls per second and a 0.9999795 simulation/wall-time ratio on the tested Apple M5 host; simulated movement response is 61.9 ms p95 across 61 commands. Render calls do not measure GPU presentation. Native/WASM numerical and loaded-contact-count checks pass, while strict raw contact counts differ in 13/1,500 states. All fifteen development shoe-lift attempts fail. Complete development, failure and timing records remain separate. This authored rigid-shoe environment establishes no learned skill, calibrated material model or physical robot qualification.Reusable shoe asset library v0.1https://research.dvidia.org/papers/shoe-asset-library-v01/https://research.dvidia.org/papers/shoe-asset-library-v01/Mon, 12 Oct 2026 12:00:00 GMTSix shoe proxies · three arms · support 27/27, disabled table 0/27 · manipulation unqualified. Six original MIT rigid shoe proxies compose with the pinned UR5e/Robotiq, PiPER and SO-101 models through versioned scene descriptors and offline native 3D previews. A frozen 54-attempt test records 27/27 supported pairs and 0/27 support successes when table collisions are disabled; all 54 gravity checks pass without numerical warnings. No loaded non-table shoe contacts appear in the 20 ms samples, which cannot rule out shorter transients. All sampled trajectories and 27,000 native-step durations remain available. On one Apple M5 host, 54 simulated seconds take 12.3616 wall seconds including fresh model construction; the slowest attempt takes 6.7047 seconds. Geometry and physical properties are authored assumptions. Loading, gravity and passive support do not establish shoe grasping, calibrated deformation, a learned policy or hardware transfer.PiPER placement reliability v0.5https://research.dvidia.org/papers/piper-reliability-v05/https://research.dvidia.org/papers/piper-reliability-v05/Mon, 12 Oct 2026 12:00:00 GMT23/36 revised grasp · 15/36 v0.4 · 8 gained, none lost · hardware unqualified. Three frozen fresh PiPER cohorts retain 252 numerically valid attempts and 144 failed negative controls. A fixed diagonal grasp with bounded pose planning completes 23/36 placements versus 15/36 for frozen v0.4 and 18/36 with second-lift recovery alone: eight gained completions and none lost. The fifteen shared successes are 0.772 simulated seconds slower on average. Three lift-only second probes qualify and finish their tasks; the full grasp candidate needs none. Twelve low-friction acquisitions and one later retention failure remain. Every trajectory, native task outcome and raw computation duration is retained, including 1,035 windows exceeding 20 ms and a 3.082-second outlier on a loaded host. Native mechanics and scoring remain unchanged. Separate development keeps 200 earlier attempts, including discarded regressions. Authored privileged simulation establishes no learned skill, real-time guarantee or physical qualification.PiPER contact recovery v0.4https://research.dvidia.org/papers/piper-contact-recovery-v04/https://research.dvidia.org/papers/piper-contact-recovery-v04/Sun, 11 Oct 2026 12:00:00 GMTAll modes 4/12 · two resumed tasks faster · controls 0/48 · hardware unqualified. A fresh frozen 84-attempt PiPER comparison completes 4/12 placements for v0.3 history, v0.4 carry-only and v0.4 full; all 48 matched open-jaw and hold controls fail. Two completed post-hold placements finish 2.440 and 2.640 simulated seconds sooner with unchanged 220/240 ms settling holds. Four low-friction failures reach refusal 2.88–2.90 seconds sooner without becoming successes. Fresh nominal-friction 60 g layouts expose two held-but-too-low resume failures in every mode; neither reaches speed restoration. Six earlier acquisition failures are traced to insufficient support under the pinned position servo and side pinch, not the actuator force cap. All native mechanics and outcome criteria remain unchanged. Complete trajectories and 84,000 raw computation windows are retained; none exceeds 20 ms on this host. Source-snapshotted development remains separate. Privileged authored simulation establishes no learned skill, true-slip accuracy or physical qualification.PiPER contact history v0.3https://research.dvidia.org/papers/piper-contact-history-v03/https://research.dvidia.org/papers/piper-contact-history-v03/Sun, 11 Oct 2026 12:00:00 GMTHistory 6/12 · baseline and probe-only 4/12 · controls 0/48 · hardware unqualified. A frozen paired PiPER experiment retains all 84 numerically valid attempts across twelve synthetic rigid-block scenes. History completes 6/12 placements versus 4/12 for both original motion and the identical acquisition probe without carry monitoring; all 48 separately matched open-jaw and hold controls fail. The two added completions occur at nominal friction with 60 g blocks. Four tasks complete after five closed holds/resumes, but only four of five resume windows meet the independent low absolute TCP-speed criterion. All six supported acquisition retries fail. Native gains, forces and outcome rules remain unchanged; requested movement reduction is distinct from actual speed, and two slowdown windows include early holds. Seventy-five of 84,000 computation windows exceed 20 ms. Complete sealed trajectories, original model notices, source-snapshotted development and pinned offline inspection preserve both success and failure. Privileged authored simulation establishes no learned skill, true-slip detector accuracy, shoe competence or physical qualification.PiPER grip validation and recoveryhttps://research.dvidia.org/papers/piper-grip-validation-and-recovery/https://research.dvidia.org/papers/piper-grip-validation-and-recovery/Sun, 11 Oct 2026 12:00:00 GMTModerate-friction regression · baseline 4/6, guarded 2/6 · hardware unqualified. A slow test lift and bounded supported retry add recorded acquisition decisions to the pinned PiPER contact pilot, but completion regresses: at moderate friction baseline completes 2/2 and guarded completes 0/2. Both complete 2/2 normal cases; neither completes the two low-friction cases. All 30 attempts remain numerically valid, including 18 failed controls and full-duration aborts. Four qualification decisions are followed by a brief independent following window, yet two tasks still fail; no successful recovery or overall improvement is demonstrated. Actual motion continues after hold requests, and 53/30,000 computation windows exceed 20 ms. Complete compressed receipts, original model notices and pinned offline inspection preserve the evidence. Authored privileged-state simulation establishes no detector accuracy, learned shoe skill, calibrated contact or physical robot qualification.Native grasp task benchmarkhttps://research.dvidia.org/papers/native-grasp-task-benchmark/https://research.dvidia.org/papers/native-grasp-task-benchmark/Sun, 11 Oct 2026 12:00:00 GMT6/6 simulated placements · controls 0/12 · real-time and hardware unqualified. A pinned six-axis arm and coupled gripper complete 6/6 frozen synthetic rigid-block placements, while matched recorded-arm/open-jaw and hold controls complete 0/12. Native contact must produce grasp, sustained lift, retained carry, release and settled placement with tool retreat. All 18 attempts and timing traces remain in the receipt; one placement takes 29.51 wall seconds for 20 simulated seconds and misses 251 of 1,000 computation windows, so real-time operation is not established. An offline 3D mesh replay exposes the recorded motion. The authored privileged-state baseline provides a benchmark for a future compact DVIDIA action policy, not a learned footage-to-action model, validated shoe simulation or physical robot qualification.Hardware matched robot simulationhttps://research.dvidia.org/papers/hardware-matched-robot-simulation/https://research.dvidia.org/papers/hardware-matched-robot-simulation/Sun, 11 Oct 2026 12:00:00 GMTPinned CPU mechanics · PiPER contact pilot · physical fidelity unmeasured. A separate UR5e/Robotiq reference preserves pinned geometry, inertias, moving-link collision proxies and coupled gripper structure: fourteen modeled coordinates and seven commands. Eighteen deterministic unloaded motion/closure runs are numerically valid; at a 1 ms step, CPU physics throughput measures 40.80–47.29 simulated seconds per measured stepping wall second on one host, excluding loading, sensing, rendering, policy and real actuation. Adjacent timestep comparisons and a matched static overlap expose numerical sensitivity and collision participation. A pure-Python paired trace scorer measures joint, tool and jaw mismatch without automatic qualification. Left/right five-finger references are separately audited; LeRobot collection with an original DVIDIA policy is proposed. Serial calibration, physical response, shoe task learning and bimanual execution remain unmeasured. An offline 3D mesh replay now exposes the recorded arm/gripper poses without rerunning physics. PiPER and SO-101 now have pinned CPU profiles with 6/6 valid unloaded numerical attempts each and explicit native command units. PiPER also has a frozen authored rigid-block contact pilot: nominal placement 2/2, low-friction placement 0/1, open-jaw 0/3 and hold 0/3, with all nine attempts numerically valid. One of 9,000 computation windows exceeds 20 ms; hardware real-time response is unqualified. SO-101 object tasks, learned policies and physical qualification remain open. YAM, OpenArm, ARX X5 and PAROL6 remain upcoming.DVIDIA Observation Lab pilothttps://research.dvidia.org/papers/dvidia-observation-lab/https://research.dvidia.org/papers/dvidia-observation-lab/Sun, 11 Oct 2026 12:00:00 GMTLocal review release · motion and actuator source pilots · hardware unqualified. DVIDIA Observation Lab supplies local object/palm proposals, reviewed timeline revisions and separate causal image-motion forecasts; real annotation and motion accuracy remain unmeasured. A new CPU pilot learns next-20-ms joint/jaw deltas from current native state and issued commands. Twelve seeded groups retain 36 recordings with 8/2/2 train/development/test grouping; 4,553 test transitions are not independent robot trials. The 6,438-byte action model computes at 0.207 ms p95 on one host and improves RMSE, while jaw MAE/worst error regress against persistence and an initial opening is substantially overpredicted. These are privileged simulator measurements and model-only timings. A proposed shoe capture plan has no footage or executed shoe benchmark. The v0.1 release and hosted synthetic review demo remain separate; no shoe policy, physical adapter, validated shoe physics or hardware qualification is established.Robotics research pipelinehttps://research.dvidia.org/papers/robotics-research-pipeline/https://research.dvidia.org/papers/robotics-research-pipeline/Fri, 09 Oct 2026 12:00:00 GMTArchitecture proposal · current simulation alpha · hardware unqualified. An updated architecture connects immutable Skillspace recordings, reviewed task-label proposals and synchronized measured telemetry to grouped datasets, separate visual and motor learners, and a proposed local state supervisor. The implemented CPU baseline retains authored phases, gripper decisions and recovery around its learned joint-delta head. New annotation and supervisor pilots, fresh frozen confirmation, physics calibration and compatible skill-installation gates define the next milestones. Automatic labeling, learned supervision, human-video action recovery and physical arm qualification remain unestablished; this publication adds no new training or hardware result.Skillspace footage training pipelinehttps://research.dvidia.org/papers/skillspace-footage-training/https://research.dvidia.org/papers/skillspace-footage-training/Thu, 08 Oct 2026 12:00:00 GMTPublic CPU package · one native scene qualified · simulation-only. The public Dvidia-Inference/dvidia-training repository provides a standalone CPU pipeline that audits Skillspace media, freezes grouped splits and exports inspectable artifacts. Ten synthetic six-second clips train a visual predictor with a 0.5106-second median pipeline time across three repeats on one macOS host. Aligned native simulator telemetry supplies 4,029 training labels for a learned joint-delta head, evaluated on one held-out recording. Its capsule passes native placement qualification in that recording's exact scene; identical joint targets with open jaws fail. Installation and hardware guides accompany the package. Human-video action recovery, shoe organization, demonstration sufficiency, independent scene generalization and physical robot competence remain unestablished.Skill capsule acquisition and optimizationhttps://research.dvidia.org/papers/skill-capsule-acquisition/https://research.dvidia.org/papers/skill-capsule-acquisition/Wed, 07 Oct 2026 12:00:00 GMTSimulation-only · teacher/student 7/8 · no overall speedup. A compact learned CPU movement head replaces joint-target computation inside an authored placement supervisor. Teacher and student both pass 7/8 in a new frozen evaluation, while episode wall time increases from 6.273 to 6.360 seconds. A 39,845-byte JSON capsule imports into the local lab, checks its exact runtime and qualifies the current scene through native placement and a recorded-arm/open-jaw control. Scene changes require a fresh test. One known scene and repeat pass native network denial; perception, complete task learning, other embodiments and hardware remain unqualified.Skillspace arm benchmarks and evolutionhttps://research.dvidia.org/papers/skillspace-arm-benchmarks/https://research.dvidia.org/papers/skillspace-arm-benchmarks/Wed, 07 Oct 2026 12:00:00 GMTMixed held-out result · 30/32 baseline, 29/32 updated. Two controller revisions improve development completion from 14/20 to 17/20, but the frozen paired test declines from 30/32 to 29/32: both pass all 24 normal-strength cases while actuator-stress completion falls from 6/8 to 5/8. Instrumented CPU throughput also declines. We publish the regression, unchanged contact and success rules, exact controller snapshots, richer physical inputs, runtime-bound installation and a proposed staged Skillspace contract. Contact forces and penetration remain uncalibrated; learned perception and physical qualification are open.Installing a Skillspace on a simulated armhttps://research.dvidia.org/papers/skillspace-arm-installation/https://research.dvidia.org/papers/skillspace-arm-installation/Wed, 07 Oct 2026 12:00:00 GMTPublic skill link to authored adapter · 6/6 scene placements. The first link-to-arm product path installs an existing DVIDIA placement envelope, binds a local authored controller to one simulated arm, and reuses it in independently grounded scenes. Native jaw contacts carry a free box; six layouts complete, while no-motion and identical joint commands with open jaws each complete zero of six. Installed execution passes process-level offline checks. Task metadata contains no learned policy; perception, cup geometry, full arm collisions and physical qualification remain open.Offline cable training environmenthttps://research.dvidia.org/papers/offline-cable-training-environment/https://research.dvidia.org/papers/offline-cable-training-environment/Wed, 07 Oct 2026 12:00:00 GMTCPU simulation-only · Controller search and frozen evaluation executed. The first runnable product loop combines an anchored cable environment, local controller-parameter search, frozen evaluation, offline replay and a manifest-backed skill pack. The selected controller and a fixed feedback baseline both completed 10/10 held-out simulated episodes; zero-force and release controls completed 0/10. A native macOS network-denial run completed 3/3 additional episodes. Ideal endpoint attachment and privileged state bound the result; physical rope calibration, gripping, sensors and GPU throughput remain untested.Fast accurate robot training simulationhttps://research.dvidia.org/papers/fast-accurate-robot-training-simulation/https://research.dvidia.org/papers/fast-accurate-robot-training-simulation/Wed, 07 Oct 2026 12:00:00 GMT54 CPU fixture trials · GPU and physical transfer untested. A design for an offline, self-hosted training environment that turns demonstrations into calibrated practice and qualified robot skills. We examine contact, deformation, sensor feedback and the performance–accuracy tradeoff, then report 54 deterministic native MuJoCo CPU fixture trials. Fine endpoint agreement did not establish accurate contact transients, and coarse timesteps missed collisions. Minimum GPU memory, rope accuracy and physical transfer remain unmeasured.Grok robotics research on Xhttps://research.dvidia.org/papers/grok-robotics-research-on-x/https://research.dvidia.org/papers/grok-robotics-research-on-x/Wed, 07 Oct 2026 12:00:00 GMTPrimary sources checked · Experiments proposed. Three Grok 4.7 research passes surface observation-to-action, synthetic-practice and reusable-repair ideas. Independent primary-source checks compare those findings with our thesis: observation supplies goals and state, language retrieves a skill, spatial expectations guide attention, and executable feedback with evaluation establishes competence. X posts remain discovery leads. No universal demonstration count or arbitrary-robot transfer follows from the reviewed results.Spatially grounded downloadable robot skillshttps://research.dvidia.org/papers/spatially-grounded-downloadable-robot-skills/https://research.dvidia.org/papers/spatially-grounded-downloadable-robot-skills/Wed, 07 Oct 2026 12:00:00 GMTSource review · DVIDIA experiments not run. A synthesis of learning from human demonstrations, language-conditioned spatial attention, simulation expansion and robot-specific skill delivery. Spatial priors can accelerate inspection but do not determine contact state or control. The proposed downloadable skill packages a controller, initiation and termination conditions, compatible sensing and hardware, recovery behavior and measured evidence. Separate experiments would test attention, data coverage, feedback and transfer.Clef and grounded video observationshttps://research.dvidia.org/papers/clef-and-grounded-video-observations/https://research.dvidia.org/papers/clef-and-grounded-video-observations/Tue, 06 Oct 2026 12:00:00 GMTFive licensed excerpts · Accuracy evaluation outstanding. A preregistered image-API feasibility run processed five licensed three-second POV excerpts using four timestamped frames, a Gemma caption and a Clef-flash check. All five completed at a price-derived inference estimate of $0.00140634. Post-run inspection found an object error accepted by the checker and omitted state changes. API and schema compatibility were demonstrated; annotation accuracy and whole-task success were not.From Skillspace to robot skillhttps://research.dvidia.org/papers/from-skillspace-to-robot-skill/https://research.dvidia.org/papers/from-skillspace-to-robot-skill/Tue, 06 Oct 2026 12:00:00 GMTGraduation and robot training not run. An Argus review and a proposal to connect original recordings, reviewed episodes, immutable datasets, training runs, held-out evaluation and robot-tested releases. The review separates video annotation from action supervision and defines compatibility beyond a robot’s degree-of-freedom count. A proposed 40-episode annotation pilot and evidence-based graduation criteria remain unrun.Turn physical footage into testable evidencehttps://research.dvidia.org/papers/physical-skill-research-agenda/https://research.dvidia.org/papers/physical-skill-research-agenda/Tue, 06 Oct 2026 12:00:00 GMTFour baseline protocols proposed. A research agenda covering visible capture evidence, relations over time, evaluation provenance and physical grounding. Each question has a bounded protocol and a failure condition. The aim is to measure whether structure and physical signals improve a defined task before scaling collection. Strategic opinions, author-reported results and our untested proposals are kept distinct. diff --git a/docs/index.html b/docs/index.html index 2fb0466..a797b5a 100644 --- a/docs/index.html +++ b/docs/index.html @@ -1,4 +1,4 @@ Research journal · DVIDIA Research -

An open research notebook

From observation
to physical skill.

We study how demonstrations become usable evidence, how simulation becomes useful practice, and when a robot skill earns a release.

Published by DVIDIA · hello@dvidia.org

Publications

31 papers and working notes

· Learned jaw adapter with matched blind controls and retained negative studies

Learning grip feedback with a frozen PiPER arm reference

After five seven-command residual candidates each failed all 16 development cases, a separate 25-node learned jaw adapter preserves the frozen six-axis PiPER arm references. Trained on 64 synthetic teacher episodes and 64,000 causal labels, it completes 6/24 frozen confirmation cases, versus 8/24 for its matched blind control, 5/24 feedforward, 24/24 authored teacher and 0/24 open jaws. Feedback succeeds in 0/12 interrupted cases; 14/24 terminal-only passes conceal strict task failures. All 120 confirmation attempts are valid, and an independent numeric audit reconstructs 360 retained attempts. CPU inference p95 is 0.501 ms; this establishes no live-feedback advantage, recovered grip, human-video learning or hardware qualification.

Jaw feedback 6/24 · blind 8/24 · feedforward 5/24 · teacher 24/24 · no demonstrated feedback advantage

· Learned feedforward primitive from synthetic demonstrations

Learning PiPER block placement from synthetic demonstrations

A compact feedforward joint-and-jaw primitive trained on 32 successful synthetic demonstrations completes 5/12 frozen PiPER placement attempts from unseen starting positions. The authored teacher completes 12/12; three controls complete 0/12 each. All twelve learned terminal poses pass settling checks, while seven fail the full native contact-history gates. Complete tapes, failed development candidates, an independent audit and an installable local runner preserve the reliability gap. This is privileged-state reference distillation for a fixed rigid block and two trained goals, not human-video learning, learned feedback or physical robot qualification.

Learned 5/12 · teacher 12/12 · three controls 0/12 each · simulation only

· Paired synthetic benchmark for camera and drive evidence

Independent camera and actuator corroboration v0.4

A fresh eight-layout PiPER simulation benchmark adds separately processed RGB and drive checks to a local physical-goal verifier. The checks prevent 69 of 112 false completion readings while leaving 43 replay, identity, orientation, support and sampling blind spots; the conservative gate abstains on 73 of 80 baseline true completions, including all 64 deliberate sensor-fault controls. The measured CPU path is 22.1 ms median and 31.5 ms p95, excluding 200 ms acquisition. All raw channels and results are preserved; this is synthetic evidence, not hardware validation.

272 cases · 69/112 false completions prevented · 43 remain · 73/80 true completions abstained

· Local CPU verifier with native, evidence-fault and forgery diagnostics

A local verifier for physical completion

An offline audit traces the seven missed Jev completions without claiming an internal model cause. A local deterministic verifier then checks timestamped pose, motion, support, release and uncertainty on 200 fresh cases. It recognizes all 16 native positives and rejects all 48 native negatives; 128 evidence faults yield 120 abstentions and 8 incomplete answers. All 8 consistent forged streams produce physical false completions, exposing the remaining sensor-trust problem. CPU-only p95 is 0.318 ms, excluding the required 200 ms history and sensing. Complete native inputs, every decision, independent review and explicit failure cohorts remain public; no new Jev calls or hardware qualification.

Native: 16/16 true complete, 0/48 false complete · forged readings: 8/8 false complete

· Frozen hosted-model evaluation with native physical references

Jev image-goal verification: the measured follow-up

The first measured run preserves all 64 requests from the image-goal v0.1 protocol. Jev returned valid structured responses for every trial and made no false-completion declaration, but recognized only 1/8 physically complete cases with privileged native telemetry and 0/8 with images alone. All four occluded complete scenes were called incomplete in both conditions. RGB abstention matched 8/16 ambiguous references. Median online latency was 1.776 seconds, p95 3.287 seconds. The release preserves every result, the independent review and the original unrun protocol. This small synthetic diagnostic does not qualify a robot completion gate, actuator loop or learned skill.

64/64 valid responses · telemetry: 0/24 false complete, 1/8 true complete

· Native diagnostic benchmark and frozen hosted-model protocol

Image-goal verification: native benchmark and Jev protocol

A frozen PiPER image-goal cohort crosses eight physical conditions with four layouts. Four moving/settled pairs have byte-identical images and different completion labels, exposing when one image cannot establish task completion. Native measurements verify 32 authored fixtures; a paired 64-request Jev protocol distinguishes RGB judgments from privileged simulator telemetry. The provider evaluation has not run because credentials were unavailable: accuracy, false-completion rates and service latency remain unmeasured. Every native state, image, frozen request hash and explicit unrun result is retained. No learned policy or hardware qualification is claimed.

32 native fixtures · 4 identical-image pairs · 64 Jev trials not run

· Research direction and passive native contact audit

A visual goal still needs physical feedback

A goal image specifies a destination; physical feedback verifies whether a robot gets there. We describe a bounded decision role for TypeSafe Jev, distinguish it from action-conditioned world-model planning, and propose an independently scored reaching experiment. A separate passive audit reproduces fourteen previously inspected rigid-box arm episodes exactly and records 249,502 native ticks. Millisecond contact gaps hidden between stored observations become visible, while task outcomes remain unchanged. Complete inputs, traces, source snapshots and a verified downloader preserve every case. This is measurement work, not a new policy benchmark, PiPER shoe result or hardware qualification.

14/14 exact passive replays · 249,502 native ticks · visual models unbenchmarked

· Frozen shape, load and attached-tool diagnostic

Give the gripper a fair test

A frozen 24-attempt PiPER diagnostic separates three rigid shapes, 30 g and 300 g loads, and mounted versus absent rake conditions. All attempts are valid; eight of twelve lightweight controls retain a direct lift while every 300 g attempt fails, including the small block. Removing the rake gains one narrow-shoe result and loses one block result, with no net completion gain. A verified settled reference, dense contact traces, unchanged drive fingerprint and independent scoring expose limits in the modeled grasp and force response. Recorded replays and runnable CPU code retain every failure. No autonomous tool removal, retrieval, learned skill or physical hardware qualification is claimed.

24/24 valid · 30 g: 8/12 lifts · 300 g: 0/12 · no net gain from removing rake

· Frozen native comparison with independent contact audit

From reaching a shoe to keeping hold

A frozen twenty-attempt native MuJoCo comparison adds continuous acquisition checks and bounded lowering to the mounted-rake PiPER controller. Both policies complete the same 2/10 fresh cases: no gains or losses. Nine bilateral-contact-loss alarms follow a trace-verified native origin by 20–52 ms, median 30 ms; three acquisition failures are separate. Twelve supported lowering responses produce no additional completion. A non-finger-support gap remains unmonitored. Complete recorded trajectories, independent audit and development failures are public. Authored policies, synthetic contact and privileged observations establish no learned skill or physical robot qualification.

20/20 valid · 2/10 completed under both policies · 9 contact-loss alarms verified

· Two native environments with authored policies and recorded benchmark evidence

Skill prerequisites: access, reach and recovery

A frozen sixteen-attempt native MuJoCo study tests hinged shoe-box access and a mounted rake for an out-of-workspace shoe. Guarded access avoids two baseline lid collisions while task completion stays at 2/4; a transient withdrawal adds 0.70 simulated seconds without being necessary in its matched baseline. The mounted rake reaches an intermediate retrieval checkpoint in 3/4 cases, but all subsequent retained lifts fail and the short-tool case is deferred. All native traces and failures remain public. The browser replays recorded states; fresh physics runs locally. Privileged observations, synthetic contact and authored policies establish no learned skill or hardware qualification.

16/16 valid · 2 lid contacts avoided · 3/4 retrieval checkpoints · 0/4 retained tool lifts

· Live browser-native simulation with authored control and separate benchmark evidence

Live PiPER shoe workstation v0.1

A six-axis PiPER and its coupled gripper respond to user targets while MuJoCo runs locally in a browser worker. The final native run retains 20 valid attempts: 14/16 target trials pass, both high-target repeats fail, and jaw, impulse, disabled-table and 600-second hold checks pass. A ten-minute headless Chromium run reports 51.389 render calls per second and a 0.9999795 simulation/wall-time ratio on the tested Apple M5 host; simulated movement response is 61.9 ms p95 across 61 commands. Render calls do not measure GPU presentation. Native/WASM numerical and loaded-contact-count checks pass, while strict raw contact counts differ in 13/1,500 states. All fifteen development shoe-lift attempts fail. Complete development, failure and timing records remain separate. This authored rigid-shoe environment establishes no learned skill, calibrated material model or physical robot qualification.

Live browser physics · 14/16 targets · strict contact parity failed · shoe lifting unqualified

· Versioned native scene assets and frozen passive-contact benchmark

Reusable shoe asset library v0.1

Six original MIT rigid shoe proxies compose with the pinned UR5e/Robotiq, PiPER and SO-101 models through versioned scene descriptors and offline native 3D previews. A frozen 54-attempt test records 27/27 supported pairs and 0/27 support successes when table collisions are disabled; all 54 gravity checks pass without numerical warnings. No loaded non-table shoe contacts appear in the 20 ms samples, which cannot rule out shorter transients. All sampled trajectories and 27,000 native-step durations remain available. On one Apple M5 host, 54 simulated seconds take 12.3616 wall seconds including fresh model construction; the slowest attempt takes 6.7047 seconds. Geometry and physical properties are authored assumptions. Loading, gravity and passive support do not establish shoe grasping, calibrated deformation, a learned policy or hardware transfer.

Six shoe proxies · three arms · support 27/27, disabled table 0/27 · manipulation unqualified

· Frozen paired reliability experiment with complete native evidence

PiPER placement reliability v0.5

Three frozen fresh PiPER cohorts retain 252 numerically valid attempts and 144 failed negative controls. A fixed diagonal grasp with bounded pose planning completes 23/36 placements versus 15/36 for frozen v0.4 and 18/36 with second-lift recovery alone: eight gained completions and none lost. The fifteen shared successes are 0.772 simulated seconds slower on average. Three lift-only second probes qualify and finish their tasks; the full grasp candidate needs none. Twelve low-friction acquisitions and one later retention failure remain. Every trajectory, native task outcome and raw computation duration is retained, including 1,035 windows exceeding 20 ms and a 3.082-second outlier on a loaded host. Native mechanics and scoring remain unchanged. Separate development keeps 200 earlier attempts, including discarded regressions. Authored privileged simulation establishes no learned skill, real-time guarantee or physical qualification.

23/36 revised grasp · 15/36 v0.4 · 8 gained, none lost · hardware unqualified

· Paired carry-recovery and supported-acquisition experiment with complete confirmation receipt

PiPER contact recovery v0.4

A fresh frozen 84-attempt PiPER comparison completes 4/12 placements for v0.3 history, v0.4 carry-only and v0.4 full; all 48 matched open-jaw and hold controls fail. Two completed post-hold placements finish 2.440 and 2.640 simulated seconds sooner with unchanged 220/240 ms settling holds. Four low-friction failures reach refusal 2.88–2.90 seconds sooner without becoming successes. Fresh nominal-friction 60 g layouts expose two held-but-too-low resume failures in every mode; neither reaches speed restoration. Six earlier acquisition failures are traced to insufficient support under the pinned position servo and side pinch, not the actuator force cap. All native mechanics and outcome criteria remain unchanged. Complete trajectories and 84,000 raw computation windows are retained; none exceeds 20 ms on this host. Source-snapshotted development remains separate. Privileged authored simulation establishes no learned skill, true-slip accuracy or physical qualification.

All modes 4/12 · two resumed tasks faster · controls 0/48 · hardware unqualified

· Paired probe-only and contact-history experiment with complete confirmation receipt

PiPER contact history v0.3

A frozen paired PiPER experiment retains all 84 numerically valid attempts across twelve synthetic rigid-block scenes. History completes 6/12 placements versus 4/12 for both original motion and the identical acquisition probe without carry monitoring; all 48 separately matched open-jaw and hold controls fail. The two added completions occur at nominal friction with 60 g blocks. Four tasks complete after five closed holds/resumes, but only four of five resume windows meet the independent low absolute TCP-speed criterion. All six supported acquisition retries fail. Native gains, forces and outcome rules remain unchanged; requested movement reduction is distinct from actual speed, and two slowdown windows include early holds. Seventy-five of 84,000 computation windows exceed 20 ms. Complete sealed trajectories, original model notices, source-snapshotted development and pinned offline inspection preserve both success and failure. Privileged authored simulation establishes no learned skill, true-slip detector accuracy, shoe competence or physical qualification.

History 6/12 · baseline and probe-only 4/12 · controls 0/48 · hardware unqualified

· Paired authored controller experiment and complete confirmation receipt

PiPER grip validation and recovery

A slow test lift and bounded supported retry add recorded acquisition decisions to the pinned PiPER contact pilot, but completion regresses: at moderate friction baseline completes 2/2 and guarded completes 0/2. Both complete 2/2 normal cases; neither completes the two low-friction cases. All 30 attempts remain numerically valid, including 18 failed controls and full-duration aborts. Four qualification decisions are followed by a brief independent following window, yet two tasks still fail; no successful recovery or overall improvement is demonstrated. Actual motion continues after hold requests, and 53/30,000 computation windows exceed 20 ms. Complete compressed receipts, original model notices and pinned offline inspection preserve the evidence. Authored privileged-state simulation establishes no detector accuracy, learned shoe skill, calibrated contact or physical robot qualification.

Moderate-friction regression · baseline 4/6, guarded 2/6 · hardware unqualified

· Native contact task implementation and authored baseline

Native grasp task benchmark

A pinned six-axis arm and coupled gripper complete 6/6 frozen synthetic rigid-block placements, while matched recorded-arm/open-jaw and hold controls complete 0/12. Native contact must produce grasp, sustained lift, retained carry, release and settled placement with tool retreat. All 18 attempts and timing traces remain in the receipt; one placement takes 29.51 wall seconds for 20 simulated seconds and misses 251 of 1,000 computation windows, so real-time operation is not established. An offline 3D mesh replay exposes the recorded motion. The authored privileged-state baseline provides a benchmark for a future compact DVIDIA action policy, not a learned footage-to-action model, validated shoe simulation or physical robot qualification.

6/6 simulated placements · controls 0/12 · real-time and hardware unqualified

· Pinned mechanics implementation and CPU numerical benchmark

Hardware matched robot simulation

A separate UR5e/Robotiq reference preserves pinned geometry, inertias, moving-link collision proxies and coupled gripper structure: fourteen modeled coordinates and seven commands. Eighteen deterministic unloaded motion/closure runs are numerically valid; at a 1 ms step, CPU physics throughput measures 40.80–47.29 simulated seconds per measured stepping wall second on one host, excluding loading, sensing, rendering, policy and real actuation. Adjacent timestep comparisons and a matched static overlap expose numerical sensitivity and collision participation. A pure-Python paired trace scorer measures joint, tool and jaw mismatch without automatic qualification. Left/right five-finger references are separately audited; LeRobot collection with an original DVIDIA policy is proposed. Serial calibration, physical response, shoe task learning and bimanual execution remain unmeasured. An offline 3D mesh replay now exposes the recorded arm/gripper poses without rerunning physics. PiPER and SO-101 now have pinned CPU profiles with 6/6 valid unloaded numerical attempts each and explicit native command units. PiPER also has a frozen authored rigid-block contact pilot: nominal placement 2/2, low-friction placement 0/1, open-jaw 0/3 and hold 0/3, with all nine attempts numerically valid. One of 9,000 computation windows exceeds 20 ms; hardware real-time response is unqualified. SO-101 object tasks, learned policies and physical qualification remain open. YAM, OpenArm, ARX X5 and PAROL6 remain upcoming.

Pinned CPU mechanics · PiPER contact pilot · physical fidelity unmeasured

· Local observation workflow and engineering feasibility pilot

DVIDIA Observation Lab pilot

DVIDIA Observation Lab supplies local object/palm proposals, reviewed timeline revisions and separate causal image-motion forecasts; real annotation and motion accuracy remain unmeasured. A new CPU pilot learns next-20-ms joint/jaw deltas from current native state and issued commands. Twelve seeded groups retain 36 recordings with 8/2/2 train/development/test grouping; 4,553 test transitions are not independent robot trials. The 6,438-byte action model computes at 0.207 ms p95 on one host and improves RMSE, while jaw MAE/worst error regress against persistence and an initial opening is substantially overpredicted. These are privileged simulator measurements and model-only timings. A proposed shoe capture plan has no footage or executed shoe benchmark. The v0.1 release and hosted synthetic review demo remain separate; no shoe policy, physical adapter, validated shoe physics or hardware qualification is established.

Local review release · motion and actuator source pilots · hardware unqualified

· Architecture proposal and staged validation roadmap

Robotics research pipeline

An updated architecture connects immutable Skillspace recordings, reviewed task-label proposals and synchronized measured telemetry to grouped datasets, separate visual and motor learners, and a proposed local state supervisor. The implemented CPU baseline retains authored phases, gripper decisions and recovery around its learned joint-delta head. New annotation and supervisor pilots, fresh frozen confirmation, physics calibration and compatible skill-installation gates define the next milestones. Automatic labeling, learned supervision, human-video action recovery and physical arm qualification remain unestablished; this publication adds no new training or hardware result.

Architecture proposal · current simulation alpha · hardware unqualified

· Product implementation and measured synthetic pilot

Skillspace footage training pipeline

The public Dvidia-Inference/dvidia-training repository provides a standalone CPU pipeline that audits Skillspace media, freezes grouped splits and exports inspectable artifacts. Ten synthetic six-second clips train a visual predictor with a 0.5106-second median pipeline time across three repeats on one macOS host. Aligned native simulator telemetry supplies 4,029 training labels for a learned joint-delta head, evaluated on one held-out recording. Its capsule passes native placement qualification in that recording's exact scene; identical joint targets with open jaws fail. Installation and hardware guides accompany the package. Human-video action recovery, shoe organization, demonstration sufficiency, independent scene generalization and physical robot competence remain unestablished.

Public CPU package · one native scene qualified · simulation-only

· Learned movement artifact and installation prototype

Skill capsule acquisition and optimization

A compact learned CPU movement head replaces joint-target computation inside an authored placement supervisor. Teacher and student both pass 7/8 in a new frozen evaluation, while episode wall time increases from 6.273 to 6.360 seconds. A 39,845-byte JSON capsule imports into the local lab, checks its exact runtime and qualifies the current scene through native placement and a recorded-arm/open-jaw control. Scene changes require a fresh test. One known scene and repeat pass native network denial; perception, complete task learning, other embodiments and hardware remain unqualified.

Simulation-only · teacher/student 7/8 · no overall speedup

· Paired simulation benchmark and product assessment

Skillspace arm benchmarks and evolution

Two controller revisions improve development completion from 14/20 to 17/20, but the frozen paired test declines from 30/32 to 29/32: both pass all 24 normal-strength cases while actuator-stress completion falls from 6/8 to 5/8. Instrumented CPU throughput also declines. We publish the regression, unchanged contact and success rules, exact controller snapshots, richer physical inputs, runtime-bound installation and a proposed staged Skillspace contract. Contact forces and penetration remain uncalibrated; learned perception and physical qualification are open.

Mixed held-out result · 30/32 baseline, 29/32 updated

· Product prototype and native contact qualification

Installing a Skillspace on a simulated arm

The first link-to-arm product path installs an existing DVIDIA placement envelope, binds a local authored controller to one simulated arm, and reuses it in independently grounded scenes. Native jaw contacts carry a free box; six layouts complete, while no-motion and identical joint commands with open jaws each complete zero of six. Installed execution passes process-level offline checks. Task metadata contains no learned policy; perception, cup geometry, full arm collisions and physical qualification remain open.

Public skill link to authored adapter · 6/6 scene placements

· Product prototype and local qualification

Offline cable training environment

The first runnable product loop combines an anchored cable environment, local controller-parameter search, frozen evaluation, offline replay and a manifest-backed skill pack. The selected controller and a fixed feedback baseline both completed 10/10 held-out simulated episodes; zero-force and release controls completed 0/10. A native macOS network-denial run completed 3/3 additional episodes. Ideal endpoint attachment and privileged state bound the result; physical rope calibration, gripping, sensors and GPU throughput remain untested.

CPU simulation-only · Controller search and frozen evaluation executed

· Decision report and local experiment

Fast accurate robot training simulation

A design for an offline, self-hosted training environment that turns demonstrations into calibrated practice and qualified robot skills. We examine contact, deformation, sensor feedback and the performance–accuracy tradeoff, then report 54 deterministic native MuJoCo CPU fixture trials. Fine endpoint agreement did not establish accurate contact transients, and coarse timesteps missed collisions. Minimum GPU memory, rope accuracy and physical transfer remain unmeasured.

54 CPU fixture trials · GPU and physical transfer untested

· Source review and thesis comparison

Grok robotics research on X

Three Grok 4.7 research passes surface observation-to-action, synthetic-practice and reusable-repair ideas. Independent primary-source checks compare those findings with our thesis: observation supplies goals and state, language retrieves a skill, spatial expectations guide attention, and executable feedback with evaluation establishes competence. X posts remain discovery leads. No universal demonstration count or arbitrary-robot transfer follows from the reviewed results.

Primary sources checked · Experiments proposed

· Research synthesis and proposed experiments

Spatially grounded downloadable robot skills

A synthesis of learning from human demonstrations, language-conditioned spatial attention, simulation expansion and robot-specific skill delivery. Spatial priors can accelerate inspection but do not determine contact state or control. The proposed downloadable skill packages a controller, initiation and termination conditions, compatible sensing and hardware, recovery behavior and measured evidence. Separate experiments would test attention, data coverage, feedback and transfer.

Source review · DVIDIA experiments not run

· Engineering feasibility pilot

Clef and grounded video observations

A preregistered image-API feasibility run processed five licensed three-second POV excerpts using four timestamped frames, a Gemma caption and a Clef-flash check. All five completed at a price-derived inference estimate of $0.00140634. Post-run inspection found an object error accepted by the checker and omitted state changes. API and schema compatibility were demonstrated; annotation accuracy and whole-task success were not.

Five licensed excerpts · Accuracy evaluation outstanding

· Source review and implementation proposal

From Skillspace to robot skill

An Argus review and a proposal to connect original recordings, reviewed episodes, immutable datasets, training runs, held-out evaluation and robot-tested releases. The review separates video annotation from action supervision and defines compatibility beyond a robot’s degree-of-freedom count. A proposed 40-episode annotation pilot and evidence-based graduation criteria remain unrun.

Graduation and robot training not run

· Research agenda

Turn physical footage into testable evidence

A research agenda covering visible capture evidence, relations over time, evaluation provenance and physical grounding. Each question has a bounded protocol and a failure condition. The aim is to measure whether structure and physical signals improve a defined task before scaling collection. Strategic opinions, author-reported results and our untested proposals are kept distinct.

Four baseline protocols proposed

Research footprints

  1. Learn a first PiPER placement primitive from demonstrations

    A compact feedforward model trained on 32 synthetic teacher demonstrations completes 5/12 frozen placement attempts from unseen starts. The teacher completes 12/12; three controls complete none. All twelve learned final poses settle near their targets, but seven fail full contact-history gates. The recorded viewer, complete evidence and installable local runner preserve the reliability gap. Learned feedback, human-video learning and physical transfer remain open.

    Read the learned-placement study · Inspect all twelve learned attempts
  2. Measure Jev against physical completion

    All 64 frozen requests returned valid responses. No false completion was observed, but images plus telemetry recognized only 1/8 completed states, and images alone recognized 0/8. Occluded finished scenes failed in both conditions. Median hosted latency was 1.776 seconds. The original protocol remains unchanged; all measured cases and independent review are public.

    Read the measured follow-up · Inspect every judgment
  3. Make image-goal uncertainty testable

    Thirty-two native PiPER fixtures include four moving/settled pairs with identical images and different physical completion labels. The frozen 64-request Jev protocol compares RGB with privileged simulator telemetry. Jev has not been run; all provider trials remain explicitly unrun, with accuracy and service latency unmeasured. Native states, images, source hashes and physical labels are public.

    Read the benchmark and protocol · Compare every current and goal image
  4. Connect visual goals to measured physical feedback

    Fourteen passive rigid-box arm replays reproduce every original sampled state and outcome, recording 249,502 native ticks. Millisecond contact gaps appear between stored observations; a successful comparator also has gaps. The separate architecture note evaluates Jev as a candidate decision layer and defines a goal-image reaching test. No visual model is integrated and no controller is promoted.

    Read the architecture and audit · Download verified evidence tool
  5. Separate object difficulty from the gripper and attached tool

    A frozen direct-grasp diagnostic retains 24 valid attempts. Lightweight controls complete 8/12 lifts; every 300 g condition fails, including the small block. Removing the rake gains one narrow-shoe result and loses one block result. Relative lift, settled references and dense contact traces make the modeled drive and geometry limits visible. These are diagnostic controls, not a shoe retrieval rate or hardware qualification.

    Read the graspability study · Inspect every recorded attempt
  6. Measure contact-loss response without claiming a better grasp

    Twenty valid fresh attempts complete the same 2/10 cases under both controllers. Nine trace-verified bilateral-contact-loss alarms arrive in 20–52 ms; twelve supported lowering responses produce no additional completion. Full trajectories and an independent audit expose failed acquisition and an unmonitored support-contamination gap. Hardware transfer remains open.

    Read the continuous-grip comparison · Inspect the recorded attempts
  7. Make skill prerequisites and failed recovery measurable

    Two native environments retain sixteen valid held-out attempts. A lid guard avoids two baseline contacts while completion remains 2/4. A mounted rake reaches three of four intermediate retrieval checkpoints, but no subsequent retained lift succeeds. Recorded replays expose every failure; fresh physics runs locally. Tool acquisition, learned control and hardware transfer remain open.

    Read the paired results · Explore both recorded experiments
  8. Move the PiPER arm in a live browser workstation

    Drag a target and nudge a free rigid trainer with live browser MuJoCo physics. The final native run keeps 20 valid attempts: 14/16 targets pass, both high-target repeats fail, and four mechanics checks pass. A ten-minute browser run reports 51.39 render calls per second on the tested host. Strict raw-contact parity fails in 13/1,500 sampled states despite numerical agreement; all fifteen development shoe-lift attempts fail. Learned shoe skills, calibrated contact and hardware transfer remain open.

    Read the workstation and evidence · Open the live workstation
  9. Load reusable shoes with three native robot arms

    Six original rigid shoe proxies now compose with UR5e/Robotiq, PiPER and SO-101. Versioned scene files, offline 3D previews and a frozen 54-attempt benchmark retain every trajectory: supported pairs 27/27, disabled-table support 0/27, all gravity checks pass. Native loading and passive contact are tested; shoe grasping, deformation and physical transfer remain unqualified.

    Read the asset library results · Browse shoes and arms
  10. Complete more placements with a bounded second lift and revised grasp

    Three fresh cohorts retain 252 valid attempts. The revised grasp completes 23/36 placements versus 15/36 for v0.4 and 18/36 with second lift alone; all 144 controls fail. Eight completions are gained and none lost, while shared successes average 0.772 simulated seconds slower. Twelve low-friction acquisitions and one retention failure remain. All 1,035 computation overruns are retained; physical qualification is open.

    Read the v0.5 reliability comparison · Inspect all three cohorts
  11. Separate faster recovery from an ineffective retry

    A fresh 84-attempt comparison completes 4/12 placements in every mode; all 48 controls fail. Two resumed tasks finish 2.440–2.640 simulated seconds sooner after unchanged settling holds, while four failed grips are refused 2.88–2.90 seconds sooner. Eight failures remain, including two held blocks below the resume-height gate. Every native trajectory and raw computation duration is retained; physical qualification is open.

    Read the v0.4 recovery comparison · Inspect every paired attempt
  12. Compare probe-only motion with contact history

    A frozen 84-attempt paired cohort completes 6/12 history placements versus 4/12 original and probe-only placements; all 48 controls fail. Four tasks complete after five closed holds/resumes, but all six supported acquisition retries fail. Only four of five resume windows meet the independent low absolute tool-speed check. Actual motion and 75 missed computation windows remain visible; physical robot qualification is open.

    Read the contact history comparison · Inspect all 84 attempts
  13. Test the grip and publish a completion regression

    A slow probe and one supported retry retain 30 valid attempts. At moderate friction baseline completes 2/2 while guarded completes 0/2; normal cases both pass and low-friction cases both fail. No successful recovery or overall improvement is demonstrated. Four brief following windows do not establish detector accuracy; 53 computation windows miss 20 ms. Every full attempt remains public.

    Read the paired results and actual stop motion · Inspect all 30 attempts
  14. Measure PiPER contact and a friction failure

    A frozen authored rigid-block pilot retains nine valid attempts: nominal placement 2/2, low-friction placement 0/1, open-jaw 0/3 and hold 0/3. One of 9,000 computation windows exceeds 20 ms. Complete task receipts remain distinct from unloaded profiles; learned skills, shoe physics and physical qualification are open.

    Read the PiPER contact pilot · Open the Robot Simulation Lab
  15. Run more arms on a local CPU

    PiPER and SO-101 join the library with pinned models, explicit arm/gripper units, preserved licenses and complete numerical receipts. Each completes six unloaded motion checks offline. Their learned-policy and physical qualification remain open. The PiPER contact pilot and UR5e grasp demonstration have separate task receipts.

    Read the profile measurements · Open the Robot Simulation Lab
  16. Give the reference arm a contact task

    A free rigid block must be grasped, lifted, carried, released and left settled at its destination. Frozen scenarios compare the authored controller with identical arm commands and open jaws, and with a hold control. Full traces and 3D replay preserve the evidence for the next compact action-learning experiment. Shoe physics, a learned task policy and hardware transfer remain open.

    Read the native task benchmark · Replay grasp and placement
  17. Build a hardware reference we can measure

    A pinned six-axis arm and coupled gripper expose fourteen coordinates and seven commands. All 18 unloaded numerical runs are valid; 1 ms CPU physics measures 40.80–47.29× elapsed-time throughput with sensing, rendering and policy excluded. A paired trace scorer makes physical mismatch measurable. Five-finger references and LeRobot collection have separate contracts; hardware fidelity and shoe competence remain unmeasured.

    Read the mechanical contract and upcoming arms · Open the 3D replay
  18. Learn a compact command-to-motion predictor

    A 6,438-byte CPU model predicts the next 20 ms of joint/jaw movement from native state and issued commands. Twelve seeded groups retain 36 recordings; 4,553 test transitions are not independent trials. Warm computation measures 0.207 ms p95, but jaw mean absolute/worst errors regress against persistence and the first opening is overpredicted. Shoe collection, hardware sensing and validated shoe physics remain open.

    Read the measured pilot and shoe collection plan
  19. Observe image motion and forecast its path

    A manually seeded CPU source alpha records pixel speed and causal forecasts. Six authored cases process 720 frames in 1.454 seconds; occlusion and low-texture losses remain visible, and a 500 ms reversal forecast misses by nearly 90 pixels. The v0.1 release/demo remain separate; real-scene accuracy, metric speed and live robot control are unqualified.

    Read the source alpha and measured limitations
  20. Publish the robotics research pipeline

    A diagram connects footage review, measured robot state, local training, frozen simulation evaluation and compatible skill installation. Automatic annotation and a learned task supervisor are proposed; their companion validation studies and physical robot qualification remain unrun.

    Read the architecture and validation gates
  21. A public skill link reaches a simulated arm

    One-click local installation binds a DVIDIA placement envelope to an authored contact controller. Six scene layouts pass; identical joint commands with open jaws and no-motion controls each pass 0/6. Physical transfer and learned policies remain open.

    Read the installation contract and arm replay
  22. The offline cable environment is runnable

    A local controller-training loop and simulation-only skill pack now run on CPU. Trained and baseline feedback each pass 10/10 simulated seeds; zero force and early release each pass 0/10. Real gripping, calibration and GPU qualification remain open.

    Read the first product build and replay
  23. Contact accuracy meets runtime

    54 deterministic CPU fixture trials recorded. Endpoint agreement coexists with unqualified force transients; coarse wall tests expose missed contact. No GPU or physical-transfer claim.

    Read the measurements
  24. Compare the observation-to-skill thesis with prior work

    Spatial attention, demonstration coverage, executable simulation and reusable repairs are reviewed separately. Three parallel Grok 4.7 searches support discovery; primary sources support the claims.

    Read the comparison
  25. Five licensed POV excerpts processed

    Ten image calls completed. A checker accepted an object error and captions omitted visible state changes. The separate accuracy pilot remains unrun.

    Read the feasibility run
  26. Define the evidence and release questions

    Four baseline protocols and a Skillspace graduation proposal establish the questions to test. A proposed protocol is distinct from an executed experiment.

    Read the agenda

Open questions

Follow new publications through RSS or inspect the source and revisions on GitHub.

+

An open research notebook

From observation
to physical skill.

We study how demonstrations become usable evidence, how simulation becomes useful practice, and when a robot skill earns a release.

Published by DVIDIA · hello@dvidia.org

Publications

32 papers and working notes

· Product architecture and local runtime verification

From a conversation to compatible robot skills

A conversation can carry a photo and desired outcome through scene review, skill retrieval, compatible installation, simulation and inspectable action traces. The local prototype uses ten LIBERO-10 task manifests sharing a pinned OpenVLA checkpoint. Real SSDlite inference and a repeatable native task-5 reset are recorded; the checkpoint transfer stopped after an 8 MiB partial file, so learned policy actions and activation-steered behavior remain unrun. The note separates observed evidence, candidate relevance, executable compatibility and measured completion, and defines staged evaluation for the chat workflow and raw MLP channel interventions. General household planning and hardware execution remain unqualified.

Photo inference and one native reset verified · OpenVLA actions and steering not run

· Learned jaw adapter with matched blind controls and retained negative studies

Learning grip feedback with a frozen PiPER arm reference

After five seven-command residual candidates each failed all 16 development cases, a separate 25-node learned jaw adapter preserves the frozen six-axis PiPER arm references. Trained on 64 synthetic teacher episodes and 64,000 causal labels, it completes 6/24 frozen confirmation cases, versus 8/24 for its matched blind control, 5/24 feedforward, 24/24 authored teacher and 0/24 open jaws. Feedback succeeds in 0/12 interrupted cases; 14/24 terminal-only passes conceal strict task failures. All 120 confirmation attempts are valid, and an independent numeric audit reconstructs 360 retained attempts. CPU inference p95 is 0.501 ms; this establishes no live-feedback advantage, recovered grip, human-video learning or hardware qualification.

Jaw feedback 6/24 · blind 8/24 · feedforward 5/24 · teacher 24/24 · no demonstrated feedback advantage

· Learned feedforward primitive from synthetic demonstrations

Learning PiPER block placement from synthetic demonstrations

A compact feedforward joint-and-jaw primitive trained on 32 successful synthetic demonstrations completes 5/12 frozen PiPER placement attempts from unseen starting positions. The authored teacher completes 12/12; three controls complete 0/12 each. All twelve learned terminal poses pass settling checks, while seven fail the full native contact-history gates. Complete tapes, failed development candidates, an independent audit and an installable local runner preserve the reliability gap. This is privileged-state reference distillation for a fixed rigid block and two trained goals, not human-video learning, learned feedback or physical robot qualification.

Learned 5/12 · teacher 12/12 · three controls 0/12 each · simulation only

· Paired synthetic benchmark for camera and drive evidence

Independent camera and actuator corroboration v0.4

A fresh eight-layout PiPER simulation benchmark adds separately processed RGB and drive checks to a local physical-goal verifier. The checks prevent 69 of 112 false completion readings while leaving 43 replay, identity, orientation, support and sampling blind spots; the conservative gate abstains on 73 of 80 baseline true completions, including all 64 deliberate sensor-fault controls. The measured CPU path is 22.1 ms median and 31.5 ms p95, excluding 200 ms acquisition. All raw channels and results are preserved; this is synthetic evidence, not hardware validation.

272 cases · 69/112 false completions prevented · 43 remain · 73/80 true completions abstained

· Local CPU verifier with native, evidence-fault and forgery diagnostics

A local verifier for physical completion

An offline audit traces the seven missed Jev completions without claiming an internal model cause. A local deterministic verifier then checks timestamped pose, motion, support, release and uncertainty on 200 fresh cases. It recognizes all 16 native positives and rejects all 48 native negatives; 128 evidence faults yield 120 abstentions and 8 incomplete answers. All 8 consistent forged streams produce physical false completions, exposing the remaining sensor-trust problem. CPU-only p95 is 0.318 ms, excluding the required 200 ms history and sensing. Complete native inputs, every decision, independent review and explicit failure cohorts remain public; no new Jev calls or hardware qualification.

Native: 16/16 true complete, 0/48 false complete · forged readings: 8/8 false complete

· Frozen hosted-model evaluation with native physical references

Jev image-goal verification: the measured follow-up

The first measured run preserves all 64 requests from the image-goal v0.1 protocol. Jev returned valid structured responses for every trial and made no false-completion declaration, but recognized only 1/8 physically complete cases with privileged native telemetry and 0/8 with images alone. All four occluded complete scenes were called incomplete in both conditions. RGB abstention matched 8/16 ambiguous references. Median online latency was 1.776 seconds, p95 3.287 seconds. The release preserves every result, the independent review and the original unrun protocol. This small synthetic diagnostic does not qualify a robot completion gate, actuator loop or learned skill.

64/64 valid responses · telemetry: 0/24 false complete, 1/8 true complete

· Native diagnostic benchmark and frozen hosted-model protocol

Image-goal verification: native benchmark and Jev protocol

A frozen PiPER image-goal cohort crosses eight physical conditions with four layouts. Four moving/settled pairs have byte-identical images and different completion labels, exposing when one image cannot establish task completion. Native measurements verify 32 authored fixtures; a paired 64-request Jev protocol distinguishes RGB judgments from privileged simulator telemetry. The provider evaluation has not run because credentials were unavailable: accuracy, false-completion rates and service latency remain unmeasured. Every native state, image, frozen request hash and explicit unrun result is retained. No learned policy or hardware qualification is claimed.

32 native fixtures · 4 identical-image pairs · 64 Jev trials not run

· Research direction and passive native contact audit

A visual goal still needs physical feedback

A goal image specifies a destination; physical feedback verifies whether a robot gets there. We describe a bounded decision role for TypeSafe Jev, distinguish it from action-conditioned world-model planning, and propose an independently scored reaching experiment. A separate passive audit reproduces fourteen previously inspected rigid-box arm episodes exactly and records 249,502 native ticks. Millisecond contact gaps hidden between stored observations become visible, while task outcomes remain unchanged. Complete inputs, traces, source snapshots and a verified downloader preserve every case. This is measurement work, not a new policy benchmark, PiPER shoe result or hardware qualification.

14/14 exact passive replays · 249,502 native ticks · visual models unbenchmarked

· Frozen shape, load and attached-tool diagnostic

Give the gripper a fair test

A frozen 24-attempt PiPER diagnostic separates three rigid shapes, 30 g and 300 g loads, and mounted versus absent rake conditions. All attempts are valid; eight of twelve lightweight controls retain a direct lift while every 300 g attempt fails, including the small block. Removing the rake gains one narrow-shoe result and loses one block result, with no net completion gain. A verified settled reference, dense contact traces, unchanged drive fingerprint and independent scoring expose limits in the modeled grasp and force response. Recorded replays and runnable CPU code retain every failure. No autonomous tool removal, retrieval, learned skill or physical hardware qualification is claimed.

24/24 valid · 30 g: 8/12 lifts · 300 g: 0/12 · no net gain from removing rake

· Frozen native comparison with independent contact audit

From reaching a shoe to keeping hold

A frozen twenty-attempt native MuJoCo comparison adds continuous acquisition checks and bounded lowering to the mounted-rake PiPER controller. Both policies complete the same 2/10 fresh cases: no gains or losses. Nine bilateral-contact-loss alarms follow a trace-verified native origin by 20–52 ms, median 30 ms; three acquisition failures are separate. Twelve supported lowering responses produce no additional completion. A non-finger-support gap remains unmonitored. Complete recorded trajectories, independent audit and development failures are public. Authored policies, synthetic contact and privileged observations establish no learned skill or physical robot qualification.

20/20 valid · 2/10 completed under both policies · 9 contact-loss alarms verified

· Two native environments with authored policies and recorded benchmark evidence

Skill prerequisites: access, reach and recovery

A frozen sixteen-attempt native MuJoCo study tests hinged shoe-box access and a mounted rake for an out-of-workspace shoe. Guarded access avoids two baseline lid collisions while task completion stays at 2/4; a transient withdrawal adds 0.70 simulated seconds without being necessary in its matched baseline. The mounted rake reaches an intermediate retrieval checkpoint in 3/4 cases, but all subsequent retained lifts fail and the short-tool case is deferred. All native traces and failures remain public. The browser replays recorded states; fresh physics runs locally. Privileged observations, synthetic contact and authored policies establish no learned skill or hardware qualification.

16/16 valid · 2 lid contacts avoided · 3/4 retrieval checkpoints · 0/4 retained tool lifts

· Live browser-native simulation with authored control and separate benchmark evidence

Live PiPER shoe workstation v0.1

A six-axis PiPER and its coupled gripper respond to user targets while MuJoCo runs locally in a browser worker. The final native run retains 20 valid attempts: 14/16 target trials pass, both high-target repeats fail, and jaw, impulse, disabled-table and 600-second hold checks pass. A ten-minute headless Chromium run reports 51.389 render calls per second and a 0.9999795 simulation/wall-time ratio on the tested Apple M5 host; simulated movement response is 61.9 ms p95 across 61 commands. Render calls do not measure GPU presentation. Native/WASM numerical and loaded-contact-count checks pass, while strict raw contact counts differ in 13/1,500 states. All fifteen development shoe-lift attempts fail. Complete development, failure and timing records remain separate. This authored rigid-shoe environment establishes no learned skill, calibrated material model or physical robot qualification.

Live browser physics · 14/16 targets · strict contact parity failed · shoe lifting unqualified

· Versioned native scene assets and frozen passive-contact benchmark

Reusable shoe asset library v0.1

Six original MIT rigid shoe proxies compose with the pinned UR5e/Robotiq, PiPER and SO-101 models through versioned scene descriptors and offline native 3D previews. A frozen 54-attempt test records 27/27 supported pairs and 0/27 support successes when table collisions are disabled; all 54 gravity checks pass without numerical warnings. No loaded non-table shoe contacts appear in the 20 ms samples, which cannot rule out shorter transients. All sampled trajectories and 27,000 native-step durations remain available. On one Apple M5 host, 54 simulated seconds take 12.3616 wall seconds including fresh model construction; the slowest attempt takes 6.7047 seconds. Geometry and physical properties are authored assumptions. Loading, gravity and passive support do not establish shoe grasping, calibrated deformation, a learned policy or hardware transfer.

Six shoe proxies · three arms · support 27/27, disabled table 0/27 · manipulation unqualified

· Frozen paired reliability experiment with complete native evidence

PiPER placement reliability v0.5

Three frozen fresh PiPER cohorts retain 252 numerically valid attempts and 144 failed negative controls. A fixed diagonal grasp with bounded pose planning completes 23/36 placements versus 15/36 for frozen v0.4 and 18/36 with second-lift recovery alone: eight gained completions and none lost. The fifteen shared successes are 0.772 simulated seconds slower on average. Three lift-only second probes qualify and finish their tasks; the full grasp candidate needs none. Twelve low-friction acquisitions and one later retention failure remain. Every trajectory, native task outcome and raw computation duration is retained, including 1,035 windows exceeding 20 ms and a 3.082-second outlier on a loaded host. Native mechanics and scoring remain unchanged. Separate development keeps 200 earlier attempts, including discarded regressions. Authored privileged simulation establishes no learned skill, real-time guarantee or physical qualification.

23/36 revised grasp · 15/36 v0.4 · 8 gained, none lost · hardware unqualified

· Paired carry-recovery and supported-acquisition experiment with complete confirmation receipt

PiPER contact recovery v0.4

A fresh frozen 84-attempt PiPER comparison completes 4/12 placements for v0.3 history, v0.4 carry-only and v0.4 full; all 48 matched open-jaw and hold controls fail. Two completed post-hold placements finish 2.440 and 2.640 simulated seconds sooner with unchanged 220/240 ms settling holds. Four low-friction failures reach refusal 2.88–2.90 seconds sooner without becoming successes. Fresh nominal-friction 60 g layouts expose two held-but-too-low resume failures in every mode; neither reaches speed restoration. Six earlier acquisition failures are traced to insufficient support under the pinned position servo and side pinch, not the actuator force cap. All native mechanics and outcome criteria remain unchanged. Complete trajectories and 84,000 raw computation windows are retained; none exceeds 20 ms on this host. Source-snapshotted development remains separate. Privileged authored simulation establishes no learned skill, true-slip accuracy or physical qualification.

All modes 4/12 · two resumed tasks faster · controls 0/48 · hardware unqualified

· Paired probe-only and contact-history experiment with complete confirmation receipt

PiPER contact history v0.3

A frozen paired PiPER experiment retains all 84 numerically valid attempts across twelve synthetic rigid-block scenes. History completes 6/12 placements versus 4/12 for both original motion and the identical acquisition probe without carry monitoring; all 48 separately matched open-jaw and hold controls fail. The two added completions occur at nominal friction with 60 g blocks. Four tasks complete after five closed holds/resumes, but only four of five resume windows meet the independent low absolute TCP-speed criterion. All six supported acquisition retries fail. Native gains, forces and outcome rules remain unchanged; requested movement reduction is distinct from actual speed, and two slowdown windows include early holds. Seventy-five of 84,000 computation windows exceed 20 ms. Complete sealed trajectories, original model notices, source-snapshotted development and pinned offline inspection preserve both success and failure. Privileged authored simulation establishes no learned skill, true-slip detector accuracy, shoe competence or physical qualification.

History 6/12 · baseline and probe-only 4/12 · controls 0/48 · hardware unqualified

· Paired authored controller experiment and complete confirmation receipt

PiPER grip validation and recovery

A slow test lift and bounded supported retry add recorded acquisition decisions to the pinned PiPER contact pilot, but completion regresses: at moderate friction baseline completes 2/2 and guarded completes 0/2. Both complete 2/2 normal cases; neither completes the two low-friction cases. All 30 attempts remain numerically valid, including 18 failed controls and full-duration aborts. Four qualification decisions are followed by a brief independent following window, yet two tasks still fail; no successful recovery or overall improvement is demonstrated. Actual motion continues after hold requests, and 53/30,000 computation windows exceed 20 ms. Complete compressed receipts, original model notices and pinned offline inspection preserve the evidence. Authored privileged-state simulation establishes no detector accuracy, learned shoe skill, calibrated contact or physical robot qualification.

Moderate-friction regression · baseline 4/6, guarded 2/6 · hardware unqualified

· Native contact task implementation and authored baseline

Native grasp task benchmark

A pinned six-axis arm and coupled gripper complete 6/6 frozen synthetic rigid-block placements, while matched recorded-arm/open-jaw and hold controls complete 0/12. Native contact must produce grasp, sustained lift, retained carry, release and settled placement with tool retreat. All 18 attempts and timing traces remain in the receipt; one placement takes 29.51 wall seconds for 20 simulated seconds and misses 251 of 1,000 computation windows, so real-time operation is not established. An offline 3D mesh replay exposes the recorded motion. The authored privileged-state baseline provides a benchmark for a future compact DVIDIA action policy, not a learned footage-to-action model, validated shoe simulation or physical robot qualification.

6/6 simulated placements · controls 0/12 · real-time and hardware unqualified

· Pinned mechanics implementation and CPU numerical benchmark

Hardware matched robot simulation

A separate UR5e/Robotiq reference preserves pinned geometry, inertias, moving-link collision proxies and coupled gripper structure: fourteen modeled coordinates and seven commands. Eighteen deterministic unloaded motion/closure runs are numerically valid; at a 1 ms step, CPU physics throughput measures 40.80–47.29 simulated seconds per measured stepping wall second on one host, excluding loading, sensing, rendering, policy and real actuation. Adjacent timestep comparisons and a matched static overlap expose numerical sensitivity and collision participation. A pure-Python paired trace scorer measures joint, tool and jaw mismatch without automatic qualification. Left/right five-finger references are separately audited; LeRobot collection with an original DVIDIA policy is proposed. Serial calibration, physical response, shoe task learning and bimanual execution remain unmeasured. An offline 3D mesh replay now exposes the recorded arm/gripper poses without rerunning physics. PiPER and SO-101 now have pinned CPU profiles with 6/6 valid unloaded numerical attempts each and explicit native command units. PiPER also has a frozen authored rigid-block contact pilot: nominal placement 2/2, low-friction placement 0/1, open-jaw 0/3 and hold 0/3, with all nine attempts numerically valid. One of 9,000 computation windows exceeds 20 ms; hardware real-time response is unqualified. SO-101 object tasks, learned policies and physical qualification remain open. YAM, OpenArm, ARX X5 and PAROL6 remain upcoming.

Pinned CPU mechanics · PiPER contact pilot · physical fidelity unmeasured

· Local observation workflow and engineering feasibility pilot

DVIDIA Observation Lab pilot

DVIDIA Observation Lab supplies local object/palm proposals, reviewed timeline revisions and separate causal image-motion forecasts; real annotation and motion accuracy remain unmeasured. A new CPU pilot learns next-20-ms joint/jaw deltas from current native state and issued commands. Twelve seeded groups retain 36 recordings with 8/2/2 train/development/test grouping; 4,553 test transitions are not independent robot trials. The 6,438-byte action model computes at 0.207 ms p95 on one host and improves RMSE, while jaw MAE/worst error regress against persistence and an initial opening is substantially overpredicted. These are privileged simulator measurements and model-only timings. A proposed shoe capture plan has no footage or executed shoe benchmark. The v0.1 release and hosted synthetic review demo remain separate; no shoe policy, physical adapter, validated shoe physics or hardware qualification is established.

Local review release · motion and actuator source pilots · hardware unqualified

· Architecture proposal and staged validation roadmap

Robotics research pipeline

An updated architecture connects immutable Skillspace recordings, reviewed task-label proposals and synchronized measured telemetry to grouped datasets, separate visual and motor learners, and a proposed local state supervisor. The implemented CPU baseline retains authored phases, gripper decisions and recovery around its learned joint-delta head. New annotation and supervisor pilots, fresh frozen confirmation, physics calibration and compatible skill-installation gates define the next milestones. Automatic labeling, learned supervision, human-video action recovery and physical arm qualification remain unestablished; this publication adds no new training or hardware result.

Architecture proposal · current simulation alpha · hardware unqualified

· Product implementation and measured synthetic pilot

Skillspace footage training pipeline

The public Dvidia-Inference/dvidia-training repository provides a standalone CPU pipeline that audits Skillspace media, freezes grouped splits and exports inspectable artifacts. Ten synthetic six-second clips train a visual predictor with a 0.5106-second median pipeline time across three repeats on one macOS host. Aligned native simulator telemetry supplies 4,029 training labels for a learned joint-delta head, evaluated on one held-out recording. Its capsule passes native placement qualification in that recording's exact scene; identical joint targets with open jaws fail. Installation and hardware guides accompany the package. Human-video action recovery, shoe organization, demonstration sufficiency, independent scene generalization and physical robot competence remain unestablished.

Public CPU package · one native scene qualified · simulation-only

· Learned movement artifact and installation prototype

Skill capsule acquisition and optimization

A compact learned CPU movement head replaces joint-target computation inside an authored placement supervisor. Teacher and student both pass 7/8 in a new frozen evaluation, while episode wall time increases from 6.273 to 6.360 seconds. A 39,845-byte JSON capsule imports into the local lab, checks its exact runtime and qualifies the current scene through native placement and a recorded-arm/open-jaw control. Scene changes require a fresh test. One known scene and repeat pass native network denial; perception, complete task learning, other embodiments and hardware remain unqualified.

Simulation-only · teacher/student 7/8 · no overall speedup

· Paired simulation benchmark and product assessment

Skillspace arm benchmarks and evolution

Two controller revisions improve development completion from 14/20 to 17/20, but the frozen paired test declines from 30/32 to 29/32: both pass all 24 normal-strength cases while actuator-stress completion falls from 6/8 to 5/8. Instrumented CPU throughput also declines. We publish the regression, unchanged contact and success rules, exact controller snapshots, richer physical inputs, runtime-bound installation and a proposed staged Skillspace contract. Contact forces and penetration remain uncalibrated; learned perception and physical qualification are open.

Mixed held-out result · 30/32 baseline, 29/32 updated

· Product prototype and native contact qualification

Installing a Skillspace on a simulated arm

The first link-to-arm product path installs an existing DVIDIA placement envelope, binds a local authored controller to one simulated arm, and reuses it in independently grounded scenes. Native jaw contacts carry a free box; six layouts complete, while no-motion and identical joint commands with open jaws each complete zero of six. Installed execution passes process-level offline checks. Task metadata contains no learned policy; perception, cup geometry, full arm collisions and physical qualification remain open.

Public skill link to authored adapter · 6/6 scene placements

· Product prototype and local qualification

Offline cable training environment

The first runnable product loop combines an anchored cable environment, local controller-parameter search, frozen evaluation, offline replay and a manifest-backed skill pack. The selected controller and a fixed feedback baseline both completed 10/10 held-out simulated episodes; zero-force and release controls completed 0/10. A native macOS network-denial run completed 3/3 additional episodes. Ideal endpoint attachment and privileged state bound the result; physical rope calibration, gripping, sensors and GPU throughput remain untested.

CPU simulation-only · Controller search and frozen evaluation executed

· Decision report and local experiment

Fast accurate robot training simulation

A design for an offline, self-hosted training environment that turns demonstrations into calibrated practice and qualified robot skills. We examine contact, deformation, sensor feedback and the performance–accuracy tradeoff, then report 54 deterministic native MuJoCo CPU fixture trials. Fine endpoint agreement did not establish accurate contact transients, and coarse timesteps missed collisions. Minimum GPU memory, rope accuracy and physical transfer remain unmeasured.

54 CPU fixture trials · GPU and physical transfer untested

· Source review and thesis comparison

Grok robotics research on X

Three Grok 4.7 research passes surface observation-to-action, synthetic-practice and reusable-repair ideas. Independent primary-source checks compare those findings with our thesis: observation supplies goals and state, language retrieves a skill, spatial expectations guide attention, and executable feedback with evaluation establishes competence. X posts remain discovery leads. No universal demonstration count or arbitrary-robot transfer follows from the reviewed results.

Primary sources checked · Experiments proposed

· Research synthesis and proposed experiments

Spatially grounded downloadable robot skills

A synthesis of learning from human demonstrations, language-conditioned spatial attention, simulation expansion and robot-specific skill delivery. Spatial priors can accelerate inspection but do not determine contact state or control. The proposed downloadable skill packages a controller, initiation and termination conditions, compatible sensing and hardware, recovery behavior and measured evidence. Separate experiments would test attention, data coverage, feedback and transfer.

Source review · DVIDIA experiments not run

· Engineering feasibility pilot

Clef and grounded video observations

A preregistered image-API feasibility run processed five licensed three-second POV excerpts using four timestamped frames, a Gemma caption and a Clef-flash check. All five completed at a price-derived inference estimate of $0.00140634. Post-run inspection found an object error accepted by the checker and omitted state changes. API and schema compatibility were demonstrated; annotation accuracy and whole-task success were not.

Five licensed excerpts · Accuracy evaluation outstanding

· Source review and implementation proposal

From Skillspace to robot skill

An Argus review and a proposal to connect original recordings, reviewed episodes, immutable datasets, training runs, held-out evaluation and robot-tested releases. The review separates video annotation from action supervision and defines compatibility beyond a robot’s degree-of-freedom count. A proposed 40-episode annotation pilot and evidence-based graduation criteria remain unrun.

Graduation and robot training not run

· Research agenda

Turn physical footage into testable evidence

A research agenda covering visible capture evidence, relations over time, evaluation provenance and physical grounding. Each question has a bounded protocol and a failure condition. The aim is to measure whether structure and physical signals improve a defined task before scaling collection. Strategic opinions, author-reported results and our untested proposals are kept distinct.

Four baseline protocols proposed

Research footprints

  1. Learn a first PiPER placement primitive from demonstrations

    A compact feedforward model trained on 32 synthetic teacher demonstrations completes 5/12 frozen placement attempts from unseen starts. The teacher completes 12/12; three controls complete none. All twelve learned final poses settle near their targets, but seven fail full contact-history gates. The recorded viewer, complete evidence and installable local runner preserve the reliability gap. Learned feedback, human-video learning and physical transfer remain open.

    Read the learned-placement study · Inspect all twelve learned attempts
  2. Measure Jev against physical completion

    All 64 frozen requests returned valid responses. No false completion was observed, but images plus telemetry recognized only 1/8 completed states, and images alone recognized 0/8. Occluded finished scenes failed in both conditions. Median hosted latency was 1.776 seconds. The original protocol remains unchanged; all measured cases and independent review are public.

    Read the measured follow-up · Inspect every judgment
  3. Make image-goal uncertainty testable

    Thirty-two native PiPER fixtures include four moving/settled pairs with identical images and different physical completion labels. The frozen 64-request Jev protocol compares RGB with privileged simulator telemetry. Jev has not been run; all provider trials remain explicitly unrun, with accuracy and service latency unmeasured. Native states, images, source hashes and physical labels are public.

    Read the benchmark and protocol · Compare every current and goal image
  4. Connect visual goals to measured physical feedback

    Fourteen passive rigid-box arm replays reproduce every original sampled state and outcome, recording 249,502 native ticks. Millisecond contact gaps appear between stored observations; a successful comparator also has gaps. The separate architecture note evaluates Jev as a candidate decision layer and defines a goal-image reaching test. No visual model is integrated and no controller is promoted.

    Read the architecture and audit · Download verified evidence tool
  5. Separate object difficulty from the gripper and attached tool

    A frozen direct-grasp diagnostic retains 24 valid attempts. Lightweight controls complete 8/12 lifts; every 300 g condition fails, including the small block. Removing the rake gains one narrow-shoe result and loses one block result. Relative lift, settled references and dense contact traces make the modeled drive and geometry limits visible. These are diagnostic controls, not a shoe retrieval rate or hardware qualification.

    Read the graspability study · Inspect every recorded attempt
  6. Measure contact-loss response without claiming a better grasp

    Twenty valid fresh attempts complete the same 2/10 cases under both controllers. Nine trace-verified bilateral-contact-loss alarms arrive in 20–52 ms; twelve supported lowering responses produce no additional completion. Full trajectories and an independent audit expose failed acquisition and an unmonitored support-contamination gap. Hardware transfer remains open.

    Read the continuous-grip comparison · Inspect the recorded attempts
  7. Make skill prerequisites and failed recovery measurable

    Two native environments retain sixteen valid held-out attempts. A lid guard avoids two baseline contacts while completion remains 2/4. A mounted rake reaches three of four intermediate retrieval checkpoints, but no subsequent retained lift succeeds. Recorded replays expose every failure; fresh physics runs locally. Tool acquisition, learned control and hardware transfer remain open.

    Read the paired results · Explore both recorded experiments
  8. Move the PiPER arm in a live browser workstation

    Drag a target and nudge a free rigid trainer with live browser MuJoCo physics. The final native run keeps 20 valid attempts: 14/16 targets pass, both high-target repeats fail, and four mechanics checks pass. A ten-minute browser run reports 51.39 render calls per second on the tested host. Strict raw-contact parity fails in 13/1,500 sampled states despite numerical agreement; all fifteen development shoe-lift attempts fail. Learned shoe skills, calibrated contact and hardware transfer remain open.

    Read the workstation and evidence · Open the live workstation
  9. Load reusable shoes with three native robot arms

    Six original rigid shoe proxies now compose with UR5e/Robotiq, PiPER and SO-101. Versioned scene files, offline 3D previews and a frozen 54-attempt benchmark retain every trajectory: supported pairs 27/27, disabled-table support 0/27, all gravity checks pass. Native loading and passive contact are tested; shoe grasping, deformation and physical transfer remain unqualified.

    Read the asset library results · Browse shoes and arms
  10. Complete more placements with a bounded second lift and revised grasp

    Three fresh cohorts retain 252 valid attempts. The revised grasp completes 23/36 placements versus 15/36 for v0.4 and 18/36 with second lift alone; all 144 controls fail. Eight completions are gained and none lost, while shared successes average 0.772 simulated seconds slower. Twelve low-friction acquisitions and one retention failure remain. All 1,035 computation overruns are retained; physical qualification is open.

    Read the v0.5 reliability comparison · Inspect all three cohorts
  11. Separate faster recovery from an ineffective retry

    A fresh 84-attempt comparison completes 4/12 placements in every mode; all 48 controls fail. Two resumed tasks finish 2.440–2.640 simulated seconds sooner after unchanged settling holds, while four failed grips are refused 2.88–2.90 seconds sooner. Eight failures remain, including two held blocks below the resume-height gate. Every native trajectory and raw computation duration is retained; physical qualification is open.

    Read the v0.4 recovery comparison · Inspect every paired attempt
  12. Compare probe-only motion with contact history

    A frozen 84-attempt paired cohort completes 6/12 history placements versus 4/12 original and probe-only placements; all 48 controls fail. Four tasks complete after five closed holds/resumes, but all six supported acquisition retries fail. Only four of five resume windows meet the independent low absolute tool-speed check. Actual motion and 75 missed computation windows remain visible; physical robot qualification is open.

    Read the contact history comparison · Inspect all 84 attempts
  13. Test the grip and publish a completion regression

    A slow probe and one supported retry retain 30 valid attempts. At moderate friction baseline completes 2/2 while guarded completes 0/2; normal cases both pass and low-friction cases both fail. No successful recovery or overall improvement is demonstrated. Four brief following windows do not establish detector accuracy; 53 computation windows miss 20 ms. Every full attempt remains public.

    Read the paired results and actual stop motion · Inspect all 30 attempts
  14. Measure PiPER contact and a friction failure

    A frozen authored rigid-block pilot retains nine valid attempts: nominal placement 2/2, low-friction placement 0/1, open-jaw 0/3 and hold 0/3. One of 9,000 computation windows exceeds 20 ms. Complete task receipts remain distinct from unloaded profiles; learned skills, shoe physics and physical qualification are open.

    Read the PiPER contact pilot · Open the Robot Simulation Lab
  15. Run more arms on a local CPU

    PiPER and SO-101 join the library with pinned models, explicit arm/gripper units, preserved licenses and complete numerical receipts. Each completes six unloaded motion checks offline. Their learned-policy and physical qualification remain open. The PiPER contact pilot and UR5e grasp demonstration have separate task receipts.

    Read the profile measurements · Open the Robot Simulation Lab
  16. Give the reference arm a contact task

    A free rigid block must be grasped, lifted, carried, released and left settled at its destination. Frozen scenarios compare the authored controller with identical arm commands and open jaws, and with a hold control. Full traces and 3D replay preserve the evidence for the next compact action-learning experiment. Shoe physics, a learned task policy and hardware transfer remain open.

    Read the native task benchmark · Replay grasp and placement
  17. Build a hardware reference we can measure

    A pinned six-axis arm and coupled gripper expose fourteen coordinates and seven commands. All 18 unloaded numerical runs are valid; 1 ms CPU physics measures 40.80–47.29× elapsed-time throughput with sensing, rendering and policy excluded. A paired trace scorer makes physical mismatch measurable. Five-finger references and LeRobot collection have separate contracts; hardware fidelity and shoe competence remain unmeasured.

    Read the mechanical contract and upcoming arms · Open the 3D replay
  18. Learn a compact command-to-motion predictor

    A 6,438-byte CPU model predicts the next 20 ms of joint/jaw movement from native state and issued commands. Twelve seeded groups retain 36 recordings; 4,553 test transitions are not independent trials. Warm computation measures 0.207 ms p95, but jaw mean absolute/worst errors regress against persistence and the first opening is overpredicted. Shoe collection, hardware sensing and validated shoe physics remain open.

    Read the measured pilot and shoe collection plan
  19. Observe image motion and forecast its path

    A manually seeded CPU source alpha records pixel speed and causal forecasts. Six authored cases process 720 frames in 1.454 seconds; occlusion and low-texture losses remain visible, and a 500 ms reversal forecast misses by nearly 90 pixels. The v0.1 release/demo remain separate; real-scene accuracy, metric speed and live robot control are unqualified.

    Read the source alpha and measured limitations
  20. Publish the robotics research pipeline

    A diagram connects footage review, measured robot state, local training, frozen simulation evaluation and compatible skill installation. Automatic annotation and a learned task supervisor are proposed; their companion validation studies and physical robot qualification remain unrun.

    Read the architecture and validation gates
  21. A public skill link reaches a simulated arm

    One-click local installation binds a DVIDIA placement envelope to an authored contact controller. Six scene layouts pass; identical joint commands with open jaws and no-motion controls each pass 0/6. Physical transfer and learned policies remain open.

    Read the installation contract and arm replay
  22. The offline cable environment is runnable

    A local controller-training loop and simulation-only skill pack now run on CPU. Trained and baseline feedback each pass 10/10 simulated seeds; zero force and early release each pass 0/10. Real gripping, calibration and GPU qualification remain open.

    Read the first product build and replay
  23. Contact accuracy meets runtime

    54 deterministic CPU fixture trials recorded. Endpoint agreement coexists with unqualified force transients; coarse wall tests expose missed contact. No GPU or physical-transfer claim.

    Read the measurements
  24. Compare the observation-to-skill thesis with prior work

    Spatial attention, demonstration coverage, executable simulation and reusable repairs are reviewed separately. Three parallel Grok 4.7 searches support discovery; primary sources support the claims.

    Read the comparison
  25. Five licensed POV excerpts processed

    Ten image calls completed. A checker accepted an object error and captions omitted visible state changes. The separate accuracy pilot remains unrun.

    Read the feasibility run
  26. Define the evidence and release questions

    Four baseline protocols and a Skillspace graduation proposal establish the questions to test. A proposed protocol is distinct from an executed experiment.

    Read the agenda

Open questions

Follow new publications through RSS or inspect the source and revisions on GitHub.

diff --git a/docs/papers/chat-to-robot-skills/index.html b/docs/papers/chat-to-robot-skills/index.html new file mode 100644 index 0000000..b9733d7 --- /dev/null +++ b/docs/papers/chat-to-robot-skills/index.html @@ -0,0 +1,183 @@ + + +From a conversation to compatible robot skills · DVIDIA Research +

Product architecture and local runtime verification

From a conversation to compatible robot skills

· DVIDIA

Photo inference and one native reset verified · OpenVLA actions and steering not run

Source & history · Download Markdown

Abstract

A conversation can carry a photo and desired outcome through scene review, skill retrieval, compatible installation, simulation and inspectable action traces. The local prototype uses ten LIBERO-10 task manifests sharing a pinned OpenVLA checkpoint. Real SSDlite inference and a repeatable native task-5 reset are recorded; the checkpoint transfer stopped after an 8 MiB partial file, so learned policy actions and activation-steered behavior remain unrun. The note separates observed evidence, candidate relevance, executable compatibility and measured completion, and defines staged evaluation for the chat workflow and raw MLP channel interventions. General household planning and hardware execution remain unqualified.

In this paper

DVIDIA research · 27 October 2026 · Product architecture and local runtime note

+

Original writing: CC BY 4.0. Referenced models, code and assets retain their own terms.

+

The intended interface is a conversation: attach a photo, describe a desired +outcome, and let DVIDIA find an executable skill for the available robot. The +conversation should carry the workflow from observation through installation, +simulation and review. A user should not need to assemble those tools manually.

+

The current prototype connects photo analysis and conservative skill retrieval +to a separate local worker for ten LIBERO-10 task descriptions sharing one +revision-pinned OpenVLA checkpoint. Real local object detection and one native +simulator reset are verified. OpenVLA model inference, learned robot actions and +activation-steered robot actions have not been executed. The checkpoint +transfer stopped after an 8 MiB partial download; this is not an installation or +a successful task. The runtime receipt +and evidence notes +preserve that boundary. A companion Hugging Face concept card +tracks the same documentation scope; it does not host OpenVLA inference.

+

Implementation follow-up — 28 October 2026

+

The local prototype now implements the photo/outcome composer and conversation +cards. A unique compatible unblocked task starts verified installation and real +scene preparation; the user sees the simulator camera before choosing Run in +simulation. Connection settings, object corrections and action/neuron inspection +are expandable. This is a typed tool workflow with conservative lexical +retrieval; the conversation does not add a general language planner.

+

Nineteen focused chat/client/planner tests and 42 worker tests pass. New +regressions reject reversed spatial destinations and wrong target colors. +Installation cancellation races and explicit stop-tracking recovery are handled. +Browser control initially timed out, then recovered in a fresh tab. Development +and built desktop previews verified real photo detection, an unsupported +dishwashing reply, actual task-5 camera capture, unique compatible retrieval and +live installation progress. Browser error logs were empty. Phone-sized layout +acceptance remains pending. The repaired downloader has resumed the real +checkpoint transfer, but no completed installation, learned action or steering +result is established. The 26 October receipt below remains unchanged.

+

Local conversational prototype showing actual simulator input and an incomplete policy download
Local conversational prototype showing actual simulator input and an incomplete policy download

+

The screenshot shows installation progress, not robot execution.

+

One conversation carries the task

+

A useful request is “Put the book in the back compartment of the caddy,” with a +photo attached. DVIDIA needs to identify the intended object and destination, +look for a compatible behavior, check its prerequisites and expose any missing +evidence. The product should present these as progress in one conversation:

+
Conversation eventWhat it represents
Photo and desired outcomeUser input, including explicit corrections
“I found a book; please identify its destination”A grounded question when the evidence is insufficient
Skill card with prerequisites and compatibilityA candidate, with reasons it can or cannot run
Download progress and verified installationActual artifact state, including interruption and retry
Simulator preview and run progressA real task reset followed by actual policy actions when available
Outcome and inspectable traceSuccess, failure or cancellation with actions and observations
“Try this intervention”An experimental rerun with the same initial simulator state
+

This is the conversational simplification implemented in the local prototype. +Detailed connection settings, manifests, action vectors and channel controls can +live in expandable cards. The conversation remains an interface to typed tools; +assistant prose cannot mark an artifact installed or a task completed.

+

“Tidy this room” expresses a valid desired outcome but leaves important +decisions unresolved: where toys belong, which clothes are dirty, whether a +container is reachable and whether the robot has the required behaviors. A photo +does not answer all of these questions. The first version should ask for missing +destinations or state evidence and report unsupported tasks. General household +task decomposition and automatic selection of multi-skill sequences are future +work. No dishwashing, laundry or toy-sorting competence is established by this +prototype.

+

Retrieve by intended change, then check compatibility

+

The architecture connects five distinct decisions:

+
Photo + desired outcome
+    -> observations with source and uncertainty
+    -> candidate skills with prerequisites and effects
+    -> compatible artifact + verified installation
+    -> actual simulator state + policy execution
+    -> measured outcome + inspectable trace
+
+

An object name is a useful search clue, but the intended change determines +whether a skill is relevant. A mug-placement task does not satisfy a request to +wash a mug. A drawer task that closes the drawer adds an unwanted effect if the +user asked for it to remain open. The current browser planner uses bounded +lexical matching and aliases; it checks object identities, destinations, +relations, state changes and negation, then rejects incompatible profiles. It is +not a learned affordance estimator or a general language planner.

+

This separation is motivated by SayCan, which +combines language-based skill relevance with state-dependent affordances. Our +current rule-based filter does not reproduce SayCan's learned value functions +or its reported performance. The design implication is to keep goal relevance, +observed prerequisites and runtime compatibility independently inspectable.

+

Every executable candidate currently carries a skill ID, instruction, required +objects, effects, prerequisites, profile ID, checkpoint repository, immutable +revision and benchmark task ID. The present profile is +libero-panda-openvla-v1: a Franka Panda simulation using seven-dimensional OSC +pose actions. Its checkpoint is +openvla/openvla-7b-finetuned-libero-10. +The ten task entries share the same checkpoint; downloading each task does not +create ten separate learned policies.

+

The example v1 manifest +is documentation only and contains no installed artifact.

+

A broader skill registry will also need versioned observation and action +schemas, units, control rate, gripper convention, tested operating envelope, +required sensors, adapter versions and task-specific evaluation receipts. +Hardware compatibility would require its own tested adapter and qualification. +Those extended registry fields are a proposed contract, not an implemented +universal skill package.

+

Execution belongs to the local worker

+

The browser communicates with an isolated loopback worker using the typed +dvidia.robot-lab.v1 protocol and an in-memory capability. The worker performs +real detector calls, artifact verification, native resets, model loading and +bounded runs. It contains no physical robot driver. Checkpoint size, revision +and digests are verified before atomic activation; partial files never count as +an installed skill. The runtime loads explicit classes from pinned OpenVLA +source rather than executing a checkpoint's remote class mapping.

+

An arbitrary household photo is planning input. It is not an executable LIBERO +state. Before a run, the user must select a supported benchmark task and capture +its real simulator camera and initial state. Task metadata is marked as +simulator evidence, distinct from image detections and user corrections. A run +is bound to that task, scene, seed and installed checkpoint. Completion must +come from the simulator's task-success signal.

+

OpenVLA supplies a language-conditioned +robot action policy. The selected checkpoint card +describes a LIBERO-10 fine-tune; it is a concrete starting point for this +benchmark-specific experiment, not a generally installable housekeeping model. +The runtime uses predict_action with the libero_10 normalization statistics. +Code, checkpoint and underlying model terms need separate review; the upstream +MIT labels do not replace the applicable Llama 2 terms or third-party asset +licenses.

+

What has actually run

+

The retained receipt was recorded on 26 October 2026 and is published here as a +historical engineering result. It is not a fresh benchmark of this note's date.

+
ComponentRecorded evidenceLimit
Local runtimePython 3.12.9, Torch 2.6.0 and MPS detected on an Apple Silicon MacFull OpenVLA memory fit and latency unverified
Real photo inferenceVerified TorchVision SSDlite checkpoint; cup 0.9957, person 0.8254 and dining table 0.4730 on one editorial imageConfidence scores are detector outputs, not measured accuracy; no dirt, color, ownership or task-state judgment
Native task resetLIBERO-10 task 5, seed 7, book/caddy camera image and flattened-state digestZero policy actions; ten no-op settling steps only
Reset repeatabilityWorker-thread and independent main-thread reset digests matchMatching initial state does not establish deterministic inference or rollouts
Artifact installationInterrupted transfer retained 8 MiB of a roughly 15.1 GB checkpointNo activated checkpoint
Policy and steeringTransport and hook fixtures testedNo learned rollout, task result or behavioral steering measurement
+

The reset digest is +b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e. +The macOS worker uses MuJoCo's official offscreen CGL context to avoid initializing +GLFW/AppKit from an HTTP worker thread. This changes context selection, not the +physics or observed state. Only the task-5 asset closure was provisioned for the +recorded runtime check; other task assets remain locally unverified.

+

Actual LIBERO task-5 reset camera, before any policy action
Actual LIBERO task-5 reset camera, before any policy action

+

The photograph used for detector testing was an editorial illustration from the +app, not a physical robot camera capture. The simulator image above is an actual +rendered reset. Neither is evidence that a robot performed the requested action.

+

Neuron exploration is a separate measured intervention

+

Lyra's neuron explorer +inspired the interaction: inspect model units, change a unit's activation and +retain the resulting behavior. Its text-model experiments do not by themselves +establish a robot control method. The directly relevant robotics precedent is +Mechanistic Interpretability for Steering Vision Language Action Models, +which studies semantic activation directions and interventions in OpenVLA and +π₀. Its research code +is a methodological reference; the DVIDIA worker uses an original hook and does +not vendor that toolkit.

+

The implemented hook observes a zero-based intermediate Llama MLP channel at the +input to down_proj. It records the last token's value on each autoregressive +forward pass and can add a bounded raw dose to that channel. The displayed +per-action value averages the pre-intervention values across generation passes. +This is a raw channel intervention, not an identified “careful,” “fast” or +“move left” neuron. It is also not an implementation-equivalent reproduction of +the paper's semantic cluster interventions. Hook cleanup and cancellation have +unit coverage; their effects on a loaded robot policy remain unmeasured.

+

Before behavioral claims, compare an unsteered baseline and intervention on the +same checkpoint, task, seed, preprocessing and exact reset digest. Record all +action dimensions, end-effector motion, activation summaries, cameras, +completion and failures. Current portable Pillow preprocessing is explicitly a +numerical variant of upstream's TensorFlow path, so published OpenVLA benchmark +scores are not a performance reference for this Mac configuration.

+

Staged evaluation and release gates

+
    +
  1. Conversation and retrieval: freeze a small set of supported and +unsupported requests, including ambiguous destinations, negation and desired +state changes. Independently label compatible candidates. Report retrieval +precision and recall, unsupported-task handling and clarification behavior. +An interaction that looks correct is not evidence of execution.
  2. +
  3. Installation and one-action diagnostic: finish the revision-pinned +transfer, verify every artifact, load the full model and execute one real +action on the recorded task-5 reset. Record peak memory, load/inference time, +actual action vector and failure status. Stop if memory or rendering is +insufficient; a substitute controller does not satisfy this gate.
  4. +
  5. Baseline simulation: run bounded unsteered episodes on a frozen set of +seeds. Retain every attempt and the actual LIBERO success signal. Report +strict completion, step counts, latency and invalid attempts separately.
  6. +
  7. Causal channel study: choose units and doses using development scenes, +then freeze them. Compare paired baseline, sham-hook, positive/negative-dose +and randomly selected channel controls on held-out seeds. Measure task +completion alongside motion changes; do not select only successful videos.
  8. +
  9. Skill sequencing and household transfer: introduce separately qualified +skills, evidence refresh after each action, explicit recovery and destination +rules. Hardware execution needs independent adapter, sensing and task +qualification. This stage is not implemented or run.
  10. +
+

Stages 2–5 remain unrun. The first deliverable is an honest conversation that +connects to real tools and makes missing capability understandable. The research +question is whether that interface can select a skill whose measured behavior +achieves the user's intended change, and whether controlled interventions alter +that behavior reproducibly without degrading completion.

+
diff --git a/docs/papers/clef-and-grounded-video-observations/index.html b/docs/papers/clef-and-grounded-video-observations/index.html index 87c33c4..e49ce30 100644 --- a/docs/papers/clef-and-grounded-video-observations/index.html +++ b/docs/papers/clef-and-grounded-video-observations/index.html @@ -1,7 +1,7 @@ Clef and grounded video observations · DVIDIA Research -

Engineering feasibility pilot

Clef and grounded video observations

· DVIDIA

Five licensed excerpts · Accuracy evaluation outstanding

Source & history · Download Markdown

Abstract

A preregistered image-API feasibility run processed five licensed three-second POV excerpts using four timestamped frames, a Gemma caption and a Clef-flash check. All five completed at a price-derived inference estimate of $0.00140634. Post-run inspection found an object error accepted by the checker and omitted state changes. API and schema compatibility were demonstrated; annotation accuracy and whole-task success were not.

In this paper

Engineering pilot executed · 6 October 2026 · Accuracy evaluation outstanding

+

Engineering feasibility pilot

Clef and grounded video observations

· DVIDIA

Five licensed excerpts · Accuracy evaluation outstanding

Source & history · Download Markdown

Abstract

A preregistered image-API feasibility run processed five licensed three-second POV excerpts using four timestamped frames, a Gemma caption and a Clef-flash check. All five completed at a price-derived inference estimate of $0.00140634. Post-run inspection found an object error accepted by the checker and omitted state changes. API and schema compatibility were demonstrated; annotation accuracy and whole-task success were not.

In this paper

Engineering pilot executed · 6 October 2026 · Accuracy evaluation outstanding

We executed a small, preregistered feasibility run: five existing public, licensed three-second POV excerpts, four original-timestamped frames per excerpt, one Gemma caption call and one Clef-flash check per excerpt. All five completed. The estimated inference cost was $0.00140634. This establishes that the image APIs and bounded data contract work; it does not establish that the annotations are accurate.

The useful finding was a failure: a caption called a pink plate a cutting board, and the second model agreed. Several descriptions listed objects or hand positions while missing visible folding or transfer. Model agreement must not become a “verified” badge. The observations below come from AI-assisted inspection of the exact input frames after outputs were frozen, not independent human annotations.

Preregistration · Complete sanitized results and source/frame hashes · Prompts and request recipes · Next evaluation protocol

diff --git a/docs/papers/dvidia-observation-lab/index.html b/docs/papers/dvidia-observation-lab/index.html index 818eb7d..a5c9f6c 100644 --- a/docs/papers/dvidia-observation-lab/index.html +++ b/docs/papers/dvidia-observation-lab/index.html @@ -1,7 +1,7 @@ DVIDIA Observation Lab pilot · DVIDIA Research -

Local observation workflow and engineering feasibility pilot

DVIDIA Observation Lab pilot

· DVIDIA

Local review release · motion and actuator source pilots · hardware unqualified

Source & history · Download Markdown

Abstract

DVIDIA Observation Lab supplies local object/palm proposals, reviewed timeline revisions and separate causal image-motion forecasts; real annotation and motion accuracy remain unmeasured. A new CPU pilot learns next-20-ms joint/jaw deltas from current native state and issued commands. Twelve seeded groups retain 36 recordings with 8/2/2 train/development/test grouping; 4,553 test transitions are not independent robot trials. The 6,438-byte action model computes at 0.207 ms p95 on one host and improves RMSE, while jaw MAE/worst error regress against persistence and an initial opening is substantially overpredicted. These are privileged simulator measurements and model-only timings. A proposed shoe capture plan has no footage or executed shoe benchmark. The v0.1 release and hosted synthetic review demo remain separate; no shoe policy, physical adapter, validated shoe physics or hardware qualification is established.

In this paper

Engineering integration passed on six public excerpts; annotation accuracy remains unmeasured. On 10 October 2026, we ran a local observation pipeline that decodes video, proposes sampled object and palm boxes, associates nearby detections and opens an editable review timeline. Review changes persist as revisions and export with their source identity. This establishes a working research tool, not reliable automatic task labeling, a cup-placement benchmark or a trained robot skill. The published implementation is anchored to DVIDIA Training commit d355a4a.

+

Local observation workflow and engineering feasibility pilot

DVIDIA Observation Lab pilot

· DVIDIA

Local review release · motion and actuator source pilots · hardware unqualified

Source & history · Download Markdown

Abstract

DVIDIA Observation Lab supplies local object/palm proposals, reviewed timeline revisions and separate causal image-motion forecasts; real annotation and motion accuracy remain unmeasured. A new CPU pilot learns next-20-ms joint/jaw deltas from current native state and issued commands. Twelve seeded groups retain 36 recordings with 8/2/2 train/development/test grouping; 4,553 test transitions are not independent robot trials. The 6,438-byte action model computes at 0.207 ms p95 on one host and improves RMSE, while jaw MAE/worst error regress against persistence and an initial opening is substantially overpredicted. These are privileged simulator measurements and model-only timings. A proposed shoe capture plan has no footage or executed shoe benchmark. The v0.1 release and hosted synthetic review demo remain separate; no shoe policy, physical adapter, validated shoe physics or hardware qualification is established.

In this paper

Engineering integration passed on six public excerpts; annotation accuracy remains unmeasured. On 10 October 2026, we ran a local observation pipeline that decodes video, proposes sampled object and palm boxes, associates nearby detections and opens an editable review timeline. Review changes persist as revisions and export with their source identity. This establishes a working research tool, not reliable automatic task labeling, a cup-placement benchmark or a trained robot skill. The published implementation is anchored to DVIDIA Training commit d355a4a.

Install the Observation Lab release · GitHub release and downloads · Try the synthetic review demo · Frozen engineering receipts

The pilot produces reviewable 2D evidence

The local adapter uses the pinned OpenCV Zoo YOLOX COCO object detector and MediaPipe palm detector. It samples at two frames per second, up to 120 frames and a 640-pixel maximum image side per clip. Proposals retain decoded presentation timestamps, frame and source hashes, raw labels and scores. Same-class image overlap associates detections; palm/object overlap proposes 2D proximity. Neither association establishes physical identity nor overlap proves touching. Palm rectangles are not full-hand segmentation. No optical flow, depth reconstruction, natural-language action captioner or language model participates in this pilot. (Observation contract, frozen model and recipe)

diff --git a/docs/papers/fast-accurate-robot-training-simulation/index.html b/docs/papers/fast-accurate-robot-training-simulation/index.html index ba332b8..85efad1 100644 --- a/docs/papers/fast-accurate-robot-training-simulation/index.html +++ b/docs/papers/fast-accurate-robot-training-simulation/index.html @@ -1,7 +1,7 @@ Fast accurate robot training simulation · DVIDIA Research -

Decision report and local experiment

Fast accurate robot training simulation

· DVIDIA

54 CPU fixture trials · GPU and physical transfer untested

Source & history · Download Markdown

Abstract

A design for an offline, self-hosted training environment that turns demonstrations into calibrated practice and qualified robot skills. We examine contact, deformation, sensor feedback and the performance–accuracy tradeoff, then report 54 deterministic native MuJoCo CPU fixture trials. Fine endpoint agreement did not establish accurate contact transients, and coarse timesteps missed collisions. Minimum GPU memory, rope accuracy and physical transfer remain unmeasured.

In this paper

Decision report · 7 October 2026 · Grok 4.7 research plus primary-source verification · 54 local CPU fixture trials; no GPU training or physical robot tests

+

Decision report and local experiment

Fast accurate robot training simulation

· DVIDIA

54 CPU fixture trials · GPU and physical transfer untested

Source & history · Download Markdown

Abstract

A design for an offline, self-hosted training environment that turns demonstrations into calibrated practice and qualified robot skills. We examine contact, deformation, sensor feedback and the performance–accuracy tradeoff, then report 54 deterministic native MuJoCo CPU fixture trials. Fine endpoint agreement did not establish accurate contact transients, and coarse timesteps missed collisions. Minimum GPU memory, rope accuracy and physical transfer remain unmeasured.

In this paper

Decision report · 7 October 2026 · Grok 4.7 research plus primary-source verification · 54 local CPU fixture trials; no GPU training or physical robot tests

Build an offline, self-hosted environment that converts demonstrations into calibrated practice and qualified robot skills. Start with existing physics engines and concentrate the product on task construction, uncertainty, failure/recovery coverage, efficient scheduling and qualification. Use native MuJoCo for local numerical checks, test MuJoCo Warp for batched rigid contact, and qualify Newton rods or another deformable backend against the actual rope task before selecting it. The microscopic intuition explains why objects resist and deform; the useful simulation interface is the resulting force, motion, friction and sensor response at robot scale. The first product vertical should be bounded rope grasping, pulling, threading and regrasping, with bow tying a later capability. One owned GPU is the starting design target; minimum useful VRAM and an “ultrafast” training rate remain unmeasured. Adding GPUs or servers should scale independent practice worlds, while a released skill retains a declared hardware profile and physical evidence.

Reproduce the response that changes the robot's decision

When a fingertip presses a table, resistance ultimately comes from electromagnetic interactions and quantum constraints on matter. “Electrons reject each other” is a useful intuition but incomplete: atoms also attract and bond, and Pauli exclusion restricts allowed electron states rather than adding a separate classical repulsive force. Molecular structure and collective interactions produce elasticity. We do not need to track electrons to calculate whether a gripper holds a lace; we need the effective mechanical behavior at the force, length and time scales the task uses. (Feynman on forces and elasticity, exclusion and matter)

diff --git a/docs/papers/from-skillspace-to-robot-skill/index.html b/docs/papers/from-skillspace-to-robot-skill/index.html index a4fd0a8..cef0f0f 100644 --- a/docs/papers/from-skillspace-to-robot-skill/index.html +++ b/docs/papers/from-skillspace-to-robot-skill/index.html @@ -1,7 +1,7 @@ From Skillspace to robot skill · DVIDIA Research -

Source review and implementation proposal

From Skillspace to robot skill

· DVIDIA

Graduation and robot training not run

Source & history · Download Markdown

Abstract

An Argus review and a proposal to connect original recordings, reviewed episodes, immutable datasets, training runs, held-out evaluation and robot-tested releases. The review separates video annotation from action supervision and defines compatibility beyond a robot’s degree-of-freedom count. A proposed 40-episode annotation pilot and evidence-based graduation criteria remain unrun.

In this paper

Source review and proposal · 6 October 2026 · Graduation experiment not run

+

Source review and implementation proposal

From Skillspace to robot skill

· DVIDIA

Graduation and robot training not run

Source & history · Download Markdown

Abstract

An Argus review and a proposal to connect original recordings, reviewed episodes, immutable datasets, training runs, held-out evaluation and robot-tested releases. The review separates video annotation from action supervision and defines compatibility beyond a robot’s degree-of-freedom count. A proposed 40-episode annotation pilot and evidence-based graduation criteria remain unrun.

In this paper

Source review and proposal · 6 October 2026 · Graduation experiment not run

Follow-up: the Clef image feasibility pilot has now run on five public excerpts. It does not complete the 40-episode accuracy protocol below.

A Skillspace can become the home for a skill's learning history. More uploaded videos do not, by themselves, establish a better policy. We need evidence that new demonstrations add useful coverage, survive quality checks, and improve held-out performance on a specified robot.

This review follows Varun Nair's pipeline post, its linked Argus article, selected upstream source files, and an audit of the DVIDIA website. The design below is our proposal, not an implemented training service. No user footage was sent to any model, no training was run, and no robot was operated for this review.

diff --git a/docs/papers/grok-robotics-research-on-x/index.html b/docs/papers/grok-robotics-research-on-x/index.html index 1620647..02f0a29 100644 --- a/docs/papers/grok-robotics-research-on-x/index.html +++ b/docs/papers/grok-robotics-research-on-x/index.html @@ -1,7 +1,7 @@ Grok robotics research on X · DVIDIA Research -

Source review and thesis comparison

Grok robotics research on X

· DVIDIA

Primary sources checked · Experiments proposed

Source & history · Download Markdown

Abstract

Three Grok 4.7 research passes surface observation-to-action, synthetic-practice and reusable-repair ideas. Independent primary-source checks compare those findings with our thesis: observation supplies goals and state, language retrieves a skill, spatial expectations guide attention, and executable feedback with evaluation establishes competence. X posts remain discovery leads. No universal demonstration count or arbitrary-robot transfer follows from the reviewed results.

In this paper

Research snapshot: 7 October 2026 · Grok 4.7 used through Pi CLI · No training, simulation or hardware experiment was run

+

Source review and thesis comparison

Grok robotics research on X

· DVIDIA

Primary sources checked · Experiments proposed

Source & history · Download Markdown

Abstract

Three Grok 4.7 research passes surface observation-to-action, synthetic-practice and reusable-repair ideas. Independent primary-source checks compare those findings with our thesis: observation supplies goals and state, language retrieves a skill, spatial expectations guide attention, and executable feedback with evaluation establishes competence. X posts remain discovery leads. No universal demonstration count or arbitrary-robot transfer follows from the reviewed results.

In this paper

Research snapshot: 7 October 2026 · Grok 4.7 used through Pi CLI · No training, simulation or hardware experiment was run

Your initial thesis has substantial support when expressed as observation-derived goals and task state, language-based retrieval, contextual spatial attention, and simulation-assisted learning of a feedback skill. The evidence does not establish that a fixed quantity of video covers every variant, that XYZ determines what to do, or that a simulated policy downloads unchanged onto arbitrary robots. The strongest interpretation of your small-sample idea is adaptation: existing models supply reusable manipulation knowledge, while new demonstrations specify a task, object or operating region. Recent work makes that interpretation more credible, and also exposes its dependencies. Robots can learn through trial and error and retain useful repairs; execution still needs observable outcomes and compatible actions. A fast virtual training environment is consequently a plausible product hypothesis, but the research first needs to establish which observed information makes training more efficient and which simulated improvements survive physical transfer.

The observation-to-skill thesis matches the evidence when each skill has a tested operating region

The following mapping preserves the sequence you described. “Supported” means a component has demonstrated precedents, not that the complete proposed architecture has been demonstrated.

diff --git a/docs/papers/hardware-matched-robot-simulation/index.html b/docs/papers/hardware-matched-robot-simulation/index.html index 229072e..5aedd58 100644 --- a/docs/papers/hardware-matched-robot-simulation/index.html +++ b/docs/papers/hardware-matched-robot-simulation/index.html @@ -1,7 +1,7 @@ Hardware matched robot simulation · DVIDIA Research -

Pinned mechanics implementation and CPU numerical benchmark

Hardware matched robot simulation

· DVIDIA

Pinned CPU mechanics · PiPER contact pilot · physical fidelity unmeasured

Source & history · Download Markdown

Abstract

A separate UR5e/Robotiq reference preserves pinned geometry, inertias, moving-link collision proxies and coupled gripper structure: fourteen modeled coordinates and seven commands. Eighteen deterministic unloaded motion/closure runs are numerically valid; at a 1 ms step, CPU physics throughput measures 40.80–47.29 simulated seconds per measured stepping wall second on one host, excluding loading, sensing, rendering, policy and real actuation. Adjacent timestep comparisons and a matched static overlap expose numerical sensitivity and collision participation. A pure-Python paired trace scorer measures joint, tool and jaw mismatch without automatic qualification. Left/right five-finger references are separately audited; LeRobot collection with an original DVIDIA policy is proposed. Serial calibration, physical response, shoe task learning and bimanual execution remain unmeasured. An offline 3D mesh replay now exposes the recorded arm/gripper poses without rerunning physics. PiPER and SO-101 now have pinned CPU profiles with 6/6 valid unloaded numerical attempts each and explicit native command units. PiPER also has a frozen authored rigid-block contact pilot: nominal placement 2/2, low-friction placement 0/1, open-jaw 0/3 and hold 0/3, with all nine attempts numerically valid. One of 9,000 computation windows exceeds 20 ms; hardware real-time response is unqualified. SO-101 object tasks, learned policies and physical qualification remain open. YAM, OpenArm, ARX X5 and PAROL6 remain upcoming.

In this paper

DVIDIA research · 11 October 2026 · Compiled reference assembly and unloaded CPU diagnostics; physical transfer unmeasured

+

Pinned mechanics implementation and CPU numerical benchmark

Hardware matched robot simulation

· DVIDIA

Pinned CPU mechanics · PiPER contact pilot · physical fidelity unmeasured

Source & history · Download Markdown

Abstract

A separate UR5e/Robotiq reference preserves pinned geometry, inertias, moving-link collision proxies and coupled gripper structure: fourteen modeled coordinates and seven commands. Eighteen deterministic unloaded motion/closure runs are numerically valid; at a 1 ms step, CPU physics throughput measures 40.80–47.29 simulated seconds per measured stepping wall second on one host, excluding loading, sensing, rendering, policy and real actuation. Adjacent timestep comparisons and a matched static overlap expose numerical sensitivity and collision participation. A pure-Python paired trace scorer measures joint, tool and jaw mismatch without automatic qualification. Left/right five-finger references are separately audited; LeRobot collection with an original DVIDIA policy is proposed. Serial calibration, physical response, shoe task learning and bimanual execution remain unmeasured. An offline 3D mesh replay now exposes the recorded arm/gripper poses without rerunning physics. PiPER and SO-101 now have pinned CPU profiles with 6/6 valid unloaded numerical attempts each and explicit native command units. PiPER also has a frozen authored rigid-block contact pilot: nominal placement 2/2, low-friction placement 0/1, open-jaw 0/3 and hold 0/3, with all nine attempts numerically valid. One of 9,000 computation windows exceeds 20 ms; hardware real-time response is unqualified. SO-101 object tasks, learned policies and physical qualification remain open. YAM, OpenArm, ARX X5 and PAROL6 remain upcoming.

In this paper

DVIDIA research · 11 October 2026 · Compiled reference assembly and unloaded CPU diagnostics; physical transfer unmeasured

Build the first simulator around a specific six-axis arm and gripper, then measure the physical response that the learner will encounter. DVIDIA's vision remains a Skillspace that supplies demonstrations, local training and an installable skill. Rendered images support visual learning, but matching a room's appearance and the arm's number of joints does not establish that a learned action will produce the same motion, grip or outcome on hardware. We have compiled a pinned UR5e and Robotiq 2F-85 reference and run eighteen unloaded CPU diagnostics; this does not select the user's eventual purchase or reproduce that robot's controller. Later, two arms with five-finger hands need a different command, sensing and contact contract. Imported structure, numerical characterization and held-out physical agreement remain separate evidence states. Useful speed is the cost of acquiring a qualified capability, measured alongside errors and failures.

Six arm joints do not specify the whole robot

DVIDIA's existing dvidia-authored-6dof-parallel-jaw-v0 environment is an authored rigid-box contact experiment. It has six arm hinges, two simulated sliding jaws and eight native position actuators, exposed through six joint targets and one total-gap target. Arm links, base and palm have collision masks disabled; the experiment therefore does not evaluate whole-arm avoidance or self-collision. Exact object state, contact counts and pad forces are native simulator observations, not measured camera or fingertip signals. Preserve this profile and its frozen results while adding a separately named hardware reference. (Current arm source)

diff --git a/docs/papers/native-grasp-task-benchmark/index.html b/docs/papers/native-grasp-task-benchmark/index.html index 6552b17..65a4026 100644 --- a/docs/papers/native-grasp-task-benchmark/index.html +++ b/docs/papers/native-grasp-task-benchmark/index.html @@ -1,7 +1,7 @@ Native grasp task benchmark · DVIDIA Research -

Native contact task implementation and authored baseline

Native grasp task benchmark

· DVIDIA

6/6 simulated placements · controls 0/12 · real-time and hardware unqualified

Source & history · Download Markdown

Abstract

A pinned six-axis arm and coupled gripper complete 6/6 frozen synthetic rigid-block placements, while matched recorded-arm/open-jaw and hold controls complete 0/12. Native contact must produce grasp, sustained lift, retained carry, release and settled placement with tool retreat. All 18 attempts and timing traces remain in the receipt; one placement takes 29.51 wall seconds for 20 simulated seconds and misses 251 of 1,000 computation windows, so real-time operation is not established. An offline 3D mesh replay exposes the recorded motion. The authored privileged-state baseline provides a benchmark for a future compact DVIDIA action policy, not a learned footage-to-action model, validated shoe simulation or physical robot qualification.

In this paper

11 October 2026 · Native simulation benchmark and next learning experiment

+

Native contact task implementation and authored baseline

Native grasp task benchmark

· DVIDIA

6/6 simulated placements · controls 0/12 · real-time and hardware unqualified

Source & history · Download Markdown

Abstract

A pinned six-axis arm and coupled gripper complete 6/6 frozen synthetic rigid-block placements, while matched recorded-arm/open-jaw and hold controls complete 0/12. Native contact must produce grasp, sustained lift, retained carry, release and settled placement with tool retreat. All 18 attempts and timing traces remain in the receipt; one placement takes 29.51 wall seconds for 20 simulated seconds and misses 251 of 1,000 computation windows, so real-time operation is not established. An offline 3D mesh replay exposes the recorded motion. The authored privileged-state baseline provides a benchmark for a future compact DVIDIA action policy, not a learned footage-to-action model, validated shoe simulation or physical robot qualification.

In this paper

11 October 2026 · Native simulation benchmark and next learning experiment

DVIDIA's native contact benchmark completed 6/6 grasp, carry and release placements across six frozen synthetic layouts; matched open-jaw replay and hold controls completed 0/12. All 18 attempts were numerically valid. The six-axis reference arm, coupled parallel-jaw gripper and freely moving rigid block provide an executable task for a future learned action policy. The first baseline is authored and uses privileged simulator state; no policy has learned this task from footage. CPU execution also showed substantial timing variation, including a 29.51-second attempt for 20 seconds of simulated motion, so the result establishes task behavior under these settings rather than real-time control. Complete receipts, unsuccessful controls and source identities accompany the replay. Shoe organization and physical robot qualification require their own evidence. (Frozen confirmation receipt)

Native contact makes the task measurable

The reference combines six UR5e joint targets and one Robotiq 2F-85 actuator command. Its eight gripper linkage joints move through the upstream coupled mechanism; the action interface is one native jaw target in 0–255, rather than eight independent finger commands or a calibrated force request. The task adds a free-joint block and a support table while preserving the pinned arm/gripper geometry, inertias, active collision proxies and linkage constraints. An authored position controller and gravity feedforward act through the bounded native servos. During the rollout, the object moves through dynamics and contact rather than a weld or runtime pose assignment. The openly licensed simulation assets do not imply open-source UR5e or Robotiq physical hardware. (Frozen task source, hardware reference)

diff --git a/docs/papers/offline-cable-training-environment/index.html b/docs/papers/offline-cable-training-environment/index.html index 20625fd..09a4209 100644 --- a/docs/papers/offline-cable-training-environment/index.html +++ b/docs/papers/offline-cable-training-environment/index.html @@ -1,7 +1,7 @@ Offline cable training environment · DVIDIA Research -

Product prototype and local qualification

Offline cable training environment

· DVIDIA

CPU simulation-only · Controller search and frozen evaluation executed

Source & history · Download Markdown

Abstract

The first runnable product loop combines an anchored cable environment, local controller-parameter search, frozen evaluation, offline replay and a manifest-backed skill pack. The selected controller and a fixed feedback baseline both completed 10/10 held-out simulated episodes; zero-force and release controls completed 0/10. A native macOS network-denial run completed 3/3 additional episodes. Ideal endpoint attachment and privileged state bound the result; physical rope calibration, gripping, sensors and GPU throughput remain untested.

In this paper

Product prototype and local qualification · 7 October 2026 · Native CPU simulation only

+

Product prototype and local qualification

Offline cable training environment

· DVIDIA

CPU simulation-only · Controller search and frozen evaluation executed

Source & history · Download Markdown

Abstract

The first runnable product loop combines an anchored cable environment, local controller-parameter search, frozen evaluation, offline replay and a manifest-backed skill pack. The selected controller and a fixed feedback baseline both completed 10/10 held-out simulated episodes; zero-force and release controls completed 0/10. A native macOS network-denial run completed 3/3 additional episodes. Ideal endpoint attachment and privileged state bound the result; physical rope calibration, gripping, sensors and GPU throughput remain untested.

In this paper

Product prototype and local qualification · 7 October 2026 · Native CPU simulation only

The first runnable product loop now connects an authored environment → local practice → a frozen controller → evaluation → an inspectable skill pack. CableEndReach-v0 bends an anchored cable using bounded forces at its terminal site. The selected controller completed 10/10 held-out simulated episodes, but a fixed feedback baseline also completed 10/10 and finished faster. The result establishes an executable research and packaging foundation, with explicit failure controls. It does not establish physical rope accuracy, shoe tying, realistic grasping or a performance advantage over existing simulators.

Runnable source and instructions · Open the recorded replay · Download the skill pack · Complete qualification

The task has an explicit operating envelope

diff --git a/docs/papers/physical-skill-research-agenda/index.html b/docs/papers/physical-skill-research-agenda/index.html index 1c657a3..43ff3b5 100644 --- a/docs/papers/physical-skill-research-agenda/index.html +++ b/docs/papers/physical-skill-research-agenda/index.html @@ -1,7 +1,7 @@ Turn physical footage into testable evidence · DVIDIA Research -

Research agenda

Turn physical footage into testable evidence

· DVIDIA

Four baseline protocols proposed

Source & history · Download Markdown

Abstract

A research agenda covering visible capture evidence, relations over time, evaluation provenance and physical grounding. Each question has a bounded protocol and a failure condition. The aim is to measure whether structure and physical signals improve a defined task before scaling collection. Strategic opinions, author-reported results and our untested proposals are kept distinct.

In this paper

DVIDIA should test the usefulness of its evidence before expanding collection volume. The immediate opportunity is to make demonstrations easier to inspect, compare and reproduce, then measure whether added structure or physical signals help a defined task. Sam Padilla's 5 October essay provides a strategic prompt; the technical sources support narrower, testable questions. The four baseline protocols proposed in this agenda have not run, and neither customer demand nor robot transfer has been established. Four contributor-sized studies cover capture quality, relations over time, provenance and physical grounding.

+

Research agenda

Turn physical footage into testable evidence

· DVIDIA

Four baseline protocols proposed

Source & history · Download Markdown

Abstract

A research agenda covering visible capture evidence, relations over time, evaluation provenance and physical grounding. Each question has a bounded protocol and a failure condition. The aim is to measure whether structure and physical signals improve a defined task before scaling collection. Strategic opinions, author-reported results and our untested proposals are kept distinct.

In this paper

DVIDIA should test the usefulness of its evidence before expanding collection volume. The immediate opportunity is to make demonstrations easier to inspect, compare and reproduce, then measure whether added structure or physical signals help a defined task. Sam Padilla's 5 October essay provides a strategic prompt; the technical sources support narrower, testable questions. The four baseline protocols proposed in this agenda have not run, and neither customer demand nor robot transfer has been established. Four contributor-sized studies cover capture quality, relations over time, provenance and physical grounding.

The essay motivates a hypothesis, not a market forecast

In Robotics Data: Bet Hard Or Get Out, Sam Padilla, 5 October 2026, argues for exceptional collection scale, distinctive physical information or synthetic generation. He warns that buyers can internalize embodiment-specific collection and points to opportunities beyond raw data supply. His conversations and volume anecdote are not independently verified demand evidence. (Original essay).

Our inference is a practical test: does an attributable demonstration with observable state changes improve a specific evaluation more than a clip with coarse tags? This would establish a useful property of a research artifact. It would not establish willingness to pay, a durable business model or an executable robot skill. The first deliverable should therefore be a small benchmark with a clear failure condition, rather than a volume target.

diff --git a/docs/papers/piper-contact-history-v03/index.html b/docs/papers/piper-contact-history-v03/index.html index 372d508..b3cb23b 100644 --- a/docs/papers/piper-contact-history-v03/index.html +++ b/docs/papers/piper-contact-history-v03/index.html @@ -1,7 +1,7 @@ PiPER contact history v0.3 · DVIDIA Research -

Paired probe-only and contact-history experiment with complete confirmation receipt

PiPER contact history v0.3

· DVIDIA

History 6/12 · baseline and probe-only 4/12 · controls 0/48 · hardware unqualified

Source & history · Download Markdown

Abstract

A frozen paired PiPER experiment retains all 84 numerically valid attempts across twelve synthetic rigid-block scenes. History completes 6/12 placements versus 4/12 for both original motion and the identical acquisition probe without carry monitoring; all 48 separately matched open-jaw and hold controls fail. The two added completions occur at nominal friction with 60 g blocks. Four tasks complete after five closed holds/resumes, but only four of five resume windows meet the independent low absolute TCP-speed criterion. All six supported acquisition retries fail. Native gains, forces and outcome rules remain unchanged; requested movement reduction is distinct from actual speed, and two slowdown windows include early holds. Seventy-five of 84,000 computation windows exceed 20 ms. Complete sealed trajectories, original model notices, source-snapshotted development and pinned offline inspection preserve both success and failure. Privileged authored simulation establishes no learned skill, true-slip detector accuracy, shoe competence or physical qualification.

In this paper

DVIDIA research · 11 October 2026 · Contact history v0.3

+

Paired probe-only and contact-history experiment with complete confirmation receipt

PiPER contact history v0.3

· DVIDIA

History 6/12 · baseline and probe-only 4/12 · controls 0/48 · hardware unqualified

Source & history · Download Markdown

Abstract

A frozen paired PiPER experiment retains all 84 numerically valid attempts across twelve synthetic rigid-block scenes. History completes 6/12 placements versus 4/12 for both original motion and the identical acquisition probe without carry monitoring; all 48 separately matched open-jaw and hold controls fail. The two added completions occur at nominal friction with 60 g blocks. Four tasks complete after five closed holds/resumes, but only four of five resume windows meet the independent low absolute TCP-speed criterion. All six supported acquisition retries fail. Native gains, forces and outcome rules remain unchanged; requested movement reduction is distinct from actual speed, and two slowdown windows include early holds. Seventy-five of 84,000 computation windows exceed 20 ms. Complete sealed trajectories, original model notices, source-snapshotted development and pinned offline inspection preserve both success and failure. Privileged authored simulation establishes no learned skill, true-slip detector accuracy, shoe competence or physical qualification.

In this paper

DVIDIA research · 11 October 2026 · Contact history v0.3

History completes 6/12 placements, versus 4/12 for both original motion and probe-only motion. The two additional completions are the 60 g, nominal-friction cases; all three modes complete both 40 g moderate-friction cases and fail the 60 g moderate-friction and all low-friction cases. All 84 planned attempts remain numerically valid, including 48 unsuccessful negative controls. Four history tasks resume after a closed hold and complete, but all six supported acquisition retries still fail. This is a descriptive result on synthetic rigid blocks with privileged state, not detector accuracy, a learned skill or physical robot qualification. (Complete original report)

The question left by the previous guard

The earlier grip experiment diff --git a/docs/papers/piper-contact-recovery-v04/index.html b/docs/papers/piper-contact-recovery-v04/index.html index ac84f9d..41b499e 100644 --- a/docs/papers/piper-contact-recovery-v04/index.html +++ b/docs/papers/piper-contact-recovery-v04/index.html @@ -1,7 +1,7 @@ PiPER contact recovery v0.4 · DVIDIA Research -

Paired carry-recovery and supported-acquisition experiment with complete confirmation receipt

PiPER contact recovery v0.4

· DVIDIA

All modes 4/12 · two resumed tasks faster · controls 0/48 · hardware unqualified

Source & history · Download Markdown

Abstract

A fresh frozen 84-attempt PiPER comparison completes 4/12 placements for v0.3 history, v0.4 carry-only and v0.4 full; all 48 matched open-jaw and hold controls fail. Two completed post-hold placements finish 2.440 and 2.640 simulated seconds sooner with unchanged 220/240 ms settling holds. Four low-friction failures reach refusal 2.88–2.90 seconds sooner without becoming successes. Fresh nominal-friction 60 g layouts expose two held-but-too-low resume failures in every mode; neither reaches speed restoration. Six earlier acquisition failures are traced to insufficient support under the pinned position servo and side pinch, not the actuator force cap. All native mechanics and outcome criteria remain unchanged. Complete trajectories and 84,000 raw computation windows are retained; none exceeds 20 ms on this host. Source-snapshotted development remains separate. Privileged authored simulation establishes no learned skill, true-slip accuracy or physical qualification.

In this paper

DVIDIA research · 11 October 2026 · Contact recovery v0.4

+

Paired carry-recovery and supported-acquisition experiment with complete confirmation receipt

PiPER contact recovery v0.4

· DVIDIA

All modes 4/12 · two resumed tasks faster · controls 0/48 · hardware unqualified

Source & history · Download Markdown

Abstract

A fresh frozen 84-attempt PiPER comparison completes 4/12 placements for v0.3 history, v0.4 carry-only and v0.4 full; all 48 matched open-jaw and hold controls fail. Two completed post-hold placements finish 2.440 and 2.640 simulated seconds sooner with unchanged 220/240 ms settling holds. Four low-friction failures reach refusal 2.88–2.90 seconds sooner without becoming successes. Fresh nominal-friction 60 g layouts expose two held-but-too-low resume failures in every mode; neither reaches speed restoration. Six earlier acquisition failures are traced to insufficient support under the pinned position servo and side pinch, not the actuator force cap. All native mechanics and outcome criteria remain unchanged. Complete trajectories and 84,000 raw computation windows are retained; none exceeds 20 ms on this host. Source-snapshotted development remains separate. Privileged authored simulation establishes no learned skill, true-slip accuracy or physical qualification.

In this paper

DVIDIA research · 11 October 2026 · Contact recovery v0.4

v0.4 completes the same 4/12 placements as v0.3 on fresh layouts, while two completed recoveries finish 2.440 and 2.640 simulated seconds sooner. The carry-only and full versions have the same completion count; none is gained or lost. Four low-friction failures reach refusal 2.88–2.90 seconds sooner. All 84 planned attempts are numerically valid, including 48 failed negative controls. Eight placements still fail in each mode. Two nominal-friction 60 g cases that succeeded in the older cohort now remain suspended below the unchanged resume-height gate and time out in every mode. This establishes faster progress after settling on two tasks, with continuing acquisition and resume limitations. It is not a learned skill, true-slip accuracy or physical robot qualification. (Complete frozen report)

The six failed placements in contact history v0.3 all fail before acquiring a load-bearing grasp. A faster carry response cannot repair a grip that never lifts its object. Version 0.4 therefore separates two changes: restoring requested movement after a settled carry hold, and avoiding an unchanged supported retry when current observations indicate that it is not helping. Completed placement, successful recovery and faster refusal remain separate results.

Diagnose the six failures first

diff --git a/docs/papers/piper-grip-validation-and-recovery/index.html b/docs/papers/piper-grip-validation-and-recovery/index.html index 269a08c..1f731e7 100644 --- a/docs/papers/piper-grip-validation-and-recovery/index.html +++ b/docs/papers/piper-grip-validation-and-recovery/index.html @@ -1,7 +1,7 @@ PiPER grip validation and recovery · DVIDIA Research -

Paired authored controller experiment and complete confirmation receipt

PiPER grip validation and recovery

· DVIDIA

Moderate-friction regression · baseline 4/6, guarded 2/6 · hardware unqualified

Source & history · Download Markdown

Abstract

A slow test lift and bounded supported retry add recorded acquisition decisions to the pinned PiPER contact pilot, but completion regresses: at moderate friction baseline completes 2/2 and guarded completes 0/2. Both complete 2/2 normal cases; neither completes the two low-friction cases. All 30 attempts remain numerically valid, including 18 failed controls and full-duration aborts. Four qualification decisions are followed by a brief independent following window, yet two tasks still fail; no successful recovery or overall improvement is demonstrated. Actual motion continues after hold requests, and 53/30,000 computation windows exceed 20 ms. Complete compressed receipts, original model notices and pinned offline inspection preserve the evidence. Authored privileged-state simulation establishes no detector accuracy, learned shoe skill, calibrated contact or physical robot qualification.

In this paper

DVIDIA research · 11 October 2026 · Authored native simulation experiment

+

Paired authored controller experiment and complete confirmation receipt

PiPER grip validation and recovery

· DVIDIA

Moderate-friction regression · baseline 4/6, guarded 2/6 · hardware unqualified

Source & history · Download Markdown

Abstract

A slow test lift and bounded supported retry add recorded acquisition decisions to the pinned PiPER contact pilot, but completion regresses: at moderate friction baseline completes 2/2 and guarded completes 0/2. Both complete 2/2 normal cases; neither completes the two low-friction cases. All 30 attempts remain numerically valid, including 18 failed controls and full-duration aborts. Four qualification decisions are followed by a brief independent following window, yet two tasks still fail; no successful recovery or overall improvement is demonstrated. Actual motion continues after hold requests, and 53/30,000 computation windows exceed 20 ms. Complete compressed receipts, original model notices and pinned offline inspection preserve the evidence. Authored privileged-state simulation establishes no detector accuracy, learned shoe skill, calibrated contact or physical robot qualification.

In this paper

DVIDIA research · 11 October 2026 · Authored native simulation experiment

The guarded controller regresses completion at moderate friction: baseline 2/2, guarded 0/2. Both complete 2/2 normal-friction placements and neither completes the two low-friction cases. Across the six paired scenes, placement completion falls from baseline 4/6 to guarded 2/6; all 18 open-jaw/hold controls fail and all 30 attempts remain numerically valid. The new controller exposes acquisition decisions, alarms, one supported retry and refusal, but demonstrates neither a successful recovery nor an overall completion improvement. These are synthetic rigid-block results, not a learned skill or physical robot qualification. (Complete original report)

A pinch is not yet a load-bearing grasp

The earlier PiPER contact pilot completed both nominal placements but failed to lift its slippery block. Both fingers registered contact while the tool moved away from the table and the block stayed supported. The missing decision was whether the object followed the gripper under load. This experiment adds a bounded test lift before the full carry. It preserves the pinned model, native servo settings, force limits, free-body object and independent task scorer. (Earlier pilot)

diff --git a/docs/papers/piper-reliability-v05/index.html b/docs/papers/piper-reliability-v05/index.html index 29d38cd..c264f82 100644 --- a/docs/papers/piper-reliability-v05/index.html +++ b/docs/papers/piper-reliability-v05/index.html @@ -1,7 +1,7 @@ PiPER placement reliability v0.5 · DVIDIA Research -

Frozen paired reliability experiment with complete native evidence

PiPER placement reliability v0.5

· DVIDIA

23/36 revised grasp · 15/36 v0.4 · 8 gained, none lost · hardware unqualified

Source & history · Download Markdown

Abstract

Three frozen fresh PiPER cohorts retain 252 numerically valid attempts and 144 failed negative controls. A fixed diagonal grasp with bounded pose planning completes 23/36 placements versus 15/36 for frozen v0.4 and 18/36 with second-lift recovery alone: eight gained completions and none lost. The fifteen shared successes are 0.772 simulated seconds slower on average. Three lift-only second probes qualify and finish their tasks; the full grasp candidate needs none. Twelve low-friction acquisitions and one later retention failure remain. Every trajectory, native task outcome and raw computation duration is retained, including 1,035 windows exceeding 20 ms and a 3.082-second outlier on a loaded host. Native mechanics and scoring remain unchanged. Separate development keeps 200 earlier attempts, including discarded regressions. Authored privileged simulation establishes no learned skill, real-time guarantee or physical qualification.

In this paper

DVIDIA research · 12 October 2026 · Placement reliability v0.5

+

Frozen paired reliability experiment with complete native evidence

PiPER placement reliability v0.5

· DVIDIA

23/36 revised grasp · 15/36 v0.4 · 8 gained, none lost · hardware unqualified

Source & history · Download Markdown

Abstract

Three frozen fresh PiPER cohorts retain 252 numerically valid attempts and 144 failed negative controls. A fixed diagonal grasp with bounded pose planning completes 23/36 placements versus 15/36 for frozen v0.4 and 18/36 with second-lift recovery alone: eight gained completions and none lost. The fifteen shared successes are 0.772 simulated seconds slower on average. Three lift-only second probes qualify and finish their tasks; the full grasp candidate needs none. Twelve low-friction acquisitions and one later retention failure remain. Every trajectory, native task outcome and raw computation duration is retained, including 1,035 windows exceeding 20 ms and a 3.082-second outlier on a loaded host. Native mechanics and scoring remain unchanged. Separate development keeps 200 earlier attempts, including discarded regressions. Authored privileged simulation establishes no learned skill, real-time guarantee or physical qualification.

In this paper

DVIDIA research · 12 October 2026 · Placement reliability v0.5

The revised grasp completes 23/36 placements, compared with 15/36 for frozen v0.4 and 18/36 with the second lift alone. Across three fresh cohorts, it gains eight completions and loses none, reducing failed placements from 21 to 13 in this narrow synthetic comparison. All 252 attempts are numerically valid; all 144 negative controls fail, with no solver warnings or prohibited native contact ticks. The fifteen shared successes take 0.772 simulated seconds longer on average with the revised grasp. Twelve low-friction acquisitions and one carry-retention failure remain. This is a measured completion improvement with a task-speed tradeoff, not learned competence or physical qualification. (Complete aggregate evidence)

The previous v0.4 recovery experiment made two recovered tasks finish sooner but did not complete more placements. Its failures exposed two different problems: a held object could settle below the original resume-height gate, or a supported object could never follow the initial pinch. Faster refusal did not solve either placement. Version 0.5 tests a separate response to each mechanism and measures the entire task again.

Two changes, one unchanged physical task

diff --git a/docs/papers/robotics-research-pipeline/index.html b/docs/papers/robotics-research-pipeline/index.html index d10c079..97d36c1 100644 --- a/docs/papers/robotics-research-pipeline/index.html +++ b/docs/papers/robotics-research-pipeline/index.html @@ -1,7 +1,7 @@ Robotics research pipeline · DVIDIA Research -

Architecture proposal and staged validation roadmap

Robotics research pipeline

· DVIDIA

Architecture proposal · current simulation alpha · hardware unqualified

Source & history · Download Markdown

Abstract

An updated architecture connects immutable Skillspace recordings, reviewed task-label proposals and synchronized measured telemetry to grouped datasets, separate visual and motor learners, and a proposed local state supervisor. The implemented CPU baseline retains authored phases, gripper decisions and recovery around its learned joint-delta head. New annotation and supervisor pilots, fresh frozen confirmation, physics calibration and compatible skill-installation gates define the next milestones. Automatic labeling, learned supervision, human-video action recovery and physical arm qualification remain unestablished; this publication adds no new training or hardware result.

In this paper

DVIDIA's product goal is a local training environment that converts Skillspace evidence into a versioned skill a compatible robot can install, test and execute with feedback. The next architectural step couples reviewed task descriptions to measured robot state, allowing a skill to decide whether a grasp worked, whether progress stopped and whether to retry or abstain. This paper specifies that pipeline and its validation gates. It adds no trained supervisor, human-video action bridge or hardware qualification. The implemented foundation remains a CPU research alpha with bounded simulation evidence. (Public training package, Published measurements)

+

Architecture proposal and staged validation roadmap

Robotics research pipeline

· DVIDIA

Architecture proposal · current simulation alpha · hardware unqualified

Source & history · Download Markdown

Abstract

An updated architecture connects immutable Skillspace recordings, reviewed task-label proposals and synchronized measured telemetry to grouped datasets, separate visual and motor learners, and a proposed local state supervisor. The implemented CPU baseline retains authored phases, gripper decisions and recovery around its learned joint-delta head. New annotation and supervisor pilots, fresh frozen confirmation, physics calibration and compatible skill-installation gates define the next milestones. Automatic labeling, learned supervision, human-video action recovery and physical arm qualification remain unestablished; this publication adds no new training or hardware result.

In this paper

DVIDIA's product goal is a local training environment that converts Skillspace evidence into a versioned skill a compatible robot can install, test and execute with feedback. The next architectural step couples reviewed task descriptions to measured robot state, allowing a skill to decide whether a grasp worked, whether progress stopped and whether to retry or abstain. This paper specifies that pipeline and its validation gates. It adds no trained supervisor, human-video action bridge or hardware qualification. The implemented foundation remains a CPU research alpha with bounded simulation evidence. (Public training package, Published measurements)

Pipeline at a glance

DVIDIA robotics pipeline: source evidence branches into reviewed label proposals and synchronized numerical telemetry; grouped datasets feed separate visual and motor learners, a proposed state supervisor, frozen simulation evaluation and compatible skill installation. Runtime observations return to bounded decisions and future reviewed recordings.
DVIDIA robotics pipeline: source evidence branches into reviewed label proposals and synchronized numerical telemetry; grouped datasets feed separate visual and motor learners, a proposed state supervisor, frozen simulation evaluation and compatible skill installation. Runtime observations return to bounded decisions and future reviewed recordings.

Figure 1. Solid elements represent implemented components within the bounded simulation pipeline; dashed elements represent proposed integrations. Physical robot execution remains unqualified. A visual model, reviewed label and executable motor policy have different evidence requirements. The diagram describes a target architecture, not a newly completed end-to-end system.

diff --git a/docs/papers/sensor-corroboration-v04/index.html b/docs/papers/sensor-corroboration-v04/index.html index 24ad5aa..4697bfd 100644 --- a/docs/papers/sensor-corroboration-v04/index.html +++ b/docs/papers/sensor-corroboration-v04/index.html @@ -1,7 +1,7 @@ Independent camera and actuator corroboration v0.4 · DVIDIA Research -

Paired synthetic benchmark for camera and drive evidence

Independent camera and actuator corroboration v0.4

· DVIDIA

272 cases · 69/112 false completions prevented · 43 remain · 73/80 true completions abstained

Source & history · Download Markdown

Abstract

A fresh eight-layout PiPER simulation benchmark adds separately processed RGB and drive checks to a local physical-goal verifier. The checks prevent 69 of 112 false completion readings while leaving 43 replay, identity, orientation, support and sampling blind spots; the conservative gate abstains on 73 of 80 baseline true completions, including all 64 deliberate sensor-fault controls. The measured CPU path is 22.1 ms median and 31.5 ms p95, excluding 200 ms acquisition. All raw channels and results are preserved; this is synthetic evidence, not hardware validation.

In this paper

DVIDIA robotics research · 19 October 2026.

+

Paired synthetic benchmark for camera and drive evidence

Independent camera and actuator corroboration v0.4

· DVIDIA

272 cases · 69/112 false completions prevented · 43 remain · 73/80 true completions abstained

Source & history · Download Markdown

Abstract

A fresh eight-layout PiPER simulation benchmark adds separately processed RGB and drive checks to a local physical-goal verifier. The checks prevent 69 of 112 false completion readings while leaving 43 replay, identity, orientation, support and sampling blind spots; the conservative gate abstains on 73 of 80 baseline true completions, including all 64 deliberate sensor-fault controls. The measured CPU path is 22.1 ms median and 31.5 ms p95, excluding 200 ms acquisition. All raw channels and results are preserved; this is synthetic evidence, not hardware validation.

In this paper

DVIDIA robotics research · 19 October 2026.

The local physical-goal verifier v0.3 trusts its observation packet. This experiment adds separately processed RGB and PiPER drive channels, then asks whether those channels can contradict a primary “complete” reading. It is a simulator study of evidence fusion, not a hardware safety claim.

The frozen benchmark contains 272 cases across eight layouts: 96 native scenes, 80 primary-packet forgeries, 64 sensor faults and 32 cross-channel controls. All 96 native cases generated validly. The v0.3 baseline called 192 cases complete, including 112 physical false completions. The v0.4 gate retained 50 complete verdicts: seven physical positives and 43 false completions. It prevented 69/112 baseline false completions, but 43 remained. It also abstained on 73/80 baseline true completions; 64 had deliberately missing, stale, gapped or malformed auxiliary evidence, and nine were physically complete but visually unobservable.

The result is useful because it shows both what a second signal catches and what two weak signals cannot establish. The camera detects large motion, position disagreement and many target-visibility failures. The drive path checks finger aperture, command, motor effort, settling and servo consistency. Neither authenticates its source. These channels still miss a 180° orientation reversal, a green lookalike, elevated support, motion that aliases between samples, and refreshed or fully replayed channels.

diff --git a/docs/papers/shoe-asset-library-v01/index.html b/docs/papers/shoe-asset-library-v01/index.html index e66a35d..17dbfd4 100644 --- a/docs/papers/shoe-asset-library-v01/index.html +++ b/docs/papers/shoe-asset-library-v01/index.html @@ -1,7 +1,7 @@ Reusable shoe asset library v0.1 · DVIDIA Research -

Versioned native scene assets and frozen passive-contact benchmark

Reusable shoe asset library v0.1

· DVIDIA

Six shoe proxies · three arms · support 27/27, disabled table 0/27 · manipulation unqualified

Source & history · Download Markdown

Abstract

Six original MIT rigid shoe proxies compose with the pinned UR5e/Robotiq, PiPER and SO-101 models through versioned scene descriptors and offline native 3D previews. A frozen 54-attempt test records 27/27 supported pairs and 0/27 support successes when table collisions are disabled; all 54 gravity checks pass without numerical warnings. No loaded non-table shoe contacts appear in the 20 ms samples, which cannot rule out shorter transients. All sampled trajectories and 27,000 native-step durations remain available. On one Apple M5 host, 54 simulated seconds take 12.3616 wall seconds including fresh model construction; the slowest attempt takes 6.7047 seconds. Geometry and physical properties are authored assumptions. Loading, gravity and passive support do not establish shoe grasping, calibrated deformation, a learned policy or hardware transfer.

In this paper

DVIDIA research · 12 October 2026 · Asset library v0.1

+

Versioned native scene assets and frozen passive-contact benchmark

Reusable shoe asset library v0.1

· DVIDIA

Six shoe proxies · three arms · support 27/27, disabled table 0/27 · manipulation unqualified

Source & history · Download Markdown

Abstract

Six original MIT rigid shoe proxies compose with the pinned UR5e/Robotiq, PiPER and SO-101 models through versioned scene descriptors and offline native 3D previews. A frozen 54-attempt test records 27/27 supported pairs and 0/27 support successes when table collisions are disabled; all 54 gravity checks pass without numerical warnings. No loaded non-table shoe contacts appear in the 20 ms samples, which cannot rule out shorter transients. All sampled trajectories and 27,000 native-step durations remain available. On one Apple M5 host, 54 simulated seconds take 12.3616 wall seconds including fresh model construction; the slowest attempt takes 6.7047 seconds. Geometry and physical properties are authored assumptions. Loading, gravity and passive support do not establish shoe grasping, calibrated deformation, a learned policy or hardware transfer.

In this paper

DVIDIA research · 12 October 2026 · Asset library v0.1

The first DVIDIA asset library loads six original rigid shoe proxies into native simulation with UR5e/Robotiq, PiPER or SO-101. A versioned scene file records the exact catalog, robot, shoe identities and timestep. Users can browse the native geometry in 3D, download a selected pair's scene descriptor, and reproduce it locally using dvidia-assets. This release supplies reusable scene components and passive contact tests. A shoe-organizing policy is a subsequent task.

Explore the asset library · Hugging Face lab · Every benchmark attempt

What a shoe asset contains

diff --git a/docs/papers/skill-capsule-acquisition/index.html b/docs/papers/skill-capsule-acquisition/index.html index 9bd9d6e..443c4ed 100644 --- a/docs/papers/skill-capsule-acquisition/index.html +++ b/docs/papers/skill-capsule-acquisition/index.html @@ -1,7 +1,7 @@ Skill capsule acquisition and optimization · DVIDIA Research -

Learned movement artifact and installation prototype

Skill capsule acquisition and optimization

· DVIDIA

Simulation-only · teacher/student 7/8 · no overall speedup

Source & history · Download Markdown

Abstract

A compact learned CPU movement head replaces joint-target computation inside an authored placement supervisor. Teacher and student both pass 7/8 in a new frozen evaluation, while episode wall time increases from 6.273 to 6.360 seconds. A 39,845-byte JSON capsule imports into the local lab, checks its exact runtime and qualifies the current scene through native placement and a recorded-arm/open-jaw control. Scene changes require a fresh test. One known scene and repeat pass native network denial; perception, complete task learning, other embodiments and hardware remain unqualified.

In this paper

Measured prototype and proposed next steps · 7 October 2026 · Simulation-only

+

Learned movement artifact and installation prototype

Skill capsule acquisition and optimization

· DVIDIA

Simulation-only · teacher/student 7/8 · no overall speedup

Source & history · Download Markdown

Abstract

A compact learned CPU movement head replaces joint-target computation inside an authored placement supervisor. Teacher and student both pass 7/8 in a new frozen evaluation, while episode wall time increases from 6.273 to 6.360 seconds. A 39,845-byte JSON capsule imports into the local lab, checks its exact runtime and qualifies the current scene through native placement and a recorded-arm/open-jaw control. Scene changes require a fresh test. One known scene and repeat pass native network denial; perception, complete task learning, other embodiments and hardware remain unqualified.

In this paper

Measured prototype and proposed next steps · 7 October 2026 · Simulation-only

DVIDIA now has a compact learned movement artifact that can be packaged for a compatible simulated arm. A normalized RBF head replaces the teacher's joint-target computation; task phases, waypoints, gripper/contact supervision, and bounded recovery remain authored. The frozen head contains 34,171 canonical JSON bytes, and the compiled candidate capsule contains 39,845 bytes. In a new eight-scene evaluation, teacher and student both completed 7/8, without teacher fallback, while total episode wall time increased from 6.273 to 6.360 seconds. This establishes a bounded teacher-to-student acquisition step, with neither reliability improvement nor overall acceleration demonstrated. The useful product path is import → compatibility check → actual local execution and negative control → scene-specific qualification → execution. Human-video learning, calibrated perception/contact, and physical-arm competence remain separate future capabilities. (Training receipt, profiling, candidate capsule).

The acquired behavior is movement inside an authored task

The experiment declares 16 training scenes, four separate development-selection scenes, and eight evaluation scenes with disjoint scene digests. Historical v0.3 held-out cases were excluded and their earlier results remain unchanged. The teacher completed 16/16 collection scenes, generating 6,344 supervised rows: 3,172 sampled native teacher calls and 3,172 extra teacher queries with perturbed Cartesian goals at the same physical states. Augmentation adds movement labels, not another 3,172 physical attempts. The full canonical dataset occupies 3,814,171 bytes. (Protocol, collection).

diff --git a/docs/papers/skillspace-arm-benchmarks/index.html b/docs/papers/skillspace-arm-benchmarks/index.html index 2f3f7e8..c59ed3a 100644 --- a/docs/papers/skillspace-arm-benchmarks/index.html +++ b/docs/papers/skillspace-arm-benchmarks/index.html @@ -1,7 +1,7 @@ Skillspace arm benchmarks and evolution · DVIDIA Research -

Paired simulation benchmark and product assessment

Skillspace arm benchmarks and evolution

· DVIDIA

Mixed held-out result · 30/32 baseline, 29/32 updated

Source & history · Download Markdown

Abstract

Two controller revisions improve development completion from 14/20 to 17/20, but the frozen paired test declines from 30/32 to 29/32: both pass all 24 normal-strength cases while actuator-stress completion falls from 6/8 to 5/8. Instrumented CPU throughput also declines. We publish the regression, unchanged contact and success rules, exact controller snapshots, richer physical inputs, runtime-bound installation and a proposed staged Skillspace contract. Contact forces and penetration remain uncalibrated; learned perception and physical qualification are open.

In this paper

Research and implementation assessment · 7 October 2026 · Simulation-only evidence

+

Paired simulation benchmark and product assessment

Skillspace arm benchmarks and evolution

· DVIDIA

Mixed held-out result · 30/32 baseline, 29/32 updated

Source & history · Download Markdown

Abstract

Two controller revisions improve development completion from 14/20 to 17/20, but the frozen paired test declines from 30/32 to 29/32: both pass all 24 normal-strength cases while actuator-stress completion falls from 6/8 to 5/8. Instrumented CPU throughput also declines. We publish the regression, unchanged contact and success rules, exact controller snapshots, richer physical inputs, runtime-bound installation and a proposed staged Skillspace contract. Contact forces and penetration remain uncalibrated; learned perception and physical qualification are open.

In this paper

Research and implementation assessment · 7 October 2026 · Simulation-only evidence

Skillspace → install → simulated arm makes sense as a grounded, compatible execution workflow. The present implementation imports task metadata, binds an explicitly authored adapter, supplies a metric scene, executes native contact dynamics, and checks completion. Two engineering rounds increased development success from 14/20 to 16/20 to 17/20, but the frozen held-out comparison declined from 30/32 to 29/32: both controllers passed all 24 nominal cases, while actuator-stress success fell from 6/8 to 5/8. There were zero held-out improvements and one regression. Observed instrumented CPU throughput also declined from 12.283× to 11.388× simulated seconds per wall second. The result establishes an auditable simulation workflow and useful failure diagnostics; it establishes neither improved held-out reliability nor calibrated physical accuracy. Skillspaces should evolve toward immutable releases that bind task, embodiment, observations, controller, and independently evaluated evidence. (Frozen assessment).

Installation supplies an explicit execution bridge

The public place_cup envelope contains a task label, robot identifier, recording format, and demonstration count. It does not contain learned weights, actions, object dimensions, metric target coordinates, or camera calibration. The local bridge supplies a declared rigid-box proxy and contact-pick-place-box-v0 adapter for dvidia-authored-6dof-parallel-jaw-v0. Thus installation selects existing authored behavior; it does not infer cup manipulation from the referenced demonstrations. This distinction preserves the useful architecture: language identifies the goal, grounding supplies the state and physical assumptions, a compatible controller acts, and a verifier judges the result. (Public metadata, execution guide).

diff --git a/docs/papers/skillspace-arm-installation/index.html b/docs/papers/skillspace-arm-installation/index.html index 3232ffd..ba06b47 100644 --- a/docs/papers/skillspace-arm-installation/index.html +++ b/docs/papers/skillspace-arm-installation/index.html @@ -1,7 +1,7 @@ Installing a Skillspace on a simulated arm · DVIDIA Research -

Product prototype and native contact qualification

Installing a Skillspace on a simulated arm

· DVIDIA

Public skill link to authored adapter · 6/6 scene placements

Source & history · Download Markdown

Abstract

The first link-to-arm product path installs an existing DVIDIA placement envelope, binds a local authored controller to one simulated arm, and reuses it in independently grounded scenes. Native jaw contacts carry a free box; six layouts complete, while no-motion and identical joint commands with open jaws each complete zero of six. Installed execution passes process-level offline checks. Task metadata contains no learned policy; perception, cup geometry, full arm collisions and physical qualification remain open.

In this paper

Research note, 7 October 2026. The current bridge makes a narrow part of the proposed skill workflow executable: import a DVIDIA skill envelope, install an explicitly authored local simulation adapter, create a scene, and run an articulated arm. It provides a concrete place to investigate observation-to-simulation-to-skill packaging. It does not yet infer an executable policy, object geometry, material properties, or robot calibration from human video.

+

Product prototype and native contact qualification

Installing a Skillspace on a simulated arm

· DVIDIA

Public skill link to authored adapter · 6/6 scene placements

Source & history · Download Markdown

Abstract

The first link-to-arm product path installs an existing DVIDIA placement envelope, binds a local authored controller to one simulated arm, and reuses it in independently grounded scenes. Native jaw contacts carry a free box; six layouts complete, while no-motion and identical joint commands with open jaws each complete zero of six. Installed execution passes process-level offline checks. Task metadata contains no learned policy; perception, cup geometry, full arm collisions and physical qualification remain open.

In this paper

Research note, 7 October 2026. The current bridge makes a narrow part of the proposed skill workflow executable: import a DVIDIA skill envelope, install an explicitly authored local simulation adapter, create a scene, and run an articulated arm. It provides a concrete place to investigate observation-to-simulation-to-skill packaging. It does not yet infer an executable policy, object geometry, material properties, or robot calibration from human video.

The public example is DVIDIA place_cup, whose metadata is served at place_cup/skill.json. The recorded 360-byte source is vdia:0, version 0.3.0, with task “Set a cup on the mark.” Its exact-byte SHA-256 is 16b4e1ef0f3799345c1d18ce9f1da83b072863ab00ce30a83424080fb467d4ec. The envelope declares a demonstration count, recording format and robot identifier; these declarations do not supply trajectories, observations, trained weights, or an evaluated controller. The installation receipt preserves the source bytes, URL, source hash, separate grounding hash, adapter identity and physical_robot_ready: false. A later download may differ; its receipt must identify its own bytes.

In the local studio, one click installs the compatible public envelope and binds contact-pick-place-box-v0 to dvidia-authored-6dof-parallel-jaw-v0. A bundled envelope supports installation without fetching the public site. JSON upload/drop accepts the supported DVIDIA formats. Missing grounding is supplied through a visible, locally authored adapter; inline grounding remains separately validated. Installation downloads metadata and executes the already installed local controller. Object and target coordinates belong to the scene and can change between executions. The browser displays recorded native simulation poses rather than inventing an object trajectory. Execution and replay are designed to work with installed dependencies and local assets; the broader product's air-gapped deployment remains a separate validation task.

The embodiment is an original six-hinge arm with two actuated sliding jaws. Native MuJoCo 3.15.0 simulates a free rigid box, gravity and jaw/object/table contact. The default proxy is a 40 mm cube weighing 40 g on a table at 0.29 m. No weld, mocap attachment, or object-state update carries the object during execution. Arm controls use bounded position servos, model-based bias compensation and current-state damped Jacobian control. The declared limits are 40 N·m per arm actuator and 15 N per jaw. Physics uses a 1 ms timestep, implicitfast integration, Newton solving and an elliptic Coulomb friction cone. These choices describe an authored fixture, not measured hardware or calibrated material fidelity.

diff --git a/docs/papers/skillspace-footage-training/index.html b/docs/papers/skillspace-footage-training/index.html index 5ed18e0..d7948d3 100644 --- a/docs/papers/skillspace-footage-training/index.html +++ b/docs/papers/skillspace-footage-training/index.html @@ -1,7 +1,7 @@ Skillspace footage training pipeline · DVIDIA Research -

Product implementation and measured synthetic pilot

Skillspace footage training pipeline

· DVIDIA

Public CPU package · one native scene qualified · simulation-only

Source & history · Download Markdown

Abstract

The public Dvidia-Inference/dvidia-training repository provides a standalone CPU pipeline that audits Skillspace media, freezes grouped splits and exports inspectable artifacts. Ten synthetic six-second clips train a visual predictor with a 0.5106-second median pipeline time across three repeats on one macOS host. Aligned native simulator telemetry supplies 4,029 training labels for a learned joint-delta head, evaluated on one held-out recording. Its capsule passes native placement qualification in that recording's exact scene; identical joint targets with open jaws fail. Installation and hardware guides accompany the package. Human-video action recovery, shoe organization, demonstration sufficiency, independent scene generalization and physical robot competence remain unestablished.

In this paper

Research pipeline evolution — 9 October 2026. The Robotics research pipeline adds the proposed annotation-review and measured-state task-supervisor layers around this implemented training path. It includes an architecture diagram, data contracts and separate validation gates. Those additions are proposals; the recorded pilots and frozen sample-count results below retain their original scope.

+

Product implementation and measured synthetic pilot

Skillspace footage training pipeline

· DVIDIA

Public CPU package · one native scene qualified · simulation-only

Source & history · Download Markdown

Abstract

The public Dvidia-Inference/dvidia-training repository provides a standalone CPU pipeline that audits Skillspace media, freezes grouped splits and exports inspectable artifacts. Ten synthetic six-second clips train a visual predictor with a 0.5106-second median pipeline time across three repeats on one macOS host. Aligned native simulator telemetry supplies 4,029 training labels for a learned joint-delta head, evaluated on one held-out recording. Its capsule passes native placement qualification in that recording's exact scene; identical joint targets with open jaws fail. Installation and hardware guides accompany the package. Human-video action recovery, shoe organization, demonstration sufficiency, independent scene generalization and physical robot competence remain unestablished.

In this paper

Research pipeline evolution — 9 October 2026. The Robotics research pipeline adds the proposed annotation-review and measured-state task-supervisor layers around this implemented training path. It includes an architecture diagram, data contracts and separate validation gates. Those additions are proposals; the recorded pilots and frozen sample-count results below retain their original scope.

The public, standalone DVIDIA training repository provides a local CPU pipeline that turns actual Skillspace media into a fitted visual model and, when aligned numeric actions are supplied, a learned movement head. The useful advance is a reproducible path from source files through provenance checks, grouped splits, measured fitting and portable artifacts. Video prediction and robot movement remain separate claims. Ten synthetic six-second clips demonstrate visual fitting; simulator telemetry supplies the movement pilot's action labels. Its exported capsule installs and passes native qualification in one exact scene, while identical joint targets with open jaws fail. Neither experiment learns shoe organization or reconstructs robot actions from human footage. This report documents the implemented path and its limited measured evidence, rather than establishing a sufficient demonstration count or hardware competence. (Standalone source archive, Qualification receipt)

Published artifacts on Hugging Face

The DVIDIA organization on Hugging Face mirrors the measured pilot for inspection and local reproduction:

diff --git a/docs/papers/spatially-grounded-downloadable-robot-skills/index.html b/docs/papers/spatially-grounded-downloadable-robot-skills/index.html index 7680843..274328d 100644 --- a/docs/papers/spatially-grounded-downloadable-robot-skills/index.html +++ b/docs/papers/spatially-grounded-downloadable-robot-skills/index.html @@ -1,7 +1,7 @@ Spatially grounded downloadable robot skills · DVIDIA Research -

Research synthesis and proposed experiments

Spatially grounded downloadable robot skills

· DVIDIA

Source review · DVIDIA experiments not run

Source & history · Download Markdown

Abstract

A synthesis of learning from human demonstrations, language-conditioned spatial attention, simulation expansion and robot-specific skill delivery. Spatial priors can accelerate inspection but do not determine contact state or control. The proposed downloadable skill packages a controller, initiation and termination conditions, compatible sensing and hardware, recovery behavior and measured evidence. Separate experiments would test attention, data coverage, feedback and transfer.

In this paper

Research synthesis and proposed experiments · 7 October 2026 · No DVIDIA experiment in this report has run

+

Research synthesis and proposed experiments

Spatially grounded downloadable robot skills

· DVIDIA

Source review · DVIDIA experiments not run

Source & history · Download Markdown

Abstract

A synthesis of learning from human demonstrations, language-conditioned spatial attention, simulation expansion and robot-specific skill delivery. Spatial priors can accelerate inspection but do not determine contact state or control. The proposed downloadable skill packages a controller, initiation and termination conditions, compatible sensing and hardware, recovery behavior and measured evidence. Separate experiments would test attention, data coverage, feedback and transfer.

In this paper

Research synthesis and proposed experiments · 7 October 2026 · No DVIDIA experiment in this report has run

My judgment is that the thesis identifies a useful architecture, but its strongest version needs a measurable operating boundary. Human demonstrations can teach what a task means, which state changes matter, and where to inspect; turning those observations into robot control needs an action and embodiment bridge. A command-conditioned spatial prior could accelerate target discovery, provided observation corrects it when a shoe is on a table or the person changes posture. Simulation can expand grounded experience, but generated volume cannot establish coverage of every material, contact, damaged object, or recovery. Fifty examples could adapt a substantial pretrained physical skill model; they are not an established threshold for acquiring all of shoe tying. The falsifiable research claim is: for a fixed pretrained model and robot, task-state annotations, spatially directed inspection, and targeted real/simulated recovery data improve held-out physical success per unit of collection cost, while preserving performance on unusual placements. A downloadable skill is the resulting tested controller, its compatibility and task contract, and evidence of where it works.

This synthesis combines three parallel primary-source research tracks with the existing DVIDIA research context. Grok 4.7 was inaccessible and was not used: the inspected browser was signed out, the exact model could not be verified, and no prompt or response exists. Recent 2026 results below include author-reported preprints; none was independently reproduced here. Architecture, experiments, budgets, and release criteria are proposals.

Demonstrations teach meaning before they supply motor control

diff --git a/docs/publications.json b/docs/publications.json index cc8269d..4bee32a 100644 --- a/docs/publications.json +++ b/docs/publications.json @@ -5,6 +5,20 @@ "follow": "https://x.com/iammrriver", "contributor": null, "publications": [ + { + "slug": "chat-to-robot-skills", + "title": "From a conversation to compatible robot skills", + "source": "reports/Chat to compatible robot skills.md", + "date": "2026-10-27", + "type": "Product architecture and local runtime verification", + "status": "Photo inference and one native reset verified · OpenVLA actions and steering not run", + "topics": [ + "physical-grounding", + "skill-relations", + "evaluation-provenance" + ], + "abstract": "A conversation can carry a photo and desired outcome through scene review, skill retrieval, compatible installation, simulation and inspectable action traces. The local prototype uses ten LIBERO-10 task manifests sharing a pinned OpenVLA checkpoint. Real SSDlite inference and a repeatable native task-5 reset are recorded; the checkpoint transfer stopped after an 8 MiB partial file, so learned policy actions and activation-steered behavior remain unrun. The note separates observed evidence, candidate relevance, executable compatibility and measured completion, and defines staged evaluation for the chat workflow and raw MLP channel interventions. General household planning and hardware execution remain unqualified." + }, { "slug": "learned-piper-grip-v02", "title": "Learning grip feedback with a frozen PiPER arm reference", diff --git a/docs/sitemap.xml b/docs/sitemap.xml index c5b70ff..2ad9018 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -1,2 +1,2 @@ -https://research.dvidia.org/2026-10-27https://research.dvidia.org/papers/learned-piper-grip-v02/2026-10-27https://research.dvidia.org/papers/learned-piper-placement-v01/2026-10-27https://research.dvidia.org/papers/sensor-corroboration-v04/2026-10-19https://research.dvidia.org/papers/local-goal-verifier-v03/2026-10-19https://research.dvidia.org/papers/image-goal-jev-v02/2026-10-19https://research.dvidia.org/papers/image-goal-jev-v01/2026-10-18https://research.dvidia.org/papers/goal-images-physical-feedback/2026-10-18https://research.dvidia.org/papers/shoe-graspability-v03/2026-10-14https://research.dvidia.org/papers/continuous-shoe-grip-v02/2026-10-13https://research.dvidia.org/papers/skill-prerequisites-v01/2026-10-13https://research.dvidia.org/papers/live-piper-shoe-workstation-v01/2026-10-13https://research.dvidia.org/papers/shoe-asset-library-v01/2026-10-12https://research.dvidia.org/papers/piper-reliability-v05/2026-10-12https://research.dvidia.org/papers/piper-contact-recovery-v04/2026-10-11https://research.dvidia.org/papers/piper-contact-history-v03/2026-10-11https://research.dvidia.org/papers/piper-grip-validation-and-recovery/2026-10-11https://research.dvidia.org/papers/native-grasp-task-benchmark/2026-10-11https://research.dvidia.org/papers/hardware-matched-robot-simulation/2026-10-11https://research.dvidia.org/papers/dvidia-observation-lab/2026-10-11https://research.dvidia.org/papers/robotics-research-pipeline/2026-10-09https://research.dvidia.org/papers/skillspace-footage-training/2026-10-08https://research.dvidia.org/papers/skill-capsule-acquisition/2026-10-07https://research.dvidia.org/papers/skillspace-arm-benchmarks/2026-10-07https://research.dvidia.org/papers/skillspace-arm-installation/2026-10-07https://research.dvidia.org/papers/offline-cable-training-environment/2026-10-07https://research.dvidia.org/papers/fast-accurate-robot-training-simulation/2026-10-07https://research.dvidia.org/papers/grok-robotics-research-on-x/2026-10-07https://research.dvidia.org/papers/spatially-grounded-downloadable-robot-skills/2026-10-07https://research.dvidia.org/papers/clef-and-grounded-video-observations/2026-10-06https://research.dvidia.org/papers/from-skillspace-to-robot-skill/2026-10-06https://research.dvidia.org/papers/physical-skill-research-agenda/2026-10-06https://research.dvidia.org/topics/capture-quality/2026-10-11https://research.dvidia.org/topics/skill-relations/2026-10-27https://research.dvidia.org/topics/evaluation-provenance/2026-10-27https://research.dvidia.org/topics/physical-grounding/2026-10-27 +https://research.dvidia.org/2026-10-27https://research.dvidia.org/papers/chat-to-robot-skills/2026-10-27https://research.dvidia.org/papers/learned-piper-grip-v02/2026-10-27https://research.dvidia.org/papers/learned-piper-placement-v01/2026-10-27https://research.dvidia.org/papers/sensor-corroboration-v04/2026-10-19https://research.dvidia.org/papers/local-goal-verifier-v03/2026-10-19https://research.dvidia.org/papers/image-goal-jev-v02/2026-10-19https://research.dvidia.org/papers/image-goal-jev-v01/2026-10-18https://research.dvidia.org/papers/goal-images-physical-feedback/2026-10-18https://research.dvidia.org/papers/shoe-graspability-v03/2026-10-14https://research.dvidia.org/papers/continuous-shoe-grip-v02/2026-10-13https://research.dvidia.org/papers/skill-prerequisites-v01/2026-10-13https://research.dvidia.org/papers/live-piper-shoe-workstation-v01/2026-10-13https://research.dvidia.org/papers/shoe-asset-library-v01/2026-10-12https://research.dvidia.org/papers/piper-reliability-v05/2026-10-12https://research.dvidia.org/papers/piper-contact-recovery-v04/2026-10-11https://research.dvidia.org/papers/piper-contact-history-v03/2026-10-11https://research.dvidia.org/papers/piper-grip-validation-and-recovery/2026-10-11https://research.dvidia.org/papers/native-grasp-task-benchmark/2026-10-11https://research.dvidia.org/papers/hardware-matched-robot-simulation/2026-10-11https://research.dvidia.org/papers/dvidia-observation-lab/2026-10-11https://research.dvidia.org/papers/robotics-research-pipeline/2026-10-09https://research.dvidia.org/papers/skillspace-footage-training/2026-10-08https://research.dvidia.org/papers/skill-capsule-acquisition/2026-10-07https://research.dvidia.org/papers/skillspace-arm-benchmarks/2026-10-07https://research.dvidia.org/papers/skillspace-arm-installation/2026-10-07https://research.dvidia.org/papers/offline-cable-training-environment/2026-10-07https://research.dvidia.org/papers/fast-accurate-robot-training-simulation/2026-10-07https://research.dvidia.org/papers/grok-robotics-research-on-x/2026-10-07https://research.dvidia.org/papers/spatially-grounded-downloadable-robot-skills/2026-10-07https://research.dvidia.org/papers/clef-and-grounded-video-observations/2026-10-06https://research.dvidia.org/papers/from-skillspace-to-robot-skill/2026-10-06https://research.dvidia.org/papers/physical-skill-research-agenda/2026-10-06https://research.dvidia.org/topics/capture-quality/2026-10-11https://research.dvidia.org/topics/skill-relations/2026-10-27https://research.dvidia.org/topics/evaluation-provenance/2026-10-27https://research.dvidia.org/topics/physical-grounding/2026-10-27 diff --git a/docs/topics/capture-quality/index.html b/docs/topics/capture-quality/index.html index 5071a30..b745d17 100644 --- a/docs/topics/capture-quality/index.html +++ b/docs/topics/capture-quality/index.html @@ -1,7 +1,7 @@ Capture quality · DVIDIA Research -

Research question

Capture quality

DVIDIA · Visible evidence

Protocols and progress are scoped below

Question

Test whether hands, relevant objects and state changes are actually observable in a POV demonstration.

Question: Test whether hands, relevant objects and state changes are actually observable in a POV demonstration.

+

Research question

Capture quality

DVIDIA · Visible evidence

Protocols and progress are scoped below

Question

Test whether hands, relevant objects and state changes are actually observable in a POV demonstration.

Question: Test whether hands, relevant objects and state changes are actually observable in a POV demonstration.

Status: scoping · Updated 6 October 2026 · Results: not run

Open a research question · Baseline protocol · Bibliography

Primary sources

diff --git a/docs/topics/evaluation-provenance/index.html b/docs/topics/evaluation-provenance/index.html index 057bc4c..781852c 100644 --- a/docs/topics/evaluation-provenance/index.html +++ b/docs/topics/evaluation-provenance/index.html @@ -1,7 +1,7 @@ Evaluation and provenance · DVIDIA Research -

Research question

Evaluation and provenance

DVIDIA · Reproducible evidence

Protocols and progress are scoped below

Question

Test episode readback, original-to-derivative lineage and split integrity using small, auditable fixtures.

Question: Test episode readback, original-to-derivative lineage and split integrity using small, auditable fixtures.

+

Research question

Evaluation and provenance

DVIDIA · Reproducible evidence

Protocols and progress are scoped below

Question

Test episode readback, original-to-derivative lineage and split integrity using small, auditable fixtures.

Question: Test episode readback, original-to-derivative lineage and split integrity using small, auditable fixtures.

Status: evaluating · Updated 6 October 2026 · Results: image feasibility run complete; reference accuracy and provenance baselines not run

Open a research question · Baseline protocol · Bibliography

Primary sources

diff --git a/docs/topics/physical-grounding/index.html b/docs/topics/physical-grounding/index.html index e6434bd..a1ff8c6 100644 --- a/docs/topics/physical-grounding/index.html +++ b/docs/topics/physical-grounding/index.html @@ -1,7 +1,16 @@ Physical grounding · DVIDIA Research -

Research question

Physical grounding

DVIDIA · Signals beyond appearance

Protocols and progress are scoped below

Question

Measure whether state/contact evidence and controlled synthetic augmentation improve a defined physical-skill task.

Live PiPER shoe workstation — 13 October 2026

+

Research question

Physical grounding

DVIDIA · Signals beyond appearance

Protocols and progress are scoped below

Question

Measure whether state/contact evidence and controlled synthetic augmentation improve a defined physical-skill task.

Chat to compatible robot skills — 27 October 2026

+

From a conversation to compatible robot skills +proposes one conversation from photo and desired outcome to grounded skill +selection, installation, simulation and trace inspection. The local prototype +records real object detection and one repeated native task-5 reset. An interrupted +8 MiB partial checkpoint is not an installation; OpenVLA actions, steering +measurements, general household planning and hardware execution remain unrun. +The runtime archive preserves the exact +receipt, actual reset image and a documentation-only skill manifest.

+

Live PiPER shoe workstation — 13 October 2026

Live PiPER shoe workstation v0.1 adds browser-local MuJoCo physics, a fixed-orientation target controller, native jaw commands and a free rigid trainer on a physical worktop. Users can move a diff --git a/docs/topics/skill-relations/index.html b/docs/topics/skill-relations/index.html index 9a12c11..5a9c9e4 100644 --- a/docs/topics/skill-relations/index.html +++ b/docs/topics/skill-relations/index.html @@ -1,7 +1,7 @@ Skill relations · DVIDIA Research -

Research question

Skill relations

DVIDIA · Events with evidence

Protocols and progress are scoped below

Question

Calibrate hand–object relations, then test temporal event graphs whose claims point back to video intervals.

Question: Calibrate hand–object relations, then test temporal event graphs whose claims point back to video intervals.

+

Research question

Skill relations

DVIDIA · Events with evidence

Protocols and progress are scoped below

Question

Calibrate hand–object relations, then test temporal event graphs whose claims point back to video intervals.

Question: Calibrate hand–object relations, then test temporal event graphs whose claims point back to video intervals.

Status: scoping · Updated 6 October 2026 · Results: not run

Open a research question · Baseline protocol · Bibliography

Primary sources

diff --git a/publications.json b/publications.json index cc8269d..4bee32a 100644 --- a/publications.json +++ b/publications.json @@ -5,6 +5,20 @@ "follow": "https://x.com/iammrriver", "contributor": null, "publications": [ + { + "slug": "chat-to-robot-skills", + "title": "From a conversation to compatible robot skills", + "source": "reports/Chat to compatible robot skills.md", + "date": "2026-10-27", + "type": "Product architecture and local runtime verification", + "status": "Photo inference and one native reset verified · OpenVLA actions and steering not run", + "topics": [ + "physical-grounding", + "skill-relations", + "evaluation-provenance" + ], + "abstract": "A conversation can carry a photo and desired outcome through scene review, skill retrieval, compatible installation, simulation and inspectable action traces. The local prototype uses ten LIBERO-10 task manifests sharing a pinned OpenVLA checkpoint. Real SSDlite inference and a repeatable native task-5 reset are recorded; the checkpoint transfer stopped after an 8 MiB partial file, so learned policy actions and activation-steered behavior remain unrun. The note separates observed evidence, candidate relevance, executable compatibility and measured completion, and defines staged evaluation for the chat workflow and raw MLP channel interventions. General household planning and hardware execution remain unqualified." + }, { "slug": "learned-piper-grip-v02", "title": "Learning grip feedback with a frozen PiPER arm reference", diff --git a/reports/Chat to compatible robot skills.md b/reports/Chat to compatible robot skills.md new file mode 100644 index 0000000..352a99f --- /dev/null +++ b/reports/Chat to compatible robot skills.md @@ -0,0 +1,233 @@ +# Chat to compatible robot skills + +DVIDIA research · 27 October 2026 · Product architecture and local runtime note + +Original writing: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Referenced models, code and assets retain their own terms. + +The intended interface is a conversation: attach a photo, describe a desired +outcome, and let DVIDIA find an executable skill for the available robot. The +conversation should carry the workflow from observation through installation, +simulation and review. A user should not need to assemble those tools manually. + +The current prototype connects photo analysis and conservative skill retrieval +to a separate local worker for ten LIBERO-10 task descriptions sharing one +revision-pinned OpenVLA checkpoint. **Real local object detection and one native +simulator reset are verified. OpenVLA model inference, learned robot actions and +activation-steered robot actions have not been executed.** The checkpoint +transfer stopped after an 8 MiB partial download; this is not an installation or +a successful task. The [runtime receipt](../topics/physical-grounding/runs/2026-10-27-robot-lab/runtime-receipt.json) +and [evidence notes](../topics/physical-grounding/runs/2026-10-27-robot-lab/README.md) +preserve that boundary. A companion [Hugging Face concept card](https://huggingface.co/spaces/Dvidia/robot-simulation-lab/blob/main/docs/robot-chat-v01/concept.md) +tracks the same documentation scope; it does not host OpenVLA inference. + +## Implementation follow-up — 28 October 2026 + +The local prototype now implements the photo/outcome composer and conversation +cards. A unique compatible unblocked task starts verified installation and real +scene preparation; the user sees the simulator camera before choosing Run in +simulation. Connection settings, object corrections and action/neuron inspection +are expandable. This is a typed tool workflow with conservative lexical +retrieval; the conversation does not add a general language planner. + +Nineteen focused chat/client/planner tests and 42 worker tests pass. New +regressions reject reversed spatial destinations and wrong target colors. +Installation cancellation races and explicit stop-tracking recovery are handled. +Browser control initially timed out, then recovered in a fresh tab. Development +and built desktop previews verified real photo detection, an unsupported +dishwashing reply, actual task-5 camera capture, unique compatible retrieval and +live installation progress. Browser error logs were empty. Phone-sized layout +acceptance remains pending. The repaired downloader has resumed the real +checkpoint transfer, but no completed installation, learned action or steering +result is established. The 26 October receipt below remains unchanged. + +![Local conversational prototype showing actual simulator input and an incomplete policy download](../topics/physical-grounding/runs/2026-10-27-robot-lab/chat-interface.png) + +The screenshot shows installation progress, not robot execution. + +## One conversation carries the task + +A useful request is “Put the book in the back compartment of the caddy,” with a +photo attached. DVIDIA needs to identify the intended object and destination, +look for a compatible behavior, check its prerequisites and expose any missing +evidence. The product should present these as progress in one conversation: + +| Conversation event | What it represents | +| --- | --- | +| Photo and desired outcome | User input, including explicit corrections | +| “I found a book; please identify its destination” | A grounded question when the evidence is insufficient | +| Skill card with prerequisites and compatibility | A candidate, with reasons it can or cannot run | +| Download progress and verified installation | Actual artifact state, including interruption and retry | +| Simulator preview and run progress | A real task reset followed by actual policy actions when available | +| Outcome and inspectable trace | Success, failure or cancellation with actions and observations | +| “Try this intervention” | An experimental rerun with the same initial simulator state | + +This is the conversational simplification implemented in the local prototype. +Detailed connection settings, manifests, action vectors and channel controls can +live in expandable cards. The conversation remains an interface to typed tools; +assistant prose cannot mark an artifact installed or a task completed. + +“Tidy this room” expresses a valid desired outcome but leaves important +decisions unresolved: where toys belong, which clothes are dirty, whether a +container is reachable and whether the robot has the required behaviors. A photo +does not answer all of these questions. The first version should ask for missing +destinations or state evidence and report unsupported tasks. General household +task decomposition and automatic selection of multi-skill sequences are future +work. No dishwashing, laundry or toy-sorting competence is established by this +prototype. + +## Retrieve by intended change, then check compatibility + +The architecture connects five distinct decisions: + +```text +Photo + desired outcome + -> observations with source and uncertainty + -> candidate skills with prerequisites and effects + -> compatible artifact + verified installation + -> actual simulator state + policy execution + -> measured outcome + inspectable trace +``` + +An object name is a useful search clue, but the intended change determines +whether a skill is relevant. A mug-placement task does not satisfy a request to +wash a mug. A drawer task that closes the drawer adds an unwanted effect if the +user asked for it to remain open. The current browser planner uses bounded +lexical matching and aliases; it checks object identities, destinations, +relations, state changes and negation, then rejects incompatible profiles. It is +not a learned affordance estimator or a general language planner. + +This separation is motivated by [SayCan](https://say-can.github.io/), which +combines language-based skill relevance with state-dependent affordances. Our +current rule-based filter does not reproduce SayCan's learned value functions +or its reported performance. The design implication is to keep goal relevance, +observed prerequisites and runtime compatibility independently inspectable. + +Every executable candidate currently carries a skill ID, instruction, required +objects, effects, prerequisites, profile ID, checkpoint repository, immutable +revision and benchmark task ID. The present profile is +`libero-panda-openvla-v1`: a Franka Panda simulation using seven-dimensional OSC +pose actions. Its checkpoint is +[`openvla/openvla-7b-finetuned-libero-10`](https://huggingface.co/openvla/openvla-7b-finetuned-libero-10/tree/80970322773f81baa2e22fe495d0487b93a05cfa). +The ten task entries share the same checkpoint; downloading each task does not +create ten separate learned policies. + +The [example v1 manifest](../topics/physical-grounding/runs/2026-10-27-robot-lab/skill-manifest.example.json) +is documentation only and contains no installed artifact. + +A broader skill registry will also need versioned observation and action +schemas, units, control rate, gripper convention, tested operating envelope, +required sensors, adapter versions and task-specific evaluation receipts. +Hardware compatibility would require its own tested adapter and qualification. +Those extended registry fields are a proposed contract, not an implemented +universal skill package. + +## Execution belongs to the local worker + +The browser communicates with an isolated loopback worker using the typed +`dvidia.robot-lab.v1` protocol and an in-memory capability. The worker performs +real detector calls, artifact verification, native resets, model loading and +bounded runs. It contains no physical robot driver. Checkpoint size, revision +and digests are verified before atomic activation; partial files never count as +an installed skill. The runtime loads explicit classes from pinned OpenVLA +source rather than executing a checkpoint's remote class mapping. + +An arbitrary household photo is planning input. It is not an executable LIBERO +state. Before a run, the user must select a supported benchmark task and capture +its real simulator camera and initial state. Task metadata is marked as +simulator evidence, distinct from image detections and user corrections. A run +is bound to that task, scene, seed and installed checkpoint. Completion must +come from the simulator's task-success signal. + +[OpenVLA](https://github.com/openvla/openvla) supplies a language-conditioned +robot action policy. The selected [checkpoint card](https://huggingface.co/openvla/openvla-7b-finetuned-libero-10) +describes a LIBERO-10 fine-tune; it is a concrete starting point for this +benchmark-specific experiment, not a generally installable housekeeping model. +The runtime uses `predict_action` with the `libero_10` normalization statistics. +Code, checkpoint and underlying model terms need separate review; the upstream +MIT labels do not replace the applicable Llama 2 terms or third-party asset +licenses. + +## What has actually run + +The retained receipt was recorded on 26 October 2026 and is published here as a +historical engineering result. It is not a fresh benchmark of this note's date. + +| Component | Recorded evidence | Limit | +| --- | --- | --- | +| Local runtime | Python 3.12.9, Torch 2.6.0 and MPS detected on an Apple Silicon Mac | Full OpenVLA memory fit and latency unverified | +| Real photo inference | Verified TorchVision SSDlite checkpoint; cup 0.9957, person 0.8254 and dining table 0.4730 on one editorial image | Confidence scores are detector outputs, not measured accuracy; no dirt, color, ownership or task-state judgment | +| Native task reset | LIBERO-10 task 5, seed 7, book/caddy camera image and flattened-state digest | Zero policy actions; ten no-op settling steps only | +| Reset repeatability | Worker-thread and independent main-thread reset digests match | Matching initial state does not establish deterministic inference or rollouts | +| Artifact installation | Interrupted transfer retained 8 MiB of a roughly 15.1 GB checkpoint | No activated checkpoint | +| Policy and steering | Transport and hook fixtures tested | No learned rollout, task result or behavioral steering measurement | + +The reset digest is +`b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e`. +The macOS worker uses MuJoCo's official offscreen CGL context to avoid initializing +GLFW/AppKit from an HTTP worker thread. This changes context selection, not the +physics or observed state. Only the task-5 asset closure was provisioned for the +recorded runtime check; other task assets remain locally unverified. + +![Actual LIBERO task-5 reset camera, before any policy action](../topics/physical-grounding/runs/2026-10-27-robot-lab/libero-task5-reset.png) + +The photograph used for detector testing was an editorial illustration from the +app, not a physical robot camera capture. The simulator image above is an actual +rendered reset. Neither is evidence that a robot performed the requested action. + +## Neuron exploration is a separate measured intervention + +[Lyra's neuron explorer](https://github.com/lyramakesmusic/neuron-explorer) +inspired the interaction: inspect model units, change a unit's activation and +retain the resulting behavior. Its text-model experiments do not by themselves +establish a robot control method. The directly relevant robotics precedent is +[Mechanistic Interpretability for Steering Vision Language Action Models](https://vla-mech-interp.github.io/), +which studies semantic activation directions and interventions in OpenVLA and +π₀. Its [research code](https://github.com/Physical-AI-Safety-Institute/mechanistic-steering-vlas) +is a methodological reference; the DVIDIA worker uses an original hook and does +not vendor that toolkit. + +The implemented hook observes a zero-based intermediate Llama MLP channel at the +input to `down_proj`. It records the last token's value on each autoregressive +forward pass and can add a bounded raw dose to that channel. The displayed +per-action value averages the pre-intervention values across generation passes. +This is **a raw channel intervention**, not an identified “careful,” “fast” or +“move left” neuron. It is also not an implementation-equivalent reproduction of +the paper's semantic cluster interventions. Hook cleanup and cancellation have +unit coverage; their effects on a loaded robot policy remain unmeasured. + +Before behavioral claims, compare an unsteered baseline and intervention on the +same checkpoint, task, seed, preprocessing and exact reset digest. Record all +action dimensions, end-effector motion, activation summaries, cameras, +completion and failures. Current portable Pillow preprocessing is explicitly a +numerical variant of upstream's TensorFlow path, so published OpenVLA benchmark +scores are not a performance reference for this Mac configuration. + +## Staged evaluation and release gates + +1. **Conversation and retrieval:** freeze a small set of supported and + unsupported requests, including ambiguous destinations, negation and desired + state changes. Independently label compatible candidates. Report retrieval + precision and recall, unsupported-task handling and clarification behavior. + An interaction that looks correct is not evidence of execution. +2. **Installation and one-action diagnostic:** finish the revision-pinned + transfer, verify every artifact, load the full model and execute one real + action on the recorded task-5 reset. Record peak memory, load/inference time, + actual action vector and failure status. Stop if memory or rendering is + insufficient; a substitute controller does not satisfy this gate. +3. **Baseline simulation:** run bounded unsteered episodes on a frozen set of + seeds. Retain every attempt and the actual LIBERO success signal. Report + strict completion, step counts, latency and invalid attempts separately. +4. **Causal channel study:** choose units and doses using development scenes, + then freeze them. Compare paired baseline, sham-hook, positive/negative-dose + and randomly selected channel controls on held-out seeds. Measure task + completion alongside motion changes; do not select only successful videos. +5. **Skill sequencing and household transfer:** introduce separately qualified + skills, evidence refresh after each action, explicit recovery and destination + rules. Hardware execution needs independent adapter, sensing and task + qualification. This stage is not implemented or run. + +Stages 2–5 remain unrun. The first deliverable is an honest conversation that +connects to real tools and makes missing capability understandable. The research +question is whether that interface can select a skill whose measured behavior +achieves the user's intended change, and whether controlled interventions alter +that behavior reproducibly without degrading completion. diff --git a/topics/physical-grounding/README.md b/topics/physical-grounding/README.md index 7235a81..ae82743 100644 --- a/topics/physical-grounding/README.md +++ b/topics/physical-grounding/README.md @@ -1,5 +1,16 @@ # Physical grounding +## Chat to compatible robot skills — 27 October 2026 + +[From a conversation to compatible robot skills](../../reports/Chat%20to%20compatible%20robot%20skills.md) +proposes one conversation from photo and desired outcome to grounded skill +selection, installation, simulation and trace inspection. The local prototype +records real object detection and one repeated native task-5 reset. An interrupted +8 MiB partial checkpoint is not an installation; OpenVLA actions, steering +measurements, general household planning and hardware execution remain unrun. +[The runtime archive](runs/2026-10-27-robot-lab/README.md) preserves the exact +receipt, actual reset image and a documentation-only skill manifest. + ## Live PiPER shoe workstation — 13 October 2026 diff --git a/topics/physical-grounding/runs/2026-10-27-robot-lab/README.md b/topics/physical-grounding/runs/2026-10-27-robot-lab/README.md new file mode 100644 index 0000000..9904fcb --- /dev/null +++ b/topics/physical-grounding/runs/2026-10-27-robot-lab/README.md @@ -0,0 +1,68 @@ +# Photo, outcome and compatible robot skills + +Publication date: 27 October 2026. Runtime receipt: 26 October 2026. + +This archive supports [Chat to compatible robot skills](../../../../reports/Chat%20to%20compatible%20robot%20skills.md). +It contains real local runtime checks and an actual simulator reset, with no +learned policy rollout or steering result. + +- [Runtime receipt](runtime-receipt.json): sanitized dependency versions, + upstream source revisions, detector outputs, task/seed, reset digest and + original worker source digests. +- [Example v1 skill manifest](skill-manifest.example.json): current catalog fields + for task 5, marked unavailable until an actual runtime checks compatibility. + It bundles no policy and is not an installation. +- [Native task-5 reset camera](libero-task5-reset.png): 256 × 256 image rendered + after ten no-op settling steps, before any learned action. + +The receipt is copied byte-for-byte from the app's original engineering evidence. +Its source digests describe that snapshot, not later installer or interface +revisions. The separate test fixtures recorded in it are explicitly synthetic; +the detector inference and simulator reset are actual runtime observations. +No private capability, photo payload or account credential is included. + +## Reproduction boundary + +The public upstream sources are pinned as follows: + +| Artifact | Revision | +| --- | --- | +| OpenVLA source | `c8f03f48af692657d3060c19588038c7220e9af9` | +| LIBERO source | `8f1084e3132a39270c3a13ebe37270a43ece2a01` | +| `openvla/openvla-7b-finetuned-libero-10` | `80970322773f81baa2e22fe495d0487b93a05cfa` | + +The model revision identifies the intended checkpoint; its weights were not +installed for this receipt. The interrupted transfer retained an 8 MiB partial +file and no activation marker. Only the task-5 study/book/caddy asset closure +was fetched. A complete standalone worker release is separate from this +documentation archive; these two evidence files are not an installable skill. + +The reset uses task order index 0, LIBERO-10 task index 5 and seed 7. It performs +ten no-op settling steps and hashes the flattened simulator state. The worker +thread and independent main-thread run produced the same digest: + +```text +b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e +``` + +Apple MPS availability does not prove the full model fits or executes correctly. +The camera is simulation evidence, not physical hardware evidence. All upstream +model, simulator and asset terms remain applicable. + +## Hugging Face publication + +[Publication receipt](huggingface-publication.json) records the documentation-only +commit, exact file hashes and anonymous readback. The update preserves all 3,771 +pre-existing files except the root card, which receives one additional concept +section. It adds no model weights, live inference endpoint or hardware control. + +[Chat implementation follow-up](huggingface-followup.json) records the second +documentation-only update. The [interface screenshot](chat-interface.png) shows +the local built prototype with an actual simulator camera and a still-incomplete +checkpoint transfer. Development and built desktop browser workflows were +verified after the initial browser-control interruption; no policy action ran. + +[Final documentation and screenshot revision](huggingface-final.json) adds the +verified desktop interface image and updates the implementation status. All +three files pass anonymous readback at the recorded immutable Hub commit; all +3,774 other existing Space files remain unchanged. diff --git a/topics/physical-grounding/runs/2026-10-27-robot-lab/chat-interface.png b/topics/physical-grounding/runs/2026-10-27-robot-lab/chat-interface.png new file mode 100644 index 0000000..c5bbf63 Binary files /dev/null and b/topics/physical-grounding/runs/2026-10-27-robot-lab/chat-interface.png differ diff --git a/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-final.json b/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-final.json new file mode 100644 index 0000000..b3b0533 --- /dev/null +++ b/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-final.json @@ -0,0 +1,25 @@ +{ + "repo": "Dvidia/robot-simulation-lab", + "parent": "f4ae2b2cdce4a4a40359fc44d4dd1c99fb954dd9", + "files": [ + { + "path": "docs/robot-chat-v01/concept.md", + "bytes": 14785, + "sha256": "92c653b5c8aa674cf4cbd9f10e0f6d59d97a8011649ce9be4309fd5a6ed8e4a9" + }, + { + "path": "README.md", + "bytes": 41682, + "sha256": "88672953b3bb44095c14a96d86aa698de40c2aa3df357568f115ff8c4a38d14c" + }, + { + "path": "docs/robot-chat-v01/chat-interface.png", + "bytes": 92631, + "sha256": "4246dbb3f82cebee62ef4678e5712c3de7869e2c1a8dcf5aa54bf07de68e08a6" + } + ], + "commit": "a2e4b8a90fa2160e0ba0cfa51a668de5fef6233b", + "anonymous_readback_verified": true, + "unchanged_existing_files": 3774, + "scope": "concept, root documentation and interface screenshot only; no live inference, model weights or hardware" +} diff --git a/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-followup.json b/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-followup.json new file mode 100644 index 0000000..18757ef --- /dev/null +++ b/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-followup.json @@ -0,0 +1,20 @@ +{ + "repo": "Dvidia/robot-simulation-lab", + "parent": "9e6b5e36e96d15d5d456f4362f548972e46c3373", + "files": [ + { + "path": "docs/robot-chat-v01/concept.md", + "bytes": 14369, + "sha256": "95b10be488051d3ae4fdaa33a577cd6517ea1ff4b912615b43feb37cb61e3e96" + }, + { + "path": "README.md", + "bytes": 41628, + "sha256": "4d6d130bbfadeb073cbbc26dc004e6f0dadb77fa4391acd0c6fd44cd9373e1b4" + } + ], + "commit": "f4ae2b2cdce4a4a40359fc44d4dd1c99fb954dd9", + "anonymous_readback_verified": true, + "unchanged_existing_files": 3774, + "scope": "two documentation files only; no live inference, model weights or hardware" +} diff --git a/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-publication.json b/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-publication.json new file mode 100644 index 0000000..e8158bf --- /dev/null +++ b/topics/physical-grounding/runs/2026-10-27-robot-lab/huggingface-publication.json @@ -0,0 +1,40 @@ +{ + "repo": "Dvidia/robot-simulation-lab", + "type": "space", + "parent": "9eaeeae67def2697ca664b54a754e915f0741c78", + "files": [ + { + "path": "README.md", + "bytes": 41454, + "sha256": "2467367a33640553b528937339a3700fd9fed62f24b03e76c77799639a40f3b9" + }, + { + "path": "docs/robot-chat-v01/concept.md", + "bytes": 13191, + "sha256": "c36d5556b7b176a3b93e0dcfc7f2ba1920806a24dc513811863e2b96de2ae5b7" + }, + { + "path": "docs/robot-chat-v01/runtime-evidence.json", + "bytes": 4868, + "sha256": "edb0ac36726120f0f65b94770183af14a0cc9234f69f212a7d383fb231260252" + }, + { + "path": "docs/robot-chat-v01/libero-reset.png", + "bytes": 62199, + "sha256": "c7bbaad0bfc572f01917f477ec03cf6149c0fb24993a6829ae74c4b0fa1fd95f" + }, + { + "path": "docs/robot-chat-v01/skill-manifest.example.json", + "bytes": 891, + "sha256": "70a4800ba1bafb3198e51de0dfe5379f029025507fc14236adf09c4dcfb5b060" + } + ], + "preserve_existing_files": 3772, + "scope": "documentation only; no live inference or hardware", + "commit": "9e6b5e36e96d15d5d456f4362f548972e46c3373", + "url": "https://huggingface.co/spaces/Dvidia/robot-simulation-lab/commit/9e6b5e36e96d15d5d456f4362f548972e46c3373", + "anonymous_readback_verified": true, + "unchanged_existing_files": 3771, + "live_inference": false, + "model_weights_uploaded": false +} diff --git a/topics/physical-grounding/runs/2026-10-27-robot-lab/libero-task5-reset.png b/topics/physical-grounding/runs/2026-10-27-robot-lab/libero-task5-reset.png new file mode 100644 index 0000000..4188d37 Binary files /dev/null and b/topics/physical-grounding/runs/2026-10-27-robot-lab/libero-task5-reset.png differ diff --git a/topics/physical-grounding/runs/2026-10-27-robot-lab/runtime-receipt.json b/topics/physical-grounding/runs/2026-10-27-robot-lab/runtime-receipt.json new file mode 100644 index 0000000..f464b45 --- /dev/null +++ b/topics/physical-grounding/runs/2026-10-27-robot-lab/runtime-receipt.json @@ -0,0 +1,152 @@ +{ + "recordedAt": "2026-10-26T07:00:08.623063+00:00", + "scope": "Local Mac runtime verification; no OpenVLA weights or learned rollout", + "profileId": "libero-panda-openvla-v1", + "model": { + "checkpoint": "openvla/openvla-7b-finetuned-libero-10", + "revision": "80970322773f81baa2e22fe495d0487b93a05cfa", + "weightsInstalled": false, + "inferenceExecuted": false, + "steeredRobotActionsExecuted": false + }, + "sources": { + "openvlaRevision": "c8f03f48af692657d3060c19588038c7220e9af9", + "liberoRevision": "8f1084e3132a39270c3a13ebe37270a43ece2a01", + "liberoAssetScope": "Only task 5 study/book/caddy asset closure was fetched; other task assets unverified" + }, + "runtime": { + "python": "3.12.9", + "mpsAvailable": true, + "torch": "2.6.0", + "torchvision": "0.21.0", + "numpy": "1.26.4", + "Pillow": "13.0.0", + "transformers": "4.40.1", + "tokenizers": "0.19.1", + "timm": "0.9.10", + "accelerate": "0.30.1", + "huggingface-hub": "0.23.0", + "safetensors": "0.4.5", + "mujoco": "3.1.1", + "robosuite": "1.4.1", + "bddl": "1.0.1", + "gym": "0.25.2", + "cloudpickle": "2.1.0", + "easydict": "1.9", + "imageio": "2.38.1", + "PyYAML": "6.0.3", + "sentencepiece": "0.2.2", + "numba": "0.68.0", + "llvmlite": "0.50.0", + "scipy": "1.17.1", + "mpsContext": "Host graphics access; sandboxed process reports unavailable", + "future": "1.0.0", + "matplotlib": "3.10.8", + "contourpy": "1.3.2" + }, + "sourceClasses": { + "verified": true, + "classes": [ + "OpenVLAConfig", + "OpenVLAForActionPrediction", + "PrismaticProcessor" + ], + "modelInstantiated": false + }, + "photoDetection": { + "verified": true, + "image": "public/skill-cup.jpg", + "imageSize": [ + 1024, + 768 + ], + "detector": "TorchVision SSDlite320 MobileNetV3 COCO", + "checkpointSha256": "a79551df90c79834bcd3bb3845ef9d966b5449a3a9b2833ae8404778ca5d65d2", + "elapsedSeconds": 22.981, + "objects": [ + { + "id": "detected-0", + "label": "cup", + "confidence": 0.9956732392311096, + "box": [ + 0.39903393387794495, + 0.47789963086446124, + 0.6025710105895996, + 0.7762130896250407 + ], + "source": "detector" + }, + { + "id": "detected-1", + "label": "person", + "confidence": 0.8253621459007263, + "box": [ + 0.4701840579509735, + 0.1547549068927765, + 1, + 0.7927147547403971 + ], + "source": "detector" + }, + { + "id": "detected-2", + "label": "dining table", + "confidence": 0.4729897677898407, + "box": [ + 0, + 0.5605224768320719, + 1, + 0.9700164794921875 + ], + "source": "detector" + } + ], + "limitations": "Coarse detector hypotheses, no color/dirt/ownership/task-state inference" + }, + "simulatorReset": { + "verified": true, + "taskId": 5, + "seed": 7, + "instruction": "pick up the book and place it in the back compartment of the caddy", + "resetHash": "b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e", + "image": "docs/evidence/robot-lab-libero-reset-20261026.png", + "imageSize": [ + 256, + 256 + ], + "actualState": [ + -0.29572729160672406, + 0.013003376967992303, + 1.1680005404829354, + 0.9995841690598055, + 0.0017969395174486705, + -0.02877830619454855, + 0.0002628040344518339, + 0.03872207014699441, + -0.038723508260440485 + ], + "elapsedSeconds": 15.78, + "threaded": true, + "policyActionsExecuted": 0, + "settlingNoopSteps": 10, + "renderer": "Official mujoco.cgl.GLContext via original constructor adapter for robosuite1.4.1", + "independentMainThreadResetHash": "b133849e34e709aaecf9f42e1a281d9dc53dfeeebf01d5e41077914b8d12893e", + "repeatResetHashMatched": true, + "cameraVisuallyInspected": true, + "initialMacFailureResolved": "GLFW initialized AppKit off the main thread; official offscreen CGL adapter removes that path" + }, + "workerTests": { + "command": "python -m unittest discover -s robot-lab/tests -q", + "python": "3.12.9", + "passed": 30, + "failed": 0, + "fixtureScope": "Unit/protocol/transport/hook tests include explicit synthetic fixtures; separate simulator and detector results are actual runtime measurements" + }, + "workerSourceSha256": { + "runtime.py": "5ff4db2097335304e755a82c299a3521830b3f3e9bc545f95dbea5f1161df07f", + "server.py": "acc0f9e5f2ad873df47feae801094696463df66ce9d388a9fe057516a4d74e76", + "artifacts.py": "0567fd4c4ccf369e8fef2574e643daa7df32b6a05e0a06c2ee7941cade05ebe6", + "catalog.py": "dbc9916f9ec403507e1f305eb31d7c7b4f3be4e67807d85fa7aedb347e45f595", + "requirements.txt": "d30b0762abaedf377dcac22270a87a895f1e587c6d241bcb9705011b05d25eac" + } +} diff --git a/topics/physical-grounding/runs/2026-10-27-robot-lab/skill-manifest.example.json b/topics/physical-grounding/runs/2026-10-27-robot-lab/skill-manifest.example.json new file mode 100644 index 0000000..4daa472 --- /dev/null +++ b/topics/physical-grounding/runs/2026-10-27-robot-lab/skill-manifest.example.json @@ -0,0 +1,26 @@ +{ + "id": "openvla-libero10-5", + "title": "Put book into caddy", + "description": "OpenVLA LIBERO-10 task 5; one shared learned checkpoint.", + "instruction": "pick up the book and place it in the back compartment of the caddy", + "objects": [ + "book", + "caddy" + ], + "effects": [ + "book in back compartment of caddy" + ], + "prerequisites": [ + "LIBERO-10 matching task reset", + "Franka Panda simulation with 7D OSC pose actions", + "Reachable task objects in benchmark arrangement" + ], + "profileId": "libero-panda-openvla-v1", + "checkpoint": "openvla/openvla-7b-finetuned-libero-10", + "revision": "80970322773f81baa2e22fe495d0487b93a05cfa", + "license": "MIT (OpenVLA code and checkpoint); LIBERO MIT; Llama 2 terms apply", + "taskId": 5, + "installed": false, + "compatible": false, + "blocker": "Documentation example; runtime compatibility has not been checked" +}