Skip to content

lmg_deploy: /lowstate torso IMU, any observation layout, arm_ik arms, game-task goals - #41

Merged
maxconwa merged 1 commit into
mainfrom
lmg_deploy/lowstate-imu-engage
Nov 21, 2026
Merged

maxconwa merged 1 commit into
mainfrom
lmg_deploy/lowstate-imu-engage

Conversation

@maxconwa

Copy link
Copy Markdown
Contributor

Why

LegsEnergyDefault200k_s43 failed its first real engage on 2026-11-20. Locomanipulation_game's response (main, ecaa375 and 16ec2c6) retrains every task: it engages from the real pre-pose range, reads the torso IMU as /lowstate reports it, and drops the accelerometer. This PR makes lmg_deploy run those exports. The policy shipped in policies/h12 is unchanged and still loads; no new policy is shipped yet, since its training is still running.

What changes

  • The /lowstate IMU. An export with imu.source: lowstate is read from LowState.imu_state:

    • the quaternion (wxyz) and accelerometer as reported;
    • the gyroscope through the export's imu.gyro_lowpass_hz, sampled at every /lowstate and timed by its tick;
    • no trim (imu_offset_* are ignored), and no wait for /livox/imu or FAST-LIO's attitude.

    The policy trained on the quaternion's measured bias, drawn per episode: pitch -3.6 to -2.0 deg, roll -0.6 to 0.4. Exports without source: lowstate keep the MID-360 path as before.

  • Any observation layout. Each actor's vector is built from the export's legs_obs_terms / arms_obs_terms. Any of the controller's terms may be left out, in its order; unknown or reordered terms are refused.

    • New LegsOnly / game legs: no base_lin_acc, arm_goal or ee_targets (79 inputs).
    • ArmsOnly / game arms: no IMU, command or gait clock (108 inputs).
  • The arms' trajectory action (arm_ik), ported from mdp.IKPostureArmAction for LocoManip-ArmsOnly-v0 and the game task:

    • the posture seed from arm_seed_table.npz's nearest row, solved for the goal, once per goal;
    • the synchronized trajectory from the target's own last value;
    • the arms' gravity torque over kp;
    • the pose-error integral;
    • the integrated residual that freezes on the goal.

    Gravity torques and the seed solve use Pinocchio (kinematics.py). reset() starts the target at the measured arm angles, as training's action term does.

  • Game-task goals. arm_goal_frame: pelvis keeps the goal on the pelvis (no SLAM odometry moves it). arm_goal_keeps_walking: true keeps the velocity command live through an arm goal. Both match how LocoManip-Marl-Direct-v0 now trains.

  • Residual cutoff. export.yaml's residual_cutoff_hz now takes precedence over the template's.

Checks

Check Result
lmg_deploy tests (golem_ros container) 68 passed, 1 skipped; new tests for trimmed layouts, arm_ik, pelvis-frame goals
Locomanipulation_game deploy_parity.py: this controller vs the training env in Isaac, a LegsOnly torso-IMU export, 300 steps passes; worst difference 4.3e-5 (arm targets)
arm_ik path vs Locomanipulation_game's numpy reference (mujoco_sim.process_actions), a LocoManip-Marl-Direct-v0 export, 4 x 300 steps targets to 1.4e-8 rad with shared seeds; gravity offset to 1.6e-16

Independently solved seeds differ from training's by up to 5e-4 rad. Training's float32 rotation error zeroes errors below about 1e-3 rad, so this cannot match exactly. Through the integral, the worst target difference is 5.3e-3 rad.

Known limits

  • Gravity offset on the lowstate path: it uses the biased torso-IMU attitude, where training used the true pelvis gravity. A 2.5 deg bias changes the horizontal gravity part by about 4%.
  • Goal start cost: starting an arm goal takes about 7 ms in that one step (the 100k-row seed search plus a 20-iteration solve).
  • Not run here: neither the real robot nor a Robocasa drive with a lowstate export; no trained export of that kind exists yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_015EG7pDM5uvsCpuEnXVWvMd

… game-task goals

For the Locomanipulation_game exports from 2026-11-20 (pre-pose engage training,
the torso IMU):

- imu.source lowstate: the node takes LowState.imu_state's quaternion and
  accelerometer as reported and its gyroscope through the export's low-pass
  (timed by the LowState tick), untrimmed, without /livox/imu or FAST-LIO's
  attitude. Other exports keep the MID-360 path.
- Observations: each actor's vector is built from the export's legs_obs_terms /
  arms_obs_terms, any of the controller's terms left out in its order (the new
  legs drop base_lin_acc, arm_goal and ee_targets: 79; ArmsOnly's arms drop the
  IMU, command and gait clock: 108). Unknown or reordered terms are refused.
- arm_ik (mdp.IKPostureArmAction, LocoManip-ArmsOnly-v0 and -Marl-Direct-v0):
  seed from arm_seed_table.npz's nearest row, solved for the goal, once per goal;
  the synchronized trajectory from the target's own last value; the arms' gravity
  torque over kp; the pose-error integral; the integrated, freezing residual.
  Pinocchio gravity torques and seed solve in kinematics.py. Matches
  Locomanipulation_game's numpy reference (mujoco_sim) to 1.4e-8 rad with shared
  seeds; independently solved seeds differ by up to 5e-4 rad (training's float32
  rotation error).
- arm_goal_frame pelvis (goals ride on the pelvis, no odometry) and
  arm_goal_keeps_walking (the velocity command live through an arm goal), as
  LocoManip-Marl-Direct-v0 trains them; export.yaml's residual_cutoff_hz over the
  template's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EG7pDM5uvsCpuEnXVWvMd
@maxconwa
maxconwa merged commit 3a66511 into main Nov 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant