Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ sole every step, so clearance remains relative to the local surface.
Each raw term is multiplied by its `RewardScales` weight and the control timestep; negative weights turn constraint
measurements such as `penalty_*` and `pose` into penalties. Curriculum-selected penalties are also multiplied by the
current `penalty_scale` according to completed episode length. This factor is exposed through
`info["metrics"]["penalty_scale"]`, and the final weighted terms through `info["Reward"]`.
`state.metrics["penalty_scale"]`, and the final weighted terms through `state.reward_terms`.

## Termination conditions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ reward_config = RewardConfig(

`base_height_target` and `initial_base_position[2]` should normally be close to the robot's default standing height.
Choose `target_foot_height` according to leg length and terrain variation: too little encourages dragging, while too much
can require unreasonable joint motion. When tuning a reward weight, inspect the matching value in `info["Reward"]`
can require unreasonable joint motion. When tuning a reward weight, inspect the matching value in `state.reward_terms`
rather than comparing configuration numbers alone.

## 7. `sensor`: sensor-name mapping
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ a body reference frame. Contact matching, swing clearance, and early swing-conta
| `swing_contact` | Measure the fraction of feet still touching the ground during swing | Penalize dragging and early touchdown |

Each raw term is first multiplied by its `RewardScales` weight; negative weights turn non-negative measurements into
penalties. `info["Reward"]` stores these weighted values before timestep scaling. Their sum is multiplied by `ctrl_dt` to
penalties. `state.reward_terms` stores these weighted values before timestep scaling. Their sum is multiplied by `ctrl_dt` to
produce the reward returned to the training algorithm. The current environment has no reward curriculum.

On rough terrain, `base_height` uses terrain height beneath the robot as its zero point. `swing_feet_z` consumes foot
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ python scripts/train.py task=g1-wbt-dance1-subject1/motrix.fastsac \
```

Check for missing motion, joint, or body names; systematic NaNs, joint-limit violations, or immediate bad-tracking after
reset; and verify that `info["Reward"]` and `info["metrics"]` reach the logs. Then start the default training run:
reset; and verify that `state.reward_terms` and `state.metrics` reach the logs. Then start the default training run:

```bash
python scripts/train.py task=g1-wbt-dance1-subject1/motrix.fastsac algo.asynchronous=true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ This makes the relative body-configuration reward independent of the current hor
| `undesired_contacts` | Count robot links whose net contact force exceeds the threshold and are absent from `allowed_contact_links` | Suppress body contacts not required by the motion |

Each raw term is multiplied by its `WbtRewardScales` weight and then by `ctrl_dt`. Negative weights turn `action_rate_l2`,
`limits_dof_pos`, and `undesired_contacts` into penalties. Final weighted terms are written to `info["Reward"]`.
`limits_dof_pos`, and `undesired_contacts` into penalties. Final weighted terms are written to `state.reward_terms`.

## Termination conditions

Expand All @@ -96,7 +96,7 @@ Each raw term is multiplied by its `WbtRewardScales` weight and then by `ctrl_dt
| Motion final frame | Neither | `motion_steps` reaches the clip end | Resample and reset motion state during training; restart at frame 0 during play |

`undesired_contacts` contributes only a reward penalty and does not terminate the episode. Termination rates and error means
are written to `info["metrics"]`.
are written to `state.metrics`.

## Reset logic

Expand Down
4 changes: 2 additions & 2 deletions docs/source/en/user_guide/envs/whole_body_tracking/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,8 +160,8 @@ python scripts/train.py task=g1-wbt-dance/motrix.fastsac \
```

During training, environments start at different points in the motion, and failure records increase the sampling probability
of difficult regions. Weighted reward terms are written to `info["Reward"]`; bad-tracking rates, motion progress, and
adaptive-sampling statistics are written to `info["metrics"]`. See [Task Environment Design](env_design.md) for definitions.
of difficult regions. Weighted reward terms are written to `state.reward_terms`; bad-tracking rates, motion progress, and
adaptive-sampling statistics are written to `state.metrics`. See [Task Environment Design](env_design.md) for definitions.

### Play the policy

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ Hooks a subclass implements:

| Hook | Responsibility |
| ------------------------------------------- | -------------------------------------------------------------------------------------- |
| `reset(env_ids)` | Write reset state (randomized initial poses, ...) for the selected rows and return an info dict; observations are produced afterwards by `compute_observation` |
| `reset(env_ids)` | Write reset state (randomized initial poses, ...) for the selected rows; observations are produced afterwards by `compute_observation` |
| `apply_action(actions, state)` | Write the action into the simulator (usually ctrl targets) |
| `compute_transition(state)` | Execute the read program and derive `state.reward`, `state.terminated`, ...; this is the only full data refresh of a step and must **not** write `state.obs` |
| `compute_observation(state)` | Assemble `state.obs` purely from already refreshed simulator data, without further reads |
Expand All @@ -156,7 +156,7 @@ Semantics:

- `terminated` marks episode-ending conditions such as task failure; `truncated` marks
the time limit at `max_episode_steps`. `ArrayEnv` combines both into `done` and
triggers auto-reset; `info["time_outs"]` flags rows that timed out without failing.
triggers auto-reset.
- The environment dimension must use vectorized NumPy operations; plain loops are only
allowed over a fixed number of joints, feet, or terms.
- Constants (initial poses, space definitions, query names) are precomputed in
Expand Down
2 changes: 1 addition & 1 deletion docs/source/en/user_guide/tutorial/building_envs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Every stage is orchestrated by the `ArrayEnv` base class; environment implementa
hooks and must not re-implement the lifecycle. Semantics:

- `terminated` marks episode-ending conditions such as task failure; `truncated` marks
the time limit, and `info["time_outs"]` flags rows that timed out without failing;
the time limit;
- environments that are done are reset automatically at the end of each step, and
observations are recomputed after the reset.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ TensorBoard logs are written under the run directory and can be viewed per envir
tensorboard --logdir runs/cartpole
```

Besides the standard return and loss curves, if an environment exposes per-term rewards via `info["Reward"]`, they are also logged to TensorBoard during training.
Besides the standard return and loss curves, if an environment exposes per-term rewards via `state.reward_terms`, they are also logged to TensorBoard during training.

## Model Evaluation and Testing

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ $$

每个原始项先乘以 `RewardScales` 中的权重,再乘以控制步长;`penalty_*` 和 `pose` 等约束项通过负权重成为惩罚。
课程指定的惩罚项还会根据已结束回合的平均长度乘以当前 `penalty_scale`。该缩放记录在
`info["metrics"]["penalty_scale"]`,最终的各加权项记录在 `info["Reward"]`。
`state.metrics["penalty_scale"]`,最终的各加权项记录在 `state.reward_terms`。

## 终止条件

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ reward_config = RewardConfig(

`base_height_target` 和 `initial_base_position[2]` 通常应接近机器人的默认站立高度。`target_foot_height` 应结合
腿长和地形起伏设置;过低容易拖脚,过高可能要求超出合理关节范围。调节某个奖励权重时,应查看
`info["Reward"]` 中对应项的量级,而不只比较配置数值。
`state.reward_terms` 中对应项的量级,而不只比较配置数值。

## 7. `sensor`:传感器名称映射

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Actor 与 Critic 共享的字段使用同一份带噪值;当前实现没有额
| `swing_feet_z` | 在摆动且未接触时,对足端高度误差应用指数核并按脚数平均 | 鼓励摆动脚达到目标离地高度 |
| `swing_contact` | 计算摆动期仍接触地面的脚所占比例 | 惩罚拖脚和过早落足 |

每个原始项先乘以 `RewardScales` 中的权重;负权重将非负度量转为惩罚。`info["Reward"]` 保存这些尚未
每个原始项先乘以 `RewardScales` 中的权重;负权重将非负度量转为惩罚。`state.reward_terms` 保存这些尚未
乘控制步长的加权项,所有项求和后再乘 `ctrl_dt`,得到返回给训练算法的单步奖励。当前环境没有奖励课程。

粗糙地形上,`base_height` 使用机器人当前位置的地形高度作为零点。`swing_feet_z` 的足端位置来自机体参考系,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ python scripts/train.py task=g1-wbt-dance1-subject1/motrix.fastsac \
```

检查 motion/joint/body 名称没有缺失,reset 后没有系统性 NaN、joint limit 违规或立即 bad-tracking,并确认
`info["Reward"]` 与 `info["metrics"]` 能进入日志。随后使用默认规模训练:
`state.reward_terms` 与 `state.metrics` 能进入日志。随后使用默认规模训练:

```bash
python scripts/train.py task=g1-wbt-dance1-subject1/motrix.fastsac algo.asynchronous=true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Actor 的参考身体姿态、基座角速度、关节位置和关节速度会
| `undesired_contacts` | 统计净接触力超过阈值且不在 `allowed_contact_links` 中的机器人 links | 抑制动作不需要的身体接触 |

每个原始项先乘以 `WbtRewardScales` 中的权重,再乘以 `ctrl_dt`;`action_rate_l2`、`limits_dof_pos` 和
`undesired_contacts` 通过负权重成为惩罚。最终加权项写入 `info["Reward"]`。
`undesired_contacts` 通过负权重成为惩罚。最终加权项写入 `state.reward_terms`。

## 终止条件

Expand All @@ -91,7 +91,7 @@ Actor 的参考身体姿态、基座角速度、关节位置和关节速度会
| 时间上限 | `truncated` | 训练回合达到 `max_episode_seconds`,内置配置为 10 s | 正常达到训练时限,不表示 bad tracking |
| Motion 末帧 | 两者都不是 | `motion_steps` 到达 clip 末尾 | 训练时重采样起始帧并重置 motion 状态;play 时从第 0 帧重播 |

`undesired_contacts` 只产生奖励惩罚,不直接终止回合。各类终止比例与误差均值记录在 `info["metrics"]`。
`undesired_contacts` 只产生奖励惩罚,不直接终止回合。各类终止比例与误差均值记录在 `state.metrics`。

## 重置逻辑

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ python scripts/train.py task=g1-wbt-dance/motrix.fastsac \
```

训练期间,环境从 motion 的不同时间点开始,并利用失败记录提高困难片段的采样概率。奖励分项写入
`info["Reward"]`,bad-tracking 比例、motion 进度和自适应采样统计写入 `info["metrics"]`;各项定义见
`state.reward_terms`,bad-tracking 比例、motion 进度和自适应采样统计写入 `state.metrics`;各项定义见
[任务环境设计](env_design.md)。

### 回放策略
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ class CartPoleEnv(DirectEnv):

| 钩子 | 职责 |
| ------------------------------------ | ---------------------------------------------------------------------------------------- |
| `reset(env_ids)` | 为选中的环境写入重置状态(随机化初始姿态等),返回 info 字典;观测交给后续的 `compute_observation` |
| `reset(env_ids)` | 为选中的环境写入重置状态(随机化初始姿态等);观测交给后续的 `compute_observation` |
| `apply_action(actions, state)` | 将动作写入仿真(通常是 ctrl 目标) |
| `compute_transition(state)` | 执行读取程序并派生 `state.reward`、`state.terminated` 等;这是每步唯一的全量数据刷新点,**不得**写 `state.obs` |
| `compute_observation(state)` | 纯粹用已刷新的仿真数据拼装 `state.obs`,不再执行读取 |
Expand All @@ -148,8 +148,7 @@ class CartPoleEnv(DirectEnv):
语义约定:

- `terminated` 表示任务失败等回合终止条件;`truncated` 表示达到 `max_episode_steps`
的时间截断。两者由 `ArrayEnv` 合成 `done` 并触发 auto-reset;
`info["time_outs"]` 标记"截断但未失败"的行。
的时间截断。两者由 `ArrayEnv` 合成 `done` 并触发 auto-reset。
- 环境维度必须使用 NumPy 向量化操作;只有遍历固定数量的关节、脚或 term 时才允许普通循环。
- 常量(初始姿态、空间定义、query 名称)在 `__init__` 或配置构造阶段预计算,
不在 step 循环中重复创建。
Expand Down
3 changes: 1 addition & 2 deletions docs/source/zh_CN/user_guide/tutorial/building_envs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,7 @@

所有阶段都由 `ArrayEnv` 基类编排,环境实现只填充钩子,不要重复实现生命周期。语义约定:

- `terminated` 是任务失败等回合终止条件;`truncated` 是达到回合时长上限的时间截断,
`info["time_outs"]` 标记"截断但未失败"的行;
- `terminated` 是任务失败等回合终止条件;`truncated` 是达到回合时长上限的时间截断;
- done 的环境在每步末尾被自动重置,观测在重置之后重新计算。

各阶段在两种工作流中分别由谁实现:DirectEnv 在
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ TensorBoard 日志写在 run 目录下,可按环境查看:
tensorboard --logdir runs/cartpole
```

除标准的回报、损失曲线外,若环境通过 `info["Reward"]` 暴露了各 reward 分项,训练时也会将其记录到 TensorBoard。
除标准的回报、损失曲线外,若环境通过 `state.reward_terms` 暴露了各 reward 分项,训练时也会将其记录到 TensorBoard。

## 模型评估和测试

Expand Down
4 changes: 2 additions & 2 deletions motrix_deploy_tasks/tests/test_training_parity.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def test_go2_training_and_deployment_task_golden_probe(env_name: str) -> None:
context = PolicyContext(
step=0,
elapsed_time_s=0.0,
command=PlanarVelocityCommand(env_state.info["commands"]),
command=PlanarVelocityCommand(env._commands),
)
task = create_task(profile.task, profile.robot)
command_scale = np.asarray(profile.task.config["command_scale"], dtype=np.float32)
Expand All @@ -53,7 +53,7 @@ def test_go2_training_and_deployment_task_golden_probe(env_name: str) -> None:
next_context = PolicyContext(
step=1,
elapsed_time_s=profile.control.period_s,
command=PlanarVelocityCommand(stepped.info["commands"]),
command=PlanarVelocityCommand(env._commands),
)
np.testing.assert_allclose(
task.build_observation(next_state, next_context),
Expand Down
37 changes: 8 additions & 29 deletions motrix_env_core/src/motrix_env_core/array/env.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,10 @@ class ArrayEnvState:
terminated: np.ndarray
truncated: np.ndarray
episode_steps: np.ndarray
info: dict
# Per-term reward breakdown for the latest transition: term name to a
# ``(num_envs,)`` array. Freshly written every transition; empty when the
# environment does not decompose its reward.
reward_terms: dict[str, np.ndarray] = dataclasses.field(default_factory=dict)
# Live diagnostics view: per-environment quantities stay unreduced as
# ``(num_envs,)`` arrays and are views into manager buffers that kernels
# overwrite every step; batch-level gauges are scalars. Values always
Expand Down Expand Up @@ -150,8 +153,7 @@ def init_state(self) -> ArrayEnvState:
terminated = np.ones((self._num_envs,), dtype=bool)
truncated = np.zeros((self._num_envs,), dtype=bool)
episode_steps = np.zeros((self._num_envs,), dtype=np.uint64)
info = {"time_outs": np.zeros((self._num_envs,), dtype=bool)}
self._state = self._new_state(obs, reward, terminated, truncated, episode_steps, info)
self._state = self._new_state(obs, reward, terminated, truncated, episode_steps)
self._reset_done_envs()
with self.perf.scope("observation"):
self._state = self.compute_observation(self._state)
Expand All @@ -166,7 +168,6 @@ def _new_state(
terminated: np.ndarray,
truncated: np.ndarray,
episode_steps: np.ndarray,
info: dict,
) -> ArrayEnvState:
"""Assemble the environment state; subclasses add simulator-owned fields."""
return ArrayEnvState(
Expand All @@ -175,7 +176,6 @@ def _new_state(
terminated=terminated,
truncated=truncated,
episode_steps=episode_steps,
info=info,
)

@property
Expand Down Expand Up @@ -223,31 +223,11 @@ def _reset_done_envs(self) -> None:
np.putmask(state.episode_steps, done, 0)
env_ids = np.flatnonzero(done)
with self.perf.scope("reset_envs"):
info1 = self.reset(env_ids)
self._merge_reset_info(state, info1, done)

def _merge_reset_info(self, state: ArrayEnvState, info1: dict, done: np.ndarray) -> None:
"""Merge one selected-row reset's info entries into the state info."""
if not info1:
return

def replace_dict_values(dst, new_values, mask):
for key, value in new_values.items():
if key not in dst:
dst[key] = value
else:
if isinstance(value, np.ndarray):
dst[key][mask] = value
elif isinstance(value, dict):
assert isinstance(dst[key], dict)
replace_dict_values(dst[key], value, mask)

with self.perf.scope("merge_info"):
replace_dict_values(state.info, info1, done)
self.reset(env_ids)

@abc.abstractmethod
def reset(self, env_ids: np.ndarray) -> dict:
"""Write reset rows for the selected environments and return reset info.
def reset(self, env_ids: np.ndarray) -> None:
"""Write reset rows for the selected environments.

Subclasses address their simulator's rows themselves. Observation
generation is deferred until :meth:`compute_observation`, after reset
Expand All @@ -265,7 +245,6 @@ def _update_truncate(self):
if not max_episode_steps:
return
self._state.truncated = self._state.episode_steps >= max_episode_steps
self._state.info["time_outs"] = self._state.truncated & ~self._state.terminated

@abc.abstractmethod
def create_renderer(self, config: RenderConfig) -> SimRenderer:
Expand Down
Loading
Loading